Evaluation & Deployment

Drills

Four tasks, from easy to harder. All of them live in one new file, drills.py, next to evaluate.py (on the right): it already has the new-image matrix TEST, the class names and empty skeletons.

Try each function on TEST and compare with the given value. Try on your own first; the solutions are at the end of the step.

1. accuracy(cm)

Return the accuracy (0…1) from the confusion matrix alone: the diagonal divided by everything.

On TEST you should get 0.935.

2. most_confused(cm, classes)

Return the triple (true class, called class, how many times) — the largest value outside the diagonal.

On TEST: ('coat', 'pullover', 11).

Hint: a copy of the matrix with zeros on the diagonal, then argmax. Ask yourself: what must you not damage — the original matrix?

3. macro_f1(cm) — and why it exists

Return the average F1 across classes (every class weighs the same), using your own class_scores.

On TEST: 0.9357.

Now build the matrix for "a model that always answers not a sandal" — made up, 10,000 images, 990 of them sandals:

always_no = torch.tensor([[9010, 0], [990, 0]])

Compute accuracy and macro_f1. Explain in one sentence why the two numbers differ so much. (It is the same trap you may recognise from lesson 3: the accuracy of a neuron that always answers "not a sandal" was 90.1 %.)

4. Error: margin_of_error(correct, n) and images_needed(p, margin)

The first: with the formula from step 7, return half the width of the 95 % interval (0…1). The second: how many images are needed so that the error is no larger than margin, when the accuracy is about p.

Check: margin_of_error(153, 160) ≈ 0.0317, margin_of_error(374, 400) ≈ 0.0242.

Then answer: how many new images do you need for the error to be ±2 points at about 90 % accuracy? And ±1 point? What does this say about a test/ folder with 20 photos per class?

Solutions

def accuracy(cm):
    return cm.diagonal().sum().item() / cm.sum().item()


def most_confused(cm, classes):
    off = cm.clone()          # do not touch the original
    off.fill_diagonal_(0)     # right answers do not matter here
    i, j = divmod(off.argmax().item(), cm.shape[1])
    return classes[i], classes[j], off[i, j].item()


def macro_f1(cm):
    return sum(class_scores(cm, i)[2] for i in range(cm.shape[0])) / cm.shape[0]


def margin_of_error(correct, n):
    p = correct / n
    return 1.96 * math.sqrt(p * (1 - p) / n)


def images_needed(p, margin):
    return math.ceil(1.96**2 * p * (1 - p) / margin**2)

What you should get:

accuracy(TEST)               0.935
most_confused(TEST, CLASSES) ('coat', 'pullover', 11)
macro_f1(TEST)               0.9357
always_no:  accuracy 0.901   macro_f1 0.474
margin_of_error(153, 160)    0.0317
margin_of_error(374, 400)    0.0242
images_needed(0.9, 0.02)     865
images_needed(0.9, 0.01)     3458

Explanation of 3. always_no says "not a sandal" and is right 9010 times out of 10,000 — 90.1 %. But the one class the whole task is about has recall 0.00 and precision 0.00. Macro F1 averages over classes, not images, so that zero becomes visible: 0.474. When classes are unequal, accuracy lulls you — the same lesson as in lesson 9 with coat (1 in 4).

Explanation of 4. ±2 points takes 865 images, ±1 point takes 3,458. The error shrinks like 1/√n: to halve it you need four times as many images. With 20 photos per class and ~90 % accuracy, one class's recall carries more than ±13 points (1.96·√(0.9·0.1/20) ≈ 0.13) — so your small test/ can show a direction, but not a fine difference between two models.

drills.py CREATE

Create this file in ./

"""drills.py — four small functions on a confusion matrix. Fill each one in."""

import math

import torch

from evaluate import class_scores

# The matrix from the run on NEW images (rows = truth, columns = prediction).
TEST = torch.tensor([[89, 11, 0, 0], [5, 93, 0, 2], [0, 0, 100, 0], [1, 7, 0, 92]])
CLASSES = ["coat", "pullover", "sneaker", "tshirt"]


def accuracy(cm: torch.Tensor) -> float:
    """The diagonal divided by everything, 0..1."""
    raise NotImplementedError


def most_confused(cm: torch.Tensor, classes: list[str]) -> tuple[str, str, int]:
    """(true class, class it was called, how many times): the largest
    value OUTSIDE the diagonal. Must not modify cm."""
    raise NotImplementedError


def macro_f1(cm: torch.Tensor) -> float:
    """The average F1 over classes; every class weighs the same."""
    raise NotImplementedError


def margin_of_error(correct: int, n: int) -> float:
    """Half the width of the 95 % interval for an accuracy, 0..1."""
    raise NotImplementedError


def images_needed(p: float, margin: float) -> int:
    """How many images keep the error within `margin` at accuracy about p."""
    raise NotImplementedError


if __name__ == "__main__":
    print("accuracy", accuracy(TEST))
    print("most_confused", most_confused(TEST, CLASSES))
    print("macro_f1", round(macro_f1(TEST), 4))
    print("margin_of_error", round(margin_of_error(153, 160), 4), round(margin_of_error(374, 400), 4))
    print("images_needed", images_needed(0.9, 0.02), images_needed(0.9, 0.01))