Evaluation & Deployment

Code

The confusion matrix — fill in two holes

In the last step the matrix appeared "out of thin air". Now we write the same table as code — and you fill in the two missing pieces.

Your job

Create evaluate.py and test_evaluate.py (see the panel on the right).

evaluate.py is complete and runs — it does not crash — but it gives wrong results. Two places are missing, marked HOLE 1 and HOLE 2:

HOLE 1 — in the confusion_matrix loop: for every image, write one tally into the table. The row is the truth t, the column is the model's answer p.
HOLE 2 — in class_scores: precision and recall. You divide the same tp by two different sums. From the last step: recall is read along the row, precision down the column.

Nothing else needs changing. print_report is already written and only prints what your functions return.

A few PyTorch things you may need:

cm[i] is the whole i-th row; cm[:, i] is the whole i-th column;
.sum() adds up; .item() turns a one-number tensor into an ordinary Python number.

test_evaluate.py is the verdict — you do not write it, you pass it. It holds the same six-image example from the last step, so you can check the answers by hand, too.

Run the test before you fill anything in

Run it exactly as it is:

$ python -m unittest

unittest prints long tracebacks; only the essentials are kept here:

FAIL: test_f1_combines_both
AssertionError: 0.0 != 0.4 within 4 places (0.4 difference)
FAIL: test_precision_is_a_column
AssertionError: 0.0 != 0.3333333333333333 within 4 places (0.3333333333333333 difference)
FAIL: test_recall_is_a_row
AssertionError: 0.0 != 0.5 within 4 places (0.5 difference)
FAIL: test_every_image_is_counted_once
AssertionError: 0 != 6
FAIL: test_rows_are_the_truth_columns_the_prediction
AssertionError: Lists differ: [[0, 0, 0], [0, 0, 0], [0, 0, 0]] != [[2, 1, 0], [1, 1, 0], [0, 1, 0]]
FAIL: test_report_prints_the_scores_per_class
AssertionError: '0.33' not found in 'dog             0.00     0.00   0.00       0'
Ran 7 tests in 0.002s
FAILED (failures=6)

That is not an error — it is the starting point. The skeleton creates the table but never fills it (all zeros), so every score is zero. Six tests fail; the seventh (never_predicted) passes by accident, because zero is exactly the answer it expects. Fill in both holes and run again — you should see OK.

When it does not pass

[[2, 1, 0], [1, 1, 1], [0, 0, 0]] != [[2, 1, 0], [1, 1, 0], [0, 1, 0]] — you wrote cm[p, t] instead of cm[t, p]. The table is mirrored: rows became columns. Accuracy (the diagonal) does not change, so this mistake stays hidden for a long time — while precision and recall silently trade places.
0.5 != 0.3333… in the precision test and 0.3333… != 0.5 in the recall test — you swapped row and column. Precision divides by the column sum (cm[:, i].sum()), recall by the row sum (cm[i].sum()).
0.1666… != 0.3333… — you divided by the number of all images (cm.sum()), not by a row or a column.
ZeroDivisionError: division by zero — the class fox was never predicted, so its column sum is 0, and Python does not let a zero division pass quietly. Make sure the divisor is never below 1: max(total, 1). The numerator is 0 too, so the result is 0.0.

evaluate.py CREATE

Create this file in ./

"""evaluate.py — look past one accuracy number: where does the model go wrong?"""

import torch


def confusion_matrix(
    targets: torch.Tensor, preds: torch.Tensor, n: int
) -> torch.Tensor:
    """cm[i, j] = how many images of true class i were predicted as class j."""
    cm = torch.zeros(n, n, dtype=torch.long)
    for t, p in zip(targets, preds):
        # HOLE 1: count this image. One more image whose TRUE class is t
        # and whose PREDICTED class is p. Rows are the truth.
        pass
    return cm


def class_scores(cm: torch.Tensor, i: int) -> tuple[float, float, float]:
    """(precision, recall, f1) of class i, read off the confusion matrix.

    A class that was never predicted has precision 0.0 - never divide by zero.
    """
    tp = cm[i, i].item()  # true positives: really class i, called class i
    # HOLE 2: precision = tp / (everything PREDICTED as i)
    #         recall    = tp / (everything that REALLY IS i)
    precision = 0.0
    recall = 0.0
    f1 = 2 * precision * recall / max(precision + recall, 1e-9)
    return precision, recall, f1


def print_report(cm: torch.Tensor, classes: list[str]) -> None:
    print(f"{'':<10}" + "".join(f"{c[:8]:>9}" for c in classes) + "   <- predicted")
    for i, c in enumerate(classes):
        print(f"{c:<10}" + "".join(f"{v.item():>9}" for v in cm[i]))

    print(f"\n{'class':<10}{'precision':>10}{'recall':>9}{'f1':>7}{'images':>8}")
    for i, c in enumerate(classes):
        precision, recall, f1 = class_scores(cm, i)
        print(f"{c:<10}{precision:>10.2f}{recall:>9.2f}{f1:>7.2f}{cm[i].sum().item():>8}")

test_evaluate.py CREATE

Create this file in ./

"""test_evaluate.py — the verdict for evaluate.py. Run: python -m unittest -v"""

import io
import unittest
from contextlib import redirect_stdout

import torch

from evaluate import class_scores, confusion_matrix, print_report

# Six images, three classes: 0 = cat, 1 = dog, 2 = fox.
# The model never says "fox" - so class 2 is never predicted.
TARGETS = torch.tensor([0, 0, 0, 1, 1, 2])
PREDS = torch.tensor([0, 0, 1, 1, 0, 1])
CLASSES = ["cat", "dog", "fox"]


class TestConfusionMatrix(unittest.TestCase):
    def test_rows_are_the_truth_columns_the_prediction(self):
        cm = confusion_matrix(TARGETS, PREDS, 3)
        self.assertEqual(cm.tolist(), [[2, 1, 0], [1, 1, 0], [0, 1, 0]])

    def test_every_image_is_counted_once(self):
        cm = confusion_matrix(TARGETS, PREDS, 3)
        self.assertEqual(cm.sum().item(), 6)


class TestClassScores(unittest.TestCase):
    def setUp(self):
        self.cm = confusion_matrix(TARGETS, PREDS, 3)

    def test_precision_is_a_column(self):
        # dog was predicted 3 times (1 right) -> 1/3
        precision, _, _ = class_scores(self.cm, 1)
        self.assertAlmostEqual(precision, 1 / 3, places=4)

    def test_recall_is_a_row(self):
        # there are 2 real dogs (1 found) -> 1/2
        _, recall, _ = class_scores(self.cm, 1)
        self.assertAlmostEqual(recall, 1 / 2, places=4)

    def test_f1_combines_both(self):
        _, _, f1 = class_scores(self.cm, 1)
        self.assertAlmostEqual(f1, 0.4, places=4)

    def test_a_class_that_is_never_predicted_does_not_crash(self):
        self.assertEqual(class_scores(self.cm, 2), (0.0, 0.0, 0.0))


class TestReport(unittest.TestCase):
    def test_report_prints_the_scores_per_class(self):
        cm = confusion_matrix(TARGETS, PREDS, 3)
        out = io.StringIO()
        with redirect_stdout(out):
            print_report(cm, CLASSES)
        lines = out.getvalue().splitlines()
        table = lines[next(i for i, l in enumerate(lines) if l.startswith("class")):]
        dog_line = next(l for l in table if l.startswith("dog"))
        self.assertIn("0.33", dog_line)  # precision
        self.assertIn("0.50", dog_line)  # recall


if __name__ == "__main__":
    unittest.main()