In lesson 9 you saw how one accuracy number can lie: the model got 93.5 % of the validation images right, while one class — coat — was right 1 time in 4. The number was real. It just did not say where the model goes wrong.

In lesson 10 you built a model that scored 95.6 % on the validation set. Today we ask three things that number does not answer:

1. Where does it go wrong? Which classes it confuses, and whose mistakes cost the most.
2. Can that number be trusted? Every decision you made was checked against the validation images — are they still honest?
3. Can anyone use it? A model that lives only inside the training script helps nobody.

What you will build

• evaluate.py — a confusion matrix and per-class scores: precision, recall, F1;
• a saved model that carries its class list and can be loaded again;
• predict.py — a prediction (inference) for one image;
• serve.py — a small HTTP endpoint: send an image, get JSON back.

Only three ideas are new: the confusion matrix, "new images", and the fact that a model has to be loaded correctly, not just trained. Everything else you have already done.

The lesson climbs in steps

You never start from a blank raise NotImplementedError. Each part arrives in steps:

1. Watch. A confusion matrix laid out by hand with real numbers. You write nothing.
2. Fill the holes. evaluate.py arrives almost complete; two places are missing, and a test tells you whether you hit them.
3. Write it yourself. Loading the model and collecting predictions — empty skeletons.
4. Run it. You put it together and see a number you did not expect.

The remaining steps are traps you will really meet, and drills.

Your data, not ours

In this lesson and the next you work with your own categories, put into folders as in lesson 9:

data/                 test/
  cat/    *.png         cat/    *.png
  dog/    *.png         dog/    *.png
  rabbit/ *.png         rabbit/ *.png

data/ is what you train on and check against. test/ holds new images that neither the model nor you have seen: at least ~20 new photos per class that are not in data/.

How the numbers here were produced. We do not have your photos, so every number in this lesson comes from a FashionMNIST stand-in: four classes (tshirt, pullover, coat, sneaker), 200 small grey images each in data/, plus 100 new ones each in test/. The code is the same; only the folders change. Your numbers will differ from the ones shown — by a point or two, maybe more if your classes are different. That is not a bug.

Guess before you read on

On the validation set (160 images) the model scored 95.6 %. Now we give it 400 images it has never seen.

Write down: will the number be higher, the same, or lower? And roughly how much?