There is no code here. What you see here is what happens when you look at a model's mistakes through more than one number.

Six pictures, three classes

A small example you can count with a pencil. Three classes: cat, dog, fox. The model saw six images:

image   truth    model said
  1     cat      cat
  2     cat      cat
  3     cat      dog     <- mistake
  4     dog      dog
  5     dog      cat     <- mistake
  6     fox      dog     <- mistake

Accuracy: 3 out of 6 = 50 %. That number says nothing more.

A confusion matrix is a table where every image is written down exactly once: the row is what it really is, the column is what the model said.

                cat      dog      fox   <- predicted
cat               2        1        0
dog               1        1        0
fox               0        1        0

Read it row by row: of the three real cats the model called two cat and one dog. The only real fox was called dog. The diagonal (2, 1, 0) holds the right answers. Everything else is a mistake, and you can see what kind.

Two different kinds of score come out of the same table:

class      precision   recall     f1  images
cat             0.67     0.67   0.67       3
dog             0.33     0.50   0.40       2
fox             0.00     0.00   0.00       1

• Recall — of the images that really are this class, how many did the model find. Read a row: dog has 2 real ones, 1 found → 1/2 = 0.50.
• Precision — of everything the model called this class, how much was right. Read a column: 3 images were called dog, 1 was right → 1/3 = 0.33.
• F1 — the (harmonic) mean of both; it drops sharply if either one is low.

For fox the model never once said "fox", so precision is not even defined (0 out of 0). The program picks 0.00 — and that is a decision, not a law of nature.

The real matrix: 160 validation images

Now the same thing with the real model from lesson 10 (ResNet-18, layer4 fine-tuned), four classes and 160 validation images — 40 of each class:

               coat pullover  sneaker   tshirt   <- predicted
coat             37        3        0        0
pullover          1       39        0        0
sneaker           0        0       40        0
tshirt            0        3        0       37

class      precision   recall     f1  images
coat            0.97     0.93   0.95      40
pullover        0.87     0.97   0.92      40
sneaker         1.00     1.00   1.00      40
tshirt          1.00     0.93   0.96      40

Three observations. Worth not skipping, because every later step rests on them.

1. Accuracy is the diagonal divided by everything. (37 + 39 + 40 + 37) / 160 = 153 / 160 = 95.6 %. The matrix loses nothing — you can recover the same number from it, but you cannot recover the matrix from the number.

2. The mistakes are concentrated. Seven mistakes, and all seven involve pullover: 3 coats and 3 tshirts were called pullover, and one pullover was called coat. sneaker is perfect. One number (95.6 %) does not reveal that the model has one confusion, not seven random ones.

3. Precision and recall are not the same thing. pullover: recall 0.97 (39 of 40 found), but precision 0.87 (it said "pullover" 45 times, 39 of them right). The model likes that answer — it also uses it for coats and tshirts. tshirt is the other way round: precision 1.00 (when it says "tshirt" it is always right), recall 0.93.

In the next step you build that table yourself — and a test tells you if your row and column swapped places.