Run it: validation and new images
All the parts are ready. Run them in order.
1. The validation set
python evaluate.py --model model.pt --data data
--data is the same folder you trained with. evaluate.py uses the same function (make_loaders, the same seed=42) to recreate the same split and takes the validation part. Our run:
evaluating the validation split of data
accuracy: 95.6% on 160 images
coat pullover sneaker tshirt <- predicted
coat 37 3 0 0
pullover 1 39 0 0
sneaker 0 0 40 0
tshirt 0 3 0 37
class precision recall f1 images
coat 0.97 0.93 0.95 40
pullover 0.87 0.97 0.92 40
sneaker 1.00 1.00 1.00 40
tshirt 1.00 0.93 0.96 40
7 mistakes, first 8:
data/coat/0088.png: truth coat, predicted pullover
data/coat/0089.png: truth coat, predicted pullover
data/coat/0008.png: truth coat, predicted pullover
data/pullover/0108.png: truth pullover, predicted coat
data/tshirt/0118.png: truth tshirt, predicted pullover
data/tshirt/0180.png: truth tshirt, predicted pullover
data/tshirt/0030.png: truth tshirt, predicted pullover
It is the same matrix you laid out by hand in step 2. Notice the last block: with file names. A single number would have said "7 mistakes"; this list says which ones to open and look at.
2. New images
python evaluate.py --model model.pt --data data --test-dir test
This time the model gets 400 images (100 per class) it has never seen — not in training, not in validation:
evaluating every image in test
accuracy: 93.5% on 400 images
coat pullover sneaker tshirt <- predicted
coat 89 11 0 0
pullover 5 93 0 2
sneaker 0 0 100 0
tshirt 1 7 0 92
class precision recall f1 images
coat 0.94 0.89 0.91 100
pullover 0.84 0.93 0.88 100
sneaker 1.00 1.00 1.00 100
tshirt 0.98 0.92 0.95 100
26 mistakes, first 8:
test/coat/0000.png: truth coat, predicted pullover
test/coat/0003.png: truth coat, predicted pullover
test/coat/0004.png: truth coat, predicted pullover
test/coat/0006.png: truth coat, predicted pullover
test/coat/0009.png: truth coat, predicted pullover
test/coat/0013.png: truth coat, predicted pullover
test/coat/0065.png: truth coat, predicted pullover
test/coat/0070.png: truth coat, predicted pullover
Compare with the earlier run: 95.6 % → 93.5 %. What that means we discuss in the next step. First, one trap that is worth seeing with your own eyes.
Delete the line model.eval() from load_model and run the same command with --test-dir:
evaluating every image in test
accuracy: 40.7% on 400 images
coat pullover sneaker tshirt <- predicted
coat 39 32 9 20
pullover 24 46 13 17
sneaker 18 14 47 21
tshirt 29 28 12 31
93.5 % became 40.7 %, and on the validation set the same model scores 51.2 %. No warning, no error. In the matrix the "mistakes" are scattered everywhere — even sneaker is confused with coat, which was perfect before.
The cause: a freshly built model is in training mode, and in that mode BatchNorm normalises by its own batch's statistics. make_eval_loader does not shuffle, so each batch is almost all one class, and the network "normalises away" whatever tells that class apart. Put model.eval() back.
What you have now
model.pt— a model that can live without the training script;- a matrix and a report, for the validation set and for new images;
- a list of concrete images the model gets wrong.
An honest question we postponed: why do the two numbers differ, and which one to believe.