In the last step you got two numbers from the same model:

set	images	right	accuracy
validation (part of data/)	160	153	95.6 %
new images (test/)	400	374	93.5 %

This is the most important result of the lesson. Not 93.5 % itself, but the fact that it is lower than the one you were pleased with.

Why the validation number was too good

The model never "saw" the validation images — they took no part in updating the weights. But you saw them, and more than once. Every decision — freeze the network or fine-tune layer4, which learning rate, which image size — was checked against the same 160 images (our comparisons in lesson 10 were too), and the variant that looked best there was kept. Choosing is itself a little bit of learning: each "I decided on that" fits the model slightly to those 160 images.

We measured the same thing with lesson 7's network (FashionMNIST, 10,000 training images): run six times with different seeds and pick the best one by validation, and 87.2 % became 86.1 % on untouched images (the means of all six — 86.2 % and 85.8 %). Small, but the direction is the same.

How much error the number itself has

The two numbers are not two truths — they are two measurements, each with an error. Roughly, the 95 % confidence interval for an accuracy p measured on n images is

± 1.96 · √( p · (1 − p) / n )

(this is the normal approximation; for small n and p near 100 % it is not very exact, but it shows the order of magnitude.)

set	n	accuracy	±
validation	160	95.6 %	±3.2 points
new images	400	93.5 %	±2.4 points

The intervals overlap: 95.6 % ± 3.2 runs from 92.4 to 98.8; 93.5 % ± 2.4 from 91.1 to 95.9. The 2.1-point difference between the two numbers fits inside the error. From one run we cannot prove that the validation number was "systematically" too good.

Honestly put:

• one validation image = 0.6 points (1/160); one new image = 0.25 points (1/400). The difference between two runs that you would like to call an "improvement" is often only a couple of pictures;
• the direction (validation above new images) agrees with that six-run measurement and with what we would expect when the choice is made on that same set — but one run does not prove it;
• and the new-image number has its own ±2.4. If your test/ has three classes of 20 images each, one image is about 1.7 points (1/60) and the error is larger still.

Which number to quote

Quote the new-image number, together with the count: "93.5 % on 400 new images". No decision used it, so it carries no selection bias.

But only as long as you do not use it for decisions. If, after seeing the test/ result, you decide "I will try another learning rate" — test/ has become a validation set, and you need a new one. The one-time rule: look at test/ once, when everything is already decided.

What the matrix said that the number did not

On new images there are 26 mistakes. 25 of them involve pullover: 11 coats called pullover, 5 pullovers called coat, 7 tshirts called pullover, 2 pullovers called tshirt. That is 16 coat↔pullover and 9 tshirt↔pullover confusions. sneaker — again 100 out of 100.

On the validation set coat recall was 0.93; on new images it is 0.89. The statement "the model gets 93.5 % right" does not say that one coat in nine comes out as a pullover. If this is your product filter, where "pullover" and "coat" cost different amounts, the average is not what matters to you — this row is.

What to take away

question	answer
Where does it go wrong?	the confusion matrix
Which mistake hurts most?	precision (when it says X, is it really X) or recall (did it find all the X) — whichever costs you more
How far can it be trusted?	the number together with the image count and the error
Which number to quote?	the new-image one, decided before you looked at it