Now you have two sets and the tools. Trusting a number depends on not making five mistakes, none of which raises an error message. All of them are my runs: MLP 256-256, 10 000 training and 5 000 validation images, seed 42.

In this step we use the official test set twice — only to demonstrate how the validation number differs from what waits beyond it. In your project, touch the test set once, at the very end.

1. You forgot model.eval() — the model measures itself at random

Training with dropout 0.3, then two measurements of the same model on the same images:

val accuracy with eval(): 86.6%   with dropout still on: 84.8%
three repeat measurements, same model, same data: 84.8% 84.3% 84.6%

Dropout switched off a random part of the neurons while measuring. The result is not only worse (84.8 vs 86.6 %) — it is different every time (84.8 / 84.3 / 84.6) for the same model. An instrument that gives different answers for the same object is no longer an instrument.

The symptom: you measure the same thing twice and get different numbers. If so — look for eval().

2. Validation leaked into training

We deliberately mixed validation images into the training set and "validated" on them:

'validation' accuracy: 96.9%   real accuracy on the untouched test set: 87.5%

96.9 % — a wonderful number, and utterly empty. The real one is 87.5 %. No code changes: the same fit, the same evaluate; the only thing that changes is which images ended up where. That is why test_fashion.py is so pedantic.

The symptom: the validation accuracy is better than plausible for this model and data. If it seems to you that "I have just started and already 97 %" — suspect a leak first.

3. Too much regularisation — no gap, and no network either

dropout 0.9        train 67.6%  val 65.6%  val loss 0.959
weight decay 0.1   train 64.6%  val 63.9%  val loss 1.010

The gap is about 2 and 1 points. Great? No: the network simply does not learn (67 % and 65 % — worse than the plain linear model of lesson 5). A small gap is a symptom, not a goal. The goal is a high validation number.

The symptom: both the training and validation accuracies are low and close. That is not overfitting but underfitting; the regularisation has to be reduced, not increased.

4. Early stopping without putting the weights back

Training for 30 epochs, then looking at where it would have been best:

best val loss at epoch 9: acc 85.8%; stopping at epoch 30 instead: val loss 0.491 (was 0.396)
val loss by epoch: 0.59 0.51 0.54 0.48 0.45 0.44 0.45 0.42 0.40 0.44 0.52 0.41 0.42 0.42 0.42 0.41 0.44 0.44 0.45 0.43 0.45 0.45 0.44 0.48 0.51 0.50 0.51 0.48 0.50 0.49

If you stop but do not take the best weights back, you hold the model from the moment you stopped — not from when it was best. The cost: 0.491 instead of 0.396. The test test_patience_puts_the_best_weights_back checks exactly that.

Look at the curve: it is not smooth. At epoch 11 the loss jumps to 0.52, at epoch 12 it is back at 0.41. That is why patience cannot be 1: one bad jump does not mean it is over. Why 3 or 5 rather than 10 — there is no rule, only a compromise.

5. The best of many — and optimism

Training six times with different seeds; we pick the best by validation.

val : 86.3% 87.2% 85.6% 85.8% 86.3% 86.3%
test: 85.8% 86.1% 85.0% 85.6% 86.0% 86.0%
best-on-val is seed 1: val 87.2% -> test 86.1%; mean val 86.2% mean test 85.8%

By validation the best is seed 1: 87.2 %. Its real number is 86.1 %. On average validation exceeded test by 0.4 points (86.2 vs 85.8 — simply two different sets of images); for the picked one — by 1.1. The extra ~0.7 points is what the act of picking "the best" added.

The optimism is small but directional: the more often you look at the validation number and choose by it, the less it is worth. That is why a set you have never looked at is needed for the end.

What to take away

Five symptoms: unequal measurements (1), too good (2), low and close (3), worse than the best moment (4), the best of many (5). None has an error message — which is why you need to know what they look like.