Now run the same project with a different decision — and see what it does. We did this with our stand-in; you will do it with your own data.

Two runs

python capstone.py --no-augment --out noaug.pt --summary noaug.json
python capstone.py --augment    --out aug.pt   --summary aug.json

Augmentation here: a mirror flip, a rotation of up to ±10°, a shift of up to 10 % and a scale of 0.9–1.1, applied freshly every time to the training images (lesson 9). The validation and test/ images stay untouched.

This is what we got. The first line is what you saw in the previous step.

without augmentation   val 95.6%   TEST 93.8%   (93.75 %, 400 new images)
with augmentation      val 90.6%   TEST 89.0%

The full report with augmentation:

epoch  1  loss 0.5118  train acc 85.6%  |  val loss 0.3012  val acc 90.6%
epoch  2  loss 0.2428  train acc 90.9%  |  val loss 0.2755  val acc 90.6%
epoch  3  loss 0.1900  train acc 93.0%  |  val loss 0.3039  val acc 89.4%
epoch  4  loss 0.1691  train acc 94.2%  |  val loss 0.4211  val acc 87.5%
early stop at epoch 4: best val loss was 0.2755

validation accuracy: 90.6%
               coat pullover  sneaker   tshirt   <- predicted
coat             27        9        1        3
pullover          0       40        0        0
sneaker           0        0       40        0
tshirt            0        2        0       38

TEST accuracy: 89.0% on 400 new images
               coat pullover  sneaker   tshirt   <- predicted
coat             72       20        2        6
pullover          1       91        0        8
sneaker           0        0      100        0
tshirt            0        6        1       93

More than two runs

These two runs still mix two things: augmentation, and the fact that with it early stopping fired sooner. So we ran all four combinations.

run	epochs	val	test	s
aug_stop2 — with augmentation, --patience 2	4	90.6 %	89.0 %	174
aug_8ep — with augmentation, --patience 0 --epochs 8	8	93.1 %	92.25 %	269
noaug_stop2 — without it, --patience 2 (the default)	6	95.6 %	93.75 %	221–293
noaug_5ep — without it, --patience 0 --epochs 5	5	93.75 %	94.0 %	232

(Seconds are from our busy computer, ±2×.)

What the table shows

1. The "good practice" did not help here. The only row where augmentation and early stopping work together is clearly worse: validation 90.6 % against 95.6 %, new images 89.0 % against 93.75 %. The loss is about 5 points on both sets, and larger than the error of a single measurement (±2.4).

2. The loss is concentrated in one class. coat recall dropped to 0.68 on validation (27/40) and 0.72 on new images (72/100) — without augmentation it was 0.93 and 0.88. On new images 20 coats became pullovers (without augmentation — 10). The number "89.0 %" would not say that; the matrix does.

3. Early stopping stopped early. With augmentation the validation loss went 0.3012 → 0.2755 → 0.3039 → 0.4211: it jumps, and the stop fired after epoch 4. The same --patience 2 without augmentation allowed 6 epochs. The training curve with augmentation was still rising; we do not know whether training longer would have got us further — the row aug_8ep (no stopping, 8 epochs) shows 92.25 %, still below the non-augmented ones.

4. Do not rank the three "non-augmented" rows. 93.75 %, 93.75 % and 94.0 % differ by one new image (0.25 points), and their error is ±2.4. noaug_5ep has the highest test but a lower val — we did not choose it, and that is right.

Why? Honestly — we do not know

We did not isolate why augmentation hurts here. Plausible reasons that we can only suspect:

• these pictures are 28×28 pixel thumbnails blown up to 224: a rotation and a shift on an image that small may erase what separates a coat from a pullover;
• 160 validation images combined with early stopping is touchy: one "bad" jump in an epoch, and it stops.

It looks plausible, but it is unchecked. Other data can behave differently: in a side experiment with coloured CIFAR-10 photos a network trained from scratch got 57.5 % with augmentation instead of 52.5 % (only 160 validation images, so a hint, not a proof). The lesson is not "augmentation is bad" but "nobody can tell you whether it helps your data" — only validation can.

The default was chosen on validation

That is why capstone.py defaults to --no-augment. What matters is how that was chosen: 95.6 % against 90.6 % on validation. The new images (93.75 % against 89.0 %) said the same, but the choice would have been identical without them. Had it gone the other way — validation for augmentation, new images against — we would still have chosen on validation and simply reported that the new images disagreed.

Your turn

Do at least two runs on your data: with and without augmentation. If you like, add a third (another --lr, another --epochs). Save each with its own --out and --summary, and write down how many runs you made: that is how many times you looked at test/.

Your answer may come out different from ours: if augmentation helps your data, that is the result.