You write nothing here. Watch what one run does from start to finish, and note where every line came from. You have done all of this separately; now it is in one script.

python capstone.py --data data --test-dir test

1. The data

classes: ['coat', 'pullover', 'sneaker', 'tshirt']
train images: 640, val images: 160
trainable parameters: 8,395,780 of 11,178,564

The classes are in alphabetical order (ImageFolder, lesson 9). The 800 images are split 80/20: 640 for training, 160 for validation. Only the early layers of ResNet-18 are frozen (about 2.8 million weights); layer4 and the new head are trained: 8,395,780 of 11,178,564 (lesson 10).

2. Training with early stopping

epoch  1  loss 0.2057  train acc 93.3%  |  val loss 0.2008  val acc 93.1%
epoch  2  loss 0.1677  train acc 92.8%  |  val loss 0.4024  val acc 88.1%
epoch  3  loss 0.0141  train acc 99.4%  |  val loss 0.1884  val acc 93.8%
epoch  4  loss 0.0134  train acc 99.8%  |  val loss 0.0930  val acc 95.6%
epoch  5  loss 0.0014  train acc 100.0%  |  val loss 0.1889  val acc 93.8%
epoch  6  loss 0.0038  train acc 99.8%  |  val loss 0.2354  val acc 93.8%
early stop at epoch 6: best val loss was 0.0930

3. The validation report

validation accuracy: 95.6%
               coat pullover  sneaker   tshirt   <- predicted
coat             37        1        0        2
pullover          2       38        0        0
sneaker           0        0       40        0
tshirt            0        2        0       38

4. New images

TEST accuracy: 93.8% on 400 new images
               coat pullover  sneaker   tshirt   <- predicted
coat             88       10        0        2
pullover          5       92        0        3
sneaker           0        0      100        0
tshirt            0        5        0       95

5. Saving and the demo

saved model.pt and summary.json
tshirt/0027.png: predicted tshirt (100%)
coat/0057.png: predicted coat (82%)
coat/0012.png: predicted coat (100%)
tshirt/0079.png: predicted tshirt (100%)
pullover/0040.png: predicted pullover (93%)

summary.json (what choose.py will compare later):

{
  "classes": ["coat", "pullover", "sneaker", "tshirt"],
  "epochs_run": 6,
  "val_accuracy": 0.9563,
  "test_accuracy": 0.9375,
  "seconds": 151.5
}

(In the file classes is written one name per line; it is shortened here. Our first run took 221 s — the computer was busy then; the "seconds" depends on your machine.)

Observations

Three.

1. Early stopping does not work where you "hear" it. The best validation loss (0.0930) was at epoch 4. It did not improve twice in a row, so it stopped after epoch 6 — but the model that is saved and evaluated is the epoch-4 one, not epoch 6. You can see it in "validation accuracy: 95.6 %": the same value as the epoch-4 line, while epoch 6 was 93.8 %. The weights are restored (fit with patience has done this since lesson 7).

2. Training accuracy reaches 100 %, validation does not. In epochs 4–6 training accuracy is 99.8–100 %, validation 93.8–95.6 %. That is no error: a network with millions of weights and 640 training images memorises. The practical side: training accuracy carries no information here, and the validation curve jumps around (88.1 % → 93.8 % → 95.6 %).

3. Validation and new images differ again: 95.6 % → 93.8 % (93.75 % — rounded when printed). It is the same story as in lesson 11, and the gap is similar — about 1.9 points, inside the error. But look at what changed: coat recall is 0.93 on validation (37/40) and 0.88 on new images (88/100). Of 100 new coats, 12 were classified wrongly, 10 of them as a pullover.