Six mistakes that nearly everyone makes. Two of them are loud (an error message), four are silent (a nice number, a wrong meaning). We made and measured all of them.

1. Forgetting to replace the head

output shape: (32, 1000)  -> predicted ids: [499, 451, 869, 465, 813, 680, 680, 550]
labels: [0, 0, 0, 0, 0, 0, 0, 0]
the loss function accepts this without complaint: 13.0499

The network emits 1,000 numbers per picture, your labels are 0…3, and the loss function does not complain: it treats labels 0…3 as valid indices among 1,000 outputs. 13.05 looks like "not trained yet". This is caught by your test_the_head_has_one_output_per_class and by the first line trainable parameters: ... of 11,689,512 (it should be of 11,178,564).

2. A gray picture into a colour network

RuntimeError: Given groups=1, weight of size [64, 3, 7, 7],
expected input[2, 1, 224, 224] to have 3 channels, but got 1 channels instead

A loud error. The first conv1 expects 3 channels. ImageFolder already gives 3 channels (which is why make_loaders(rgb=True) works), but if you ever hand over a Grayscale picture by hand — you will see this.

3. A head with the wrong number of classes

IndexError: Target 3 is out of bounds.

Four folders, but build_resnet(3). Label 3 where there are only outputs 0, 1, 2. A loud error — but only when a batch happens to contain that class. If there were 4 folders and a head of 5, there would be no error: the extra output is quietly never trained (like lesson 9's num_classes=10).

4. Pictures not resized (silent)

The same frozen network, the same settings (--epochs 3 --lr 0.01), only the pictures are left at 28 × 28 (--size 28) instead of 224:

	validation
224 × 224	90.0 %
28 × 28, not resized	74.4 %

No error — the network works at any size. But it was trained to see things across 224 pixels, and a 28-pixel "animal" occupies only a few of its "eye" cells. The speed is tempting (9 s instead of 93 s), but you pay 15.6 points for it.

5. Everything unlocked, learning rate too big (silent)

The same set-up, but with nothing frozen (in your build_resnet change p.requires_grad = False to True), lr = 0.01:

	train	validation	time
frozen, lr 0.01	92.8 %	90.0 %	93 s
nothing frozen, lr 0.01	69.1 %	63.7 %	181 s

A big learning rate breaks the borrowed features while the new head is still random: the network "forgets" what it learned on a million photographs. That is why --finetune uses lr 0.001, not 0.01, and why layer4 is unlocked only once the head can already do something.

6. "Frozen" does not mean "stops changing" (silent)

requires_grad = False stops the weights from being updated. But BatchNorm layers also have running averages, which are not parameters — they are updated on every batch when the network is in train() mode. Check it yourself:

bn1.weight.requires_grad: False
running_mean changed by (max abs): 0.0128      # in train() mode
in eval mode, changed by: 0.0000               # in eval() mode

So even in a "frozen" ResNet, how it normalises your non-ImageNet data keeps changing. Is that good or bad? We measured: keeping the BatchNorm layers in eval() mode, validation after 3 epochs was 92.5 % instead of 90.0 %. But that is one seed, 160 pictures, and in step 5 we saw that merely changing one evaluate call moves the result by 1.3 points. So: the principle is real, the benefit unproven. If you train with very small batch sizes, it is worth trying.

Take away

• Print trainable parameters: X of Y first. It catches mistakes 1 and 5 at once.
• Prepare pictures the way the network saw them. 4 and part of 2.
• Freeze until the head has learned something, only then unlock. 5.
• Do not trust a single seed. 6 and the "noise" from step 5.