Three things that do not break with data — they just hand you a crooked number. We measured each of them.

1. A number that changes by itself

With --augment the training accuracy was 91.1 %. Let us repeat the measurement — the same model, the same data, three times in a row:

import torch
from data import make_loaders
from model import CNN
from train import evaluate, fit

torch.manual_seed(42)
train_loader, val_loader, classes = make_loaders("standin_data", 32, augment=True)
model = CNN(num_classes=len(classes), dropout=0.3)
fit(model, train_loader, 30, 1e-3, verbose=False)
print(f"clean validation: {evaluate(model, val_loader)[1]:.1%}")
for _ in range(3):
    print(f"augmented train loader: {evaluate(model, train_loader)[1]:.1%}")

Our output:

clean validation: 91.9%
augmented train loader: 90.5%
augmented train loader: 90.0%
augmented train loader: 88.1%

The same model, the same 640 pictures — three different accuracies (90.5 / 90.0 / 88.1), because the random augmentation is re-rolled each time. The validation number, which goes through the non-augmented val_loader, stays at 91.9 %.

What to do: never use an augmented loader as a measuring instrument. That is exactly why make_loaders builds two ImageFolder views. If you ever think of merging them "for simplicity", the validation curve will start to jump about and you will be hunting for what you broke in the network.

2. Class imbalance — accuracy lies

Take the same dataset, but leave only 20 pictures in the coat class (the others have 200). With 20 % for validation coat gets only 4 validation pictures.

This is standin_imbalanced. Run it (with the same settings as before):

python train.py --model cnn --data standin_imbalanced --epochs 30 --batch-size 32 --dropout 0.3
python train.py --model cnn --data standin_imbalanced --epochs 30 --batch-size 32 --dropout 0.3 --balance

class counts: {'coat': 20, 'pullover': 200, 'sneaker': 200, 'tshirt': 200}
cnn: 11,076 parameters, 4 classes
   ...
final: train 97.4%  val 93.5%  gap +3.8%

93.5 % — it even looks better than on the even dataset (92.5 %). Now let us see how each class separately did (a simple counter over the validation loader):

balance=False: val acc 93.5%  coat 1/4  pullover 39/40  sneaker 39/40  tshirt 37/40

Of 4 coats, one was recognised. The overall accuracy said "great" because 120 of the 124 validation pictures belong to other classes — and those did well.

With --balance:

balance=True:  val acc 92.7%  coat 2/4  pullover 37/40  sneaker 39/40  tshirt 37/40

2 of 4 coats recognised (more), while the overall accuracy dropped (93.5 → 92.7 %). How to read that:

• The overall number is the wrong instrument here. It rises or falls with how the big classes do and barely sees the small one.
• Four coats are an anecdote, not proof. 1/4 versus 2/4 is one picture. Balancing did not turn out to be a victory: the train-to-validation gap grew (+3.8 → +5.6), because the 16 training pictures of the rare class are now repeated many times.
• What is measured per class matches what a person wants to know. In lesson 11 this becomes a tool: the confusion matrix, precision / recall. In the drills below you write that counter yourself.

3. Normalisation — is it really needed?

HOLE 2 started as mean, std = 0.0, 1.0 — that is, no normalisation. That is very nearly the experiment itself: the same CNN, 10 epochs, no augmentation, validation accuracy:

	epoch 1	epoch 5	epoch 10
normalised	76.9 %	90.0 %	93.1 %
raw 0..1 pixels	81.9 %	88.1 %	89.4 %

Without normalisation the first epoch was better (81.9 vs 76.9), but at epochs 5 and 10 it was worse. One seed, 160 pictures — take it as a direction ("normalised is better in the long run"), not as a number.

Normalisation is neither magic nor mandatory. But it costs two lines, and an unnormalised dataset with different ranges of numbers (say colour photographs, 0–255 rather than 0–1) can quietly throw a network off the rails.

What next

You have a clean, checked data path and an honest baseline. In the next lesson you will no longer train a network from nothing — you will borrow one that has already been trained.