Drills

Drills

Four tasks, from easy to harder. You do not need to create any files — write and run everything in a python console or a throwaway scratch.py (in the folder with data.py, model.py, train.py). Each one has a check.


1. The split — a rounding trap

Before running, guess: how many pictures from each class go to validation if the class sizes are 60, 40, 10 and 2, and val_frac = 0.25?

from collections import Counter
from data import split_indices

targets = [0] * 60 + [1] * 40 + [2] * 10 + [3] * 2
train_idx, val_idx = split_indices(targets, 0.25, seed=0)
print(Counter(targets[i] for i in val_idx))

Check — you should get {0: 15, 1: 10, 2: 2, 3: 1}. Look at the third: 10 · 0.25 = 2.5, and Python's round(2.5) is 2 (it rounds to even), not 3. And the fourth: round(0.5) is 0 — without max(1, …) that class would have no validation picture at all.


2. Accuracy per class

Write per_class(model, loader, classes) returning a dictionary {class: (correct, total)}. Do not forget model.eval() and torch.no_grad() — why both?

Run it on a model trained on standin_imbalanced (see step 7; coat = 20 pictures) so that you get the same line as we did:

{'coat': (1, 4), 'pullover': (39, 40), 'sneaker': (39, 40), 'tshirt': (37, 40)}

This is your tool for the confusion matrix in lesson 11. Settings: torch.manual_seed(42), make_loaders(root, 32), CNN(num_classes=4, dropout=0.3), fit(model, loader, 30, 1e-3, val_loader, verbose=False). If your numbers differ, check that make_loaders(..., seed=42) and torch.manual_seed(42) are the same.


3. Does the balancer do what it promises?

Write expected_share(counts, balance) returning what share of a batch each class takes: without balancing — in proportion to the sizes, with balancing — equal.

c = {"a": 20, "b": 200, "c": 200}
expected_share(c, balance=False)  # a: 0.0476, b: 0.4762, c: 0.4762
expected_share(c, balance=True)   # all 0.3333

Now check it by experiment: build a WeightedRandomSampler with weights 1 / size, draw 4,200 indices and count the classes. With torch.manual_seed(0) we got {0: 1378, 1: 1415, 2: 1407} — each about 1,400 (= 4,200 / 3).


4. A data audit

Write audit(root, minimum=60) that walks every class subfolder of a folder and returns a list of warnings: a class has fewer than minimum pictures; a picture is not 28 × 28.

Check: standin_data → []. standin_imbalanced → ['coat: only 20 images (< 60)']. And a photo folder you did not run through prepare.py will give one size (...) expected (28, 28) line per picture.

Run audit on your own data/ before every training run. It is the cheapest mistake-catcher in this step.