Drills
Four tasks, from easy to harder. You do not need to create any files — write and
run everything in a python console or a throwaway scratch.py (in the folder
with data.py, model.py, train.py). Each one has a check.
1. The split — a rounding trap
Before running, guess: how many pictures from each class go to validation if the
class sizes are 60, 40, 10 and 2, and val_frac = 0.25?
from collections import Counter
from data import split_indices
targets = [0] * 60 + [1] * 40 + [2] * 10 + [3] * 2
train_idx, val_idx = split_indices(targets, 0.25, seed=0)
print(Counter(targets[i] for i in val_idx))
Check — you should get {0: 15, 1: 10, 2: 2, 3: 1}. Look at the third:
10 · 0.25 = 2.5, and Python's round(2.5) is 2 (it rounds to even), not 3.
And the fourth: round(0.5) is 0 — without max(1, …) that class would have no
validation picture at all.
2. Accuracy per class
Write per_class(model, loader, classes) returning a dictionary
{class: (correct, total)}. Do not forget model.eval() and torch.no_grad() —
why both?
Run it on a model trained on standin_imbalanced (see step 7; coat = 20
pictures) so that you get the same line as we did:
{'coat': (1, 4), 'pullover': (39, 40), 'sneaker': (39, 40), 'tshirt': (37, 40)}
This is your tool for the confusion matrix in lesson 11. Settings: torch.manual_seed(42),
make_loaders(root, 32), CNN(num_classes=4, dropout=0.3), fit(model, loader, 30, 1e-3, val_loader, verbose=False). If your numbers differ,
check that make_loaders(..., seed=42) and torch.manual_seed(42) are the same.
3. Does the balancer do what it promises?
Write expected_share(counts, balance) returning what share of a batch each class
takes: without balancing — in proportion to the sizes, with balancing — equal.
c = {"a": 20, "b": 200, "c": 200}
expected_share(c, balance=False) # a: 0.0476, b: 0.4762, c: 0.4762
expected_share(c, balance=True) # all 0.3333
Now check it by experiment: build a WeightedRandomSampler with weights
1 / size, draw 4,200 indices and count the classes. With
torch.manual_seed(0) we got {0: 1378, 1: 1415, 2: 1407} — each about 1,400
(= 4,200 / 3).
4. A data audit
Write audit(root, minimum=60) that walks every class subfolder of a folder and
returns a list of warnings: a class has fewer than minimum pictures; a
picture is not 28 × 28.
Check: standin_data → []. standin_imbalanced →
['coat: only 20 images (< 60)']. And a photo folder you did not run through
prepare.py will give one size (...) expected (28, 28) line per picture.
Run audit on your own data/ before every training run. It is the cheapest
mistake-catcher in this step.