Theory
From one step to a loop
In lesson 4 you trained one neuron by hand. The loop there was:
loss.backward()
with torch.no_grad():
W -= lr * W.grad
b -= lr * b.grad
W.grad.zero_()
b.grad.zero_()
It worked: 300 steps and 81.7 % accuracy. But look at what hurt:
- Every step looked at all 10 000 images. One update = one pass over the whole set. With 60 000 images, or with your own photos, that gets slow, and sometimes it does not fit in memory at all.
- You wrote the update and the gradient clearing by hand for every tensor. Two tensors (
Wandb) — four lines. The next lesson's network has four or six, later ones dozens. Forget onezero_()and you silently get a wrong result. Wandbare two loose variables. Nothing holds them together: you cannot ask "show me all the model's numbers", and you cannot save them as one thing.
This lesson fixes that. The tools arrive because the manual way already hurts — not because "everybody does it".
What this lesson brings
| What hurt | What arrives | What it is |
|---|---|---|
| the whole set per step | a batch and DataLoader |
64 images at a time, a different 64 each time |
| "how many times have I seen the data?" | an epoch | one full pass over the whole set |
W and b as loose variables |
nn.Module and nn.Linear |
one object that holds all the weights |
hand-written -log(p) |
nn.CrossEntropyLoss |
a loss from raw scores (softmax is inside) |
W -= lr * W.grad and zero_() |
an optimizer (SGD) |
optimizer.step() and optimizer.zero_grad() |
There are no new ideas here. The arithmetic is exactly what you already did — just packaged.
The lesson climbs in steps
- Watch. Real numbers: what one batch looks like, how many numbers
nn.Linearholds, and thatoptimizer.step()really is the sameW -= lr * W.grad. You write nothing. - Fill the holes. The training loop arrives almost complete. Two lines are missing.
- Write it yourself. The model as an
nn.Module, and amain()that wires everything together. - Run it. You get your number — and see that it does not climb in a straight line.
Two things so they do not surprise you
- The accuracy you will see is measured on the very images the model trained on. It is not yet the real accuracy. In lesson 7 you will see how flattering it is; for now read it as "how training is going", not "how it will do on new images".
- Your numbers will differ from mine by a point or two. With the same seed my run reproduces exactly, but on another machine rounding can shift things. That is not a bug.