Theory

From one step to a loop

In lesson 4 you trained one neuron by hand. The loop there was:

loss.backward()
with torch.no_grad():
    W -= lr * W.grad
    b -= lr * b.grad
W.grad.zero_()
b.grad.zero_()

It worked: 300 steps and 81.7 % accuracy. But look at what hurt:

  1. Every step looked at all 10 000 images. One update = one pass over the whole set. With 60 000 images, or with your own photos, that gets slow, and sometimes it does not fit in memory at all.
  2. You wrote the update and the gradient clearing by hand for every tensor. Two tensors (W and b) — four lines. The next lesson's network has four or six, later ones dozens. Forget one zero_() and you silently get a wrong result.
  3. W and b are two loose variables. Nothing holds them together: you cannot ask "show me all the model's numbers", and you cannot save them as one thing.

This lesson fixes that. The tools arrive because the manual way already hurts — not because "everybody does it".

What this lesson brings

What hurt What arrives What it is
the whole set per step a batch and DataLoader 64 images at a time, a different 64 each time
"how many times have I seen the data?" an epoch one full pass over the whole set
W and b as loose variables nn.Module and nn.Linear one object that holds all the weights
hand-written -log(p) nn.CrossEntropyLoss a loss from raw scores (softmax is inside)
W -= lr * W.grad and zero_() an optimizer (SGD) optimizer.step() and optimizer.zero_grad()

There are no new ideas here. The arithmetic is exactly what you already did — just packaged.

The lesson climbs in steps

  1. Watch. Real numbers: what one batch looks like, how many numbers nn.Linear holds, and that optimizer.step() really is the same W -= lr * W.grad. You write nothing.
  2. Fill the holes. The training loop arrives almost complete. Two lines are missing.
  3. Write it yourself. The model as an nn.Module, and a main() that wires everything together.
  4. Run it. You get your number — and see that it does not climb in a straight line.

Two things so they do not surprise you

  • The accuracy you will see is measured on the very images the model trained on. It is not yet the real accuracy. In lesson 7 you will see how flattering it is; for now read it as "how training is going", not "how it will do on new images".
  • Your numbers will differ from mine by a point or two. With the same seed my run reproduces exactly, but on another machine rounding can shift things. That is not a bug.