In lesson 3 you got a number to beat: 10.2 %. In this lesson you beat it for the first time — and see why three ideas are enough for that.

Three ideas

1. Loss — the size of the error, as something you can compute. Accuracy says "right / wrong", and that is too coarse: if the probability given to the correct class rose from 0.14 to 0.16, accuracy may not change at all, even though the model took a step in the right direction. Loss sees such steps.

2. Gradient — "which way to push". The loss depends on 7,850 numbers (784 × 10 weights + 10 biases). The gradient tells each one: if you increase this number, the loss will rise or fall, and how steeply. In lesson 2 you saw it for one w; here — 7,850 at once, and the same .backward() is enough.

3. Update — a small step against the gradient. weight -= speed * gradient. The "speed" is called the learning rate, and it is the most important number in this lesson.

Repeat that hundreds of times — and you get what we called in lesson 1 "guess → compare → adjust → repeat". There is nothing more.

Why loss, and not accuracy

Accuracy is a staircase: the number 10.2 % does not change until at least one picture's answer changes. Loss is a smooth slope: every small improvement shows up in it. You can walk down a slope, not a staircase.

We will keep computing accuracy — it matters to us humans. But the model learns from the loss.

The lesson climbs in steps

1. Watch. One training step, with real numbers: loss, gradient, update, new accuracy.
2. Fill the holes. cross_entropy — the loss function, with two empty places.
3. Write it yourself. train — the whole training loop, from creating the weights to the end.
4. Run it. 300 steps — and an accuracy that is finally no longer 10 %.

After that two steps where you break the training: forgetting to clear the gradients, and a learning rate that is too big or too small.

Guess before reading on

1. After one step — what will the accuracy be? (Right now it is 10.2 %.)
2. After 300 steps?
3. How many percentage points lower will accuracy be on pictures not seen in training (the test set)?

The answers are in steps 2 and 5.