Nothing to write here. Here you watch one training step from the inside — with the same random weights as in lesson 3 (seed 42) and 10,000 pictures. All numbers are real.

1. Loss: the error for one picture

The first picture is an ankle boot. The untrained model gave it the probability:

p(correct class) = 0.1388
loss = −ln(0.1388) = 1.9747

The loss for one picture is minus the natural logarithm of the probability the model gave the correct class. A few points on the curve to get the shape:

p(correct class)     loss
0.90                 0.105      almost right — almost nothing
0.50                 0.693
0.10                 2.303      = ln 10: "no clue at all", ten classes
0.01                 4.605      confidently wrong — painfully

Two things. The loss is never negative and is 0 only when the probability is 1. And the more confidently wrong, the steeper it grows: from 0.1 to 0.01 the cost doubles, although both probabilities look "small".

The loss over all 10,000 is the mean of such numbers:

step 0   loss 2.3207   accuracy 10.2%

2.3207 is very close to ln 10 = 2.3026. The untrained model has exactly the loss of a "random guess among ten classes".

2. Gradient: which way to push

One .backward() fills two tensors:

W.grad   shape [784, 10]     7,840 numbers
b.grad   shape [10]          10 numbers

The ten bias-gradient numbers can be explained at a glance. Here they are beside the model's "average confidence" and the true share of the class in the data:

class         avg. prob.     share in data     b.grad
T-shirt         0.0771          0.0942          -0.0171
Trouser         0.1014          0.1027          -0.0013
Pullover        0.1025          0.1016          +0.0009
Dress           0.0990          0.1019          -0.0029
Coat            0.0866          0.0974          -0.0108
Sandal          0.0958          0.0989          -0.0031
Shirt           0.0995          0.1021          -0.0026
Sneaker         0.0936          0.1022          -0.0086
Bag             0.1217          0.0990          +0.0227
Ankle boot      0.1229          0.1000          +0.0229

The bias gradient is exactly that difference: average probability minus share in the data. For T-shirts the model gives 7.7 % on average while they are 9.4 % — the gradient is negative: increasing this bias would lower the loss. For bags — the opposite: the model is overconfident (12.2 % vs 9.9 %), the gradient is positive. As in lesson 2, a gradient is advice: the sign gives the direction, the size says how much it matters.

The weight gradient means the same, only for each of the 784 pixels separately.

3. The update

speed = 0.1. For every number:

new = old − 0.1 · gradient

For the biases that means: T-shirt +0.0017, bag −0.0023. Small steps: the largest change of a single weight in the whole [784, 10] tensor is 0.0059.

4. The new loss and accuracy

step 0   loss 2.3207   accuracy 10.2%
step 1   loss 2.0832   accuracy 28.5%
step 2   loss 1.9216   accuracy 58.5%

After one step — 28.5 %. The answer to question 1 of "Guess": a lot more than you could expect from one small correction. After two — 58.5 %.

What changed? Before the step the model nearly always said "bag" or "ankle boot". After one step its distribution of answers:

T-shirt 4    Trouser 623   Pullover 1457   Dress 52    Coat 3915
Sandal 0     Shirt 1151    Sneaker 80      Bag 1133    Ankle boot 1585

It spread itself across several classes — and started hitting more often.

Three observations

1. The loss moves smoothly, accuracy in jumps. 2.3207 → 2.0832 → 1.9216, while accuracy goes 10.2 → 28.5 → 58.5. The first steps seem to fix the crudest error — overconfidence in two classes — which is why accuracy rises sharply.

2. The first steps do most of the work. In the table in step 5 you will see that from step 50 to step 300 accuracy rises only 74.1 → 81.7.

3. A small step, but 7,850 at once. Every weight moves by thousandths, but they all move together, in the direction that lowers the same number. Learning is many small coordinated changes.

The same in words

repeat:
    1. forward      probs ← forward(x, W, b)
    2. loss         loss ← cross_entropy(probs, y)
    3. gradients    loss.backward()             → W.grad, b.grad
    4. update       W ← W − speed · W.grad      (and b the same)
    5. clean up     W.grad ← 0, b.grad ← 0      (otherwise they pile up — lesson 2's fifth breakage)

The fifth line is lesson 2's silent failure, which you will remember again.

What next

In the next step you write the first piece — the loss function. You have just seen its formula.