All along we used lr = 0.1. Now see what happens when you change this single number — and why it matters so much.

All numbers here are from your train with steps=300, seed=42, 10,000 pictures; only lr changes.

Too big: lr = 1.0

step   0  loss 2.3207  accuracy 10.2%
step   1  loss 2.4480  accuracy 18.1%
step   2  loss inf     accuracy 48.1%
step  10  loss nan     accuracy 9.4%
step 100  loss nan     accuracy 9.4%

The first step overshoots the minimum: the loss did not fall, it rose (2.32 → 2.45). The second — inf (infinity): for some picture the correct class's probability became 0, and −ln(0) is infinity. By step 10 — nan, and there is no recovering.

Notice step 2: accuracy 48.1 %, while the loss is infinite. It is the same "two indicators disagree" that you saw when zero_() was forgotten.

Too small: lr = 0.0001

step   0  loss 2.3207  accuracy 10.2%
step   1  loss 2.3204  accuracy 10.2%
step   2  loss 2.3201  accuracy 10.2%
step  10  loss 2.3177  accuracy 10.4%
step 100  loss 2.2924  accuracy 12.2%

A hundred steps — and the loss fell from 2.3207 to 2.2924. Nothing broke. The learning "looks stuck", although it is simply going very slowly (after 300 steps — 2.2413 / 21.4 %). There is no loud message: you feel it only by the clock.

The whole scale

300 steps, each rate separately:

lr	final loss	accuracy
0.0001	2.2413	21.4 %
0.01	0.9486	70.3 %
0.05	0.6482	79.4 %
0.1	0.5646	81.7 %
0.2	0.5414	81.0 %
0.5	1.1620	80.2 %
1.0	nan	9.4 %

The lesson of this table is not "take 0.1".

1. A wide good interval. From 0.05 to 0.2 — all three results lie between 79.4 % and 81.7 %. The exact number is not very important here; the order of magnitude is: 0.0001 and 1.0 are outside the interval on both sides.

2. Accuracy is not the same as loss. lr = 0.2 gives a lower loss (0.5414 vs 0.5646), but a lower accuracy (81.0 % vs 81.7 %). The difference is one percent, which is the edge of chance. But lr = 0.5 shows the same more clearly: accuracy 80.2 %, while the loss is 1.16 — twice as bad as at 0.1.

3. A large rate wanders. At lr = 0.5 the loss jumped up to 6.72 in the first steps, and the accuracy reached 83.0 % at step 281 — and finished at 80.2 %. If you took the best point for a report, you would get a prettier number than the truth. For quoting, take the final one.

How to find the rate

There is no formula. In practice:

• start with 0.1 (or 0.01 if the loss jumps right away);
• if the loss is nan or jumping — lower it 3–10 times;
• if the loss falls but barely moves after a hundred steps — raise it;
• watch the loss, not only the accuracy.

This number stays with you for the whole course: other ways to compute the step will appear, but the learning rate never goes away.