The training loop is four lines, and in four lines you can go wrong in ways that produce no error message at all. Here are mistakes I made myself and recorded. All with 5 epochs, the same data, seed 42, lr=0.1, batch 64.

The reference point

epoch 1  loss 0.8366  train accuracy 74.4%
epoch 2  loss 0.5810  train accuracy 81.2%
epoch 3  loss 0.6405  train accuracy 77.8%
epoch 4  loss 0.5197  train accuracy 82.0%
epoch 5  loss 0.5514  train accuracy 80.0%

Why 74.4 % and not 74.0 % as in the previous step? This run does not call evaluate before training. Creating a DataLoader iterator itself consumes the random number generator, so one extra evaluate changes the shuffle order of every later epoch. Same seed, same code, one extra call — different numbers. That is not a bug; it is the nature of randomness.

1. You forgot zero_grad() — the loss grows, the accuracy does not

epoch 1  loss 4.5791  train accuracy 78.6%
epoch 2  loss 10.6960  train accuracy 74.9%
epoch 3  loss 12.7572  train accuracy 78.6%
epoch 4  loss 16.1850  train accuracy 80.6%
epoch 5  loss 29.6789  train accuracy 76.9%

The loss grows from 4.6 to 29.7. Meanwhile accuracy sits at 75–80 %, almost the same as the reference. Loss and accuracy disagree. If you only looked at the accuracy you would notice nothing.

The cause: gradients pile up (that is why lesson 4 had zero_()), so the steps get bigger and bigger. The argmax is still "roughly" right, but the scores get distorted. The rule: watch both numbers.

2. Double softmax — it learns worse, silently

epoch 1  loss 1.5961  train accuracy 61.8%
epoch 2  loss 1.6124  train accuracy 62.3%
epoch 3  loss 1.1816  train accuracy 67.9%
epoch 4  loss 1.1269  train accuracy 72.7%
epoch 5  loss 1.1444  train accuracy 74.7%

nn.CrossEntropyLoss applies softmax itself. If forward already did — that is softmax twice: the scores are squashed into (0, 1) and the gradients weaken. After 5 epochs: 74.7 % instead of 80.0 %. No error; the model just struggles.

3. A learning rate that is too big

lr = 5.0:

epoch 1  loss 18.9563  train accuracy 59.4%
epoch 2  loss 13.6333  train accuracy 69.8%
epoch 3  loss 22.7658  train accuracy 61.2%
epoch 4  loss 4.3661  train accuracy 84.2%
epoch 5  loss 29.9741  train accuracy 67.0%

The loss bounces between 4 and 30, the accuracy between 59 and 84 %. Epoch 4 is even prettier than the reference, but there is no order to it — lr is still too big. Do not trust the result of a single epoch.

4. Batch size

batch	updates per epoch	what happened
64 (reference)	157	2.6 s for 5 epochs
1	10 000	one epoch took 5.1 s; 76.5 %
10 000 (the whole set)	1	only 5 updates in 5 epochs: 63.8 %

Batch 1 is slow (a separate Python loop turn per image). Batch 10 000 is lesson 4's "everything per step" again: fast per update, but only one per epoch. 5 epochs = 5 steps, and the accuracy after them is 63.8 % and still climbing (24.2 → 51.8 → 58.7 → 62.1 → 63.8).

The seconds in this table come from a busy machine and are unreliable by ±2×. Compare the ratio, not the absolute numbers.

5. Two mistakes that at least shout

RuntimeError: mat1 and mat2 shapes cannot be multiplied (1792x28 and 784x10)

The images were not flattened: [N, 28, 28] instead of [N, 784]. 1792 = 28 · 64 — the batch got merged with one of the image's dimensions.

RuntimeError: expected target dtype to be Long or Byte, but got Float

Class labels must be integers (int64), not float32.

What to take from it

Four markers of mistakes with no error message: the loss grows but the accuracy does not (1), accuracy worse than it should be (2), a bouncing loss (3), too few updates (4). If something "works but feels off" — look at the loss, not just the accuracy.