Three ways a multilayer network "does not work" — one loudly, two silently. All of them are my runs.

1. Loudly: the wrong in_features

RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x784 and 748x128)

MLP(in_features=748) — two digits swapped. The first layer expects 748 pixels, and the image has 784. Read the "4x784" and "748x128" as "what I gave / what I was expecting": on the left is what you passed in, on the right what the layer accepts. The same as lesson 5's 1792x28 and 784x10.

2. Silently: an lr that is too big for Adam

epoch  1  loss 0.9857  train accuracy 60.5%
epoch  5  loss 1.2960  train accuracy 47.6%

MLP(128), Adam, lr = 0.1. No error — but after five epochs the accuracy is lower than after the first, and the loss is growing. In step 4 you saw the same over 10 epochs: 45.6 %. This is the signal: if the loss after a few epochs is larger than at the start, lr is too big, not "it needs more training".

With SGD a too-big lr ended in nan in lesson 4; with Adam it is simply "bad", with no warning. A rule of thumb: lr = 0.001 is a safe start for Adam.

3. Silently: dead ReLUs

I broke it on purpose: I set the bias of all 128 hidden neurons to −100.

all hidden units dead: loss 2.3023  acc 10.3%  (ln 10 = 2.3026)

Every hidden neuron gets a very negative number, ReLU turns it into zero — for every image. The second layer receives the same row of zeros each time, so all it can learn is its own bias. The loss 2.3023 is almost exactly ln 10 = 2.3026: the network says "all 10 classes are equally likely".

Important: I built this myself. I am not saying it will happen in your code on its own — I have not recorded such a case happening naturally. But the mechanism (a neuron whose ReLU switches on for no image gets no gradient) explains why sometimes "part of a network just dies" after a big step.

What helps: look at the loss in the first epochs. If it is stuck at ln(number of classes) (10 classes → 2.3026; 2 classes → 0.6931), the network is learning nothing, and that is not "slow learning".