That "more layers" automatically means "better" is the most common wrong hunch about neural networks. Here are my runs that test it.

All runs: Adam lr = 0.001, 10 epochs, batch 64, 10 000 images, training accuracy, shuffle seed 0 (so results are comparable with each other).

1. Depth without an activation is not depth

Three layers 784 → 128 → 64 → 10, but without ReLU between them — compared with the same network with ReLU and with a single layer:

no ReLU    109,386 params   train loss 0.4059   acc 85.5%
with ReLU  109,386 params   train loss 0.3280   acc 87.9%
linear       7,850 params   train loss 0.4536   acc 84.3%

109 386 numbers without ReLU give 85.5 % — only 1.2 points above the single layer with 7 850 numbers. Those extra 100 000 numbers did almost nothing because (as you saw in step 2) together they form one linear map. Why not exactly the same? The function class is the same, but optimising three layers goes differently, so the accuracy is not identical. Adding ReLU: +2.4 points and a much lower loss (0.328 vs 0.406).

2. Wider or deeper?

hidden            params  train loss  train acc
()                 7,850      0.4536      84.3%
(16,)             12,730      0.4555      83.4%
(128,)           101,770      0.3405      87.9%
(512,)           407,050      0.2721      90.0%
(128, 64)        109,386      0.3280      87.9%
(128, 128, 128)  134,794      0.3138      87.9%
(64, 64, 64, 64, 64, 64)  71,690      0.3990      85.1%

What to read:

• (16,) — 83.4 %, worse than no hidden layer at all (84.3 %). 16 neurons are probably too few to pass the information of 784 pixels through. A layer and a ReLU by themselves guarantee nothing.
• Width (128 → 512) still helps: 87.9 → 90.0 %, loss 0.34 → 0.27.
• Depth does not help: (128, 64) and (128, 128, 128) are both 87.9 % — the same as the single (128,) layer.
• Six narrow layers (64 × 6) — 85.1 %, less than one wide layer, although it has 71 690 numbers.

In this task, with these settings, width beats depth. That is not a general rule (later you will see that images suit a different kind of network), it is the result for this dataset.

3. What to take away

• Depth without an activation is empty.
• More parameters does not mean better: (64,)×6 with 71 690 numbers loses to (128,) with 101 770.
• Two runs with the same accuracy (87.9 %) can have different loss (0.3405 vs 0.3138). When accuracy is tied, the loss still says something.

All these numbers are training accuracy and a single run; they are a direction, not a proof. In lesson 7 you will see what happens when a network becomes too big for its data.