You write nothing here. Three short experiments with real numbers (seed 42).

1. The rings

Two rings: class 0 sits on a circle of radius 0.5, class 1 on a circle of radius 1.0. A little noise is added to every point. (The file that generates them arrives in step 5; here is just what it produces.)

>>> x, y = make_circles(1000)
>>> x[:4]
tensor([[-0.1142, -0.5086],
        [-0.1912,  0.9612],
        [ 0.0389,  0.5184],
        [-0.4086, -0.2770]])
>>> y[:4]
tensor([0, 1, 0, 0])
>>> x.norm(dim=1)[y == 0].mean(), x.norm(dim=1)[y == 1].mean()      # mean radius
(0.503, 1.001)
>>> x[y == 0].mean(dim=0), x[y == 1].mean(dim=0)                    # class centres
(tensor([-0.0096,  0.0272]), tensor([-0.0285, -0.0685]))

Both classes have almost the same centre — near (0, 0). A line that separates two masses' positions has nothing to separate.

2. Two layers with nothing between them

>>> lin1, lin2 = nn.Linear(784, 128), nn.Linear(128, 10)
>>> stacked = lin2(lin1(images[:5]))           # 5 images
>>> W = lin2.weight @ lin1.weight              # [10,128] @ [128,784] -> [10,784]
>>> b = lin2.weight @ lin1.bias + lin2.bias
>>> one_layer = images[:5] @ W.T + b
>>> torch.allclose(stacked, one_layer, atol=1e-5)
True
>>> (stacked - one_layer).abs().max().item()
1.6391277313232422e-07

Two layers hold 101 770 numbers, and one layer of 7 850 gives the same answer. All those extra numbers add not one new capability: it is just one layer written longer.

3. ReLU between them

>>> z = lin1(images[0:1])                           # 128 hidden neurons, first image
>>> z[0, :8]
tensor([-0.4233, -0.2020, -0.1601,  0.1539, -0.4513, -0.6325,  0.5224, -0.3414])
>>> torch.relu(z)[0, :8]
tensor([0.0000, 0.0000, 0.0000, 0.1539, 0.0000, 0.0000, 0.5224, 0.0000])
>>> (z < 0).sum().item(), z.numel()
(58, 128)
>>> after_relu = lin2(torch.relu(lin1(images[:5])))
>>> torch.allclose(after_relu, one_layer, atol=1e-5)
False
>>> (after_relu - one_layer).abs().max().item()
0.2815892696380615

What to notice

1. Two layers with no activation are one layer with different numbers. The matrix product W2 @ W1 is simply one matrix. So 10 layers without ReLU would still be one line. Depth alone gives nothing.

2. ReLU "switches off" neurons. Of 128 hidden neurons, 58 were negative for the first image and became zero. For every image a different set of neurons turns on — the network effectively picks a different linear function for different regions.

3. With ReLU the answer is no longer the same. The difference is 0.28, not 10⁻⁷. Now the two-layer network cannot be rewritten as one layer.