Drills
Four tasks. All of them need what your train returns: W, b, history = train(x, y, steps=300, lr=0.1). Write them in the console or in drills.py; after each — what you should see.
Load:
x, y = load_fashion("train", n=10_000)
x_test, y_test = load_fashion("test")
1. The loss on the test set
Compute cross_entropy(forward(x_test, W, b), y_test) and compare it with the training loss from history[-1].
Check: training — 0.5646, test — about 0.6014. Accuracy: 81.7 % and 80.0 %.
Answer: which difference is more visible — in the loss or in the accuracy? Why is the loss more "sensitive"? (Hint: the loss also counts how confidently a picture is got wrong.)
2. Top three guesses
Write top3(W, b, image) — it takes one flattened picture [784] and returns three (class, probability) pairs, the largest first. Use probs.topk(3).
Run it on x_test[0].
Check: the true class is Ankle boot, and
Ankle boot 0.595
Sneaker 0.204
Sandal 0.162
The model is right, but only 59.5 % sure. All three are footwear, and that is all the model "knows": the category "shoe" is clear, which shoe — less so.
3. Check the gradient by hand
You trust backward(). Check it. With the seed = 42 weights (untrained, as in step 2) make one loss.backward().
Now compute by hand:
d = probs − one_hot(y) # [n, 10]
b_grad = d.mean(dim=0) # [10]
W_grad = x.T @ d / n # [784, 10]
one_hot is an [n, 10] table of zeros in which row i has a 1 at position y[i]. Build it yourself: torch.zeros(n, 10) and paired indexing, as in cross_entropy.
Compare with b.grad and W.grad (torch.allclose(..., atol=1e-6)).
Check: both — True. The largest difference: ~2e-09 for the biases, ~2e-08 for the weights.
That means backward() does nothing magical: the gradient of cross-entropy over softmax is simply "what the model said, minus what it should have said". You already saw the bias gradient in the table in step 2.
4. What the Trouser neuron "sees"
Take W[:, 1] (the weights of the trousers class) after training and reshape it to [28, 28]. Print it with three characters: # where the weight is > 0.03, - where it is < −0.03, . elsewhere.
Check (yours — almost the same; details depend on seed and PyTorch):
......###############.......
........#############.......
.........##########.........
........-##########....-....
......---##########.--......
.......--###########--......
......-.-##########.--......
......---##########.--.-....
......---##########.-----...
....----.#####.####.-----...
.....-.--#####-####.--.-.-..
.......--.#.#.-####.---.-...
.....-.-..###--###..--..-...
.......-..###--##..----..-..
......--.-.##--##..---....-.
.....-.---.##--.#.-------...
.....-----.##--##.-----.....
.......---.##--.#.----...--.
.-....----.##--.##.---.-.--.
......----.##---##.---..-...
.......---.##---##.--....-..
........--.##---##.--.......
........--.##--.##.--.......
........--.#.---#..--.......
........--.#.---##.-........
........--.##---##.--.......
.........-.##---##..........
.......##..##---###.#.......
At the top — a wide band (the waist), below — two vertical bands (#, the two legs) and between them a gap of negative weights (-): there a bright pixel tells the neuron "not trousers". At the sides, outside the legs, - as well. Nobody programmed this; it is all that 300 steps did.
Compare with the "templates" of class 0 (T-shirt) or 5 (Sandal).