Training: Loss & Gradients

Code

300 steps: the first real accuracy

The tests pass — your training loop works. Now run it in full: 300 steps and a number that is finally no longer 10 %.

The panel shows the complete files

On the right now are the complete versions of your files:

neuron.py — cross_entropy filled in; nothing else changed;
train_scratch.py — train written, and main() added with two imports (time, load_fashion).

main() runs train with steps=300, lr=0.1, prints selected steps, and then tries the same weights on the official test set — 10,000 pictures that training never saw. test_train_scratch.py got one test for the same.

Run it

python train_scratch.py

step   0  loss 2.3207  accuracy 10.2%
step   1  loss 2.0832  accuracy 28.5%
step   2  loss 1.9216  accuracy 58.5%
step  50  loss 0.8289  accuracy 74.1%
step 100  loss 0.7066  accuracy 77.7%
step 150  loss 0.6480  accuracy 79.5%
step 200  loss 0.6110  accuracy 80.6%
step 250  loss 0.5847  accuracy 81.2%
step 300  loss 0.5646  accuracy 81.7%
accuracy on the official test set: 80.0%
done in 5.6s

(Seconds depend on the machine; ours took from 2 to 6 s.)

What these numbers say

The answers to "Guess":

After one step: 28.5 %.
After 300: 81.7 % (on the training pictures).
On the test set: 80.0 %, i.e. 1.7 points lower.

The loss fell 2.3207 → 0.5646 — to a quarter of where it started. Accuracy from 10.2 % to 81.7 %. That is not chance: in the last lesson chance moved within 3 % – 15 %.

The first jump is the biggest, afterwards — small change. From step 0 to 2 accuracy rose 10.2 → 58.5; from 50 to 300 — 74.1 → 81.7, i.e. 250 steps gave less than the second step did. Here you see for the first time diminishing returns, which will accompany you through the course.

The training-versus-test gap is small (1.7 points). This model has 7,850 numbers and 10,000 pictures; it has nothing to memorise with. That gap will become a problem a few lessons from now.

Does it repeat?

With seed = 42 the first step gives 28.5 %. Five different seeds (0–4), the same program:

seed   step 0   step 1   step 300 (training)   test
 0      3.3 %   32.7 %        81.8 %           79.9 %
 1     11.4 %   35.2 %        81.8 %           80.0 %
 2      8.0 %   27.8 %        81.9 %           80.0 %
 3     10.9 %   28.9 %        81.8 %           80.0 %
 4     11.3 %   30.5 %        81.7 %           80.0 %

Two different stories:

The start is widely scattered. The initial accuracy is from 3.3 % to 11.4 % (remember lesson 3's 3.3 – 15.2 %), after one step — from 27.8 % to 35.2 %. "One step gives 28.5 %" is this seed's answer, not a law of nature.
The end hardly changes. After 300 steps — 81.7 – 81.9 % on training and 79.9 – 80.0 % on test, the loss 0.5642 – 0.5646. This model (one layer + softmax) has one "lowest point", and from any start it arrives there.

Your numbers may differ by a point or two with another PyTorch version or processor. What matters is the shape: the loss falls from ~2.3 to ~0.56, accuracy climbs from ~10 % to ~80 %.

neuron.py MODIFY 2 lines changed

Edit this file in ./

"""neuron.py — a neuron built by hand from raw tensor math. No nn.Module."""

import torch

from fashion import CLASSES, load_fashion


def neuron(x: torch.Tensor, w: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
    """One neuron per column of w: a weighted sum of the inputs plus a bias."""
    return x @ w + b


def sigmoid(z: torch.Tensor) -> torch.Tensor:
    """Squash any number into (0, 1): 'how sure is this neuron?'"""
    return 1 / (1 + torch.exp(-z))


def softmax(z: torch.Tensor) -> torch.Tensor:
    """Turn a row of scores into probabilities that sum to 1."""
    e = torch.exp(z - z.max(dim=1, keepdim=True).values)  # subtract max: avoids overflow
    return e / e.sum(dim=1, keepdim=True)


def forward(x: torch.Tensor, W: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
    """Scores for every image, turned into probabilities.

    x is [n, 784], W is [784, 10], b is [10]; the result is [n, 10] and every
    row sums to 1. Use neuron() and softmax() from above.
    """
    return softmax(neuron(x, W, b))


def cross_entropy(probs: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
    """Average -log(probability given to the correct class). 0 = perfect."""
    correct = probs[torch.arange(len(y)), y]
    return -torch.log(correct).mean()


def accuracy(probs: torch.Tensor, y: torch.Tensor) -> float:
    """The fraction of images whose most probable class is the true label.

    probs is [n, 10], y is [n] of class numbers. Return a plain Python float
    between 0 and 1.
    """
    return (probs.argmax(dim=1) == y).float().mean().item()


def main() -> None:
    torch.manual_seed(42)
    x, y = load_fashion("train", n=10_000)

    # Ten neurons side by side (one per class) = one [784, 10] weight matrix.
    W = torch.randn(784, 10) * 0.01
    b = torch.zeros(10)

    # One neuron is just one column of it: 784 weights and 1 bias.
    z = neuron(x[0], W[:, 0], b[0])
    print(f"one neuron, one image: score={z.item():.4f} sigmoid={sigmoid(z).item():.4f}")

    probs = forward(x, W, b)
    print("probs shape:", probs.shape, "row sums to", probs[0].sum().item())
    print(f"first image: predicted {CLASSES[probs[0].argmax()]}, truth {CLASSES[y[0]]}")

    acc = accuracy(probs, y)
    print(f"untrained accuracy: {acc:.1%}   (chance = {1 / len(CLASSES):.1%})")


if __name__ == "__main__":
    main()

train_scratch.py MODIFY 47 lines changed

Edit this file in ./

"""train_scratch.py — train the 10 neurons by hand: loss, gradients, manual update."""

import time

import torch

from fashion import load_fashion
from neuron import accuracy, cross_entropy, forward


def train(x: torch.Tensor, y: torch.Tensor, steps: int = 300, lr: float = 0.1, seed: int = 42):
    """Train a fresh [784, 10] weight matrix and a [10] bias, by hand.

    Start from `torch.manual_seed(seed)`, W = torch.randn(784, 10) * 0.01 and
    b = zeros(10), both marked to track gradients. Then, `steps` times:

        1. forward pass         probs = forward(x, W, b)
        2. measure              loss = cross_entropy(probs, y)
        3. gradients            fill W.grad and b.grad from the loss
        4. update               W and b step by lr against their gradients
        5. clean up             clear W.grad and b.grad for the next round

    Return (W, b, history). history[i] is the pair (loss, accuracy) measured on
    the weights AFTER i updates, so it has steps + 1 entries: history[0] is the
    untrained model and history[steps] is the finished one.
    """
    torch.manual_seed(seed)

    # Small random weights, now marked "track gradients".
    W = (torch.randn(784, 10) * 0.01).requires_grad_()
    b = torch.zeros(10, requires_grad=True)

    history = []
    for step in range(steps + 1):
        probs = forward(x, W, b)
        loss = cross_entropy(probs, y)
        history.append((loss.item(), accuracy(probs, y)))
        if step == steps:
            break  # that was the measurement after the last update

        loss.backward()  # fills W.grad and b.grad
        with torch.no_grad():  # the update itself must not be tracked
            W -= lr * W.grad
            b -= lr * b.grad
        W.grad.zero_()  # gradients ACCUMULATE: clear them for the next step
        b.grad.zero_()

    return W, b, history


def main() -> None:
    x, y = load_fashion("train", n=10_000)

    start = time.time()
    W, b, history = train(x, y, steps=300, lr=0.1)
    seconds = time.time() - start

    for step, (loss, acc) in enumerate(history):
        if step % 50 == 0 or step < 3:
            print(f"step {step:3d}  loss {loss:.4f}  accuracy {acc:.1%}")

    # The same weights, on 10 000 images they have never seen.
    x_test, y_test = load_fashion("test")
    test_acc = accuracy(forward(x_test, W, b), y_test)
    print(f"accuracy on the official test set: {test_acc:.1%}")
    print(f"done in {seconds:.1f}s")


if __name__ == "__main__":
    main()

test_train_scratch.py MODIFY 12 lines changed

Edit this file in ./

"""test_train_scratch.py — the verdict for train_scratch.py.

Run it from this folder:   python -m unittest
"""

import math
import unittest

import torch

import train_scratch as ts
from fashion import load_fashion


class TestTrain(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        cls.x, cls.y = load_fashion("train", n=10_000)
        cls.W, cls.b, cls.history = ts.train(cls.x, cls.y, steps=100, lr=0.1)

    def test_history_has_one_entry_per_measurement(self):
        self.assertEqual(len(self.history), 101)

    def test_it_starts_at_the_shrug(self):
        loss, acc = self.history[0]
        self.assertAlmostEqual(loss, math.log(10), delta=0.05, msg=f"start loss {loss:.4f}")
        self.assertLess(acc, 0.20, f"start accuracy {acc:.1%} - the model should still be guessing")

    def test_every_loss_is_a_real_number(self):
        for step, (loss, _) in enumerate(self.history):
            self.assertTrue(math.isfinite(loss), f"loss is {loss} at step {step}")

    def test_the_loss_goes_down(self):
        first, last = self.history[0][0], self.history[-1][0]
        self.assertLess(last, first, f"loss {first:.4f} -> {last:.4f}: nothing was learned")
        self.assertLess(last, 1.0, f"loss after 100 steps is {last:.4f}, want below 1.0")

    def test_one_step_is_already_enough_to_move_the_accuracy(self):
        self.assertGreater(self.history[1][1], self.history[0][1] + 0.05)

    def test_the_accuracy_leaves_chance_far_behind(self):
        acc = self.history[-1][1]
        self.assertGreater(acc, 0.70, f"accuracy after 100 steps is {acc:.1%}, want above 70%")

    def test_it_returns_the_trained_weights(self):
        self.assertEqual(tuple(self.W.shape), (784, 10))
        self.assertEqual(tuple(self.b.shape), (10,))
        self.assertFalse(torch.equal(self.b, torch.zeros(10)), "the bias never moved")


class TestFullRun(unittest.TestCase):
    def test_300_steps_reach_the_lessons_numbers(self):
        x, y = load_fashion("train", n=10_000)
        W, b, history = ts.train(x, y, steps=300, lr=0.1)
        loss, acc = history[-1]
        self.assertLess(loss, 0.7, f"final loss {loss:.4f}")
        self.assertGreater(acc, 0.80, f"final train accuracy {acc:.1%}")
        x_test, y_test = load_fashion("test")
        test_acc = ts.accuracy(ts.forward(x_test, W, b), y_test)
        self.assertGreater(test_acc, 0.75, f"test accuracy {test_acc:.1%}")


if __name__ == "__main__":
    unittest.main()