Training: Loss & Gradients

Code

The loss function — fill in two holes

The first piece is the loss. In the last step you saw its formula for one picture; now you write it for all of them at once.

Pseudocode

cross_entropy(probs, y):                  # probs [n, 10], y [n]
    for every picture i:
        correct[i] ← probs[i, y[i]]       # the probability the model gave the CORRECT class
    return  mean( −ln(correct) )

The name is cross-entropy. The word does not concern us: it is the same "minus the logarithm of the probability the model gave the right answer", averaged.

Two things that look simple here and are not.

"Take the correct class's probability." probs is [n, 10] and y is [n] numbers. From every row you must take one column — and a different one for each row. PyTorch has "paired" indexing for that: probs[row_numbers, column_numbers] picks one element per pair. The row numbers are torch.arange(n).

"Mean" and "logarithm" — which first. You must average the losses, not the probabilities: mean(−log p), not −log(mean p). They are different things.

Your job

neuron.py (right panel) has a new function, cross_entropy, with two holes in it:

HOLE 1 — for each picture, take its correct class's probability. The placeholder "takes class zero for every picture", so the code runs but is wrong almost always.
HOLE 2 — turn them into a loss: minus the logarithm, then the mean. The placeholder simply averages the probabilities.

Everything else in the file is last lesson's neuron, softmax, forward, accuracy, unchanged.

test_neuron.py grew: a TestCrossEntropy with five tests appeared. Run it first without editing anything:

$ python -m unittest
...FFFFF...........
FAIL: test_a_shrug_costs_ln_of_the_number_of_classes (...)
AssertionError: 0.10000000149011612 != 2.302585 within 4 places (...)
FAIL: test_confidently_wrong_is_punished_hard (...)
AssertionError: 0.009999999776482582 not greater than 1.600000023841858
FAIL: test_it_averages_the_losses_not_the_probabilities (...)
AssertionError: 0.30000001192092896 != 1.4978660000000001 within 4 places (...)
FAIL: test_it_looks_at_the_probability_of_the_true_class (...)
AssertionError: 0.375 != 0.693147 within 4 places (...)
FAIL: test_perfect_predictions_cost_nothing (...)
AssertionError: 0.5 != 0.0 within 6 places (0.5 difference)
----------------------------------------------------------------------
Ran 19 tests in 0.056s

FAILED (failures=5)

Fill both holes — you should see OK.

Three of the five tests are worth a look. a_shrug_costs_ln_of_the_number_of_classes: a model that gives 0.1 to every one of ten classes must get ln 10 = 2.3026 — the same "no clue" number you saw in the last step. confidently_wrong_is_punished_hard: 0.01 for the correct class must cost more than four times as much as 0.4. perfect_predictions_cost_nothing: everything right — zero loss.

When it does not pass

All of this was tried:

-0.1 != 2.302585 — you returned -correct.mean() and forgot the logarithm. The loss must be positive; the probability is below 1, and a minus without a logarithm only flips the sign.
1.2039 != 1.4978 (only the averages_the_losses test fails) — you wrote -torch.log(correct.mean()). The logarithm of the mean is not the mean of the logarithms; this test was made precisely to catch that version.
-2.3025 != 2.302585 — you wrote torch.log(correct).mean() without the minus. A probability below 1 has a negative log — you got a negative "loss". Then training would increase the error.
IndexError: index 7 is out of bounds for dimension 0 with size 4 — probs[y] instead of probs[torch.arange(len(y)), y]. One index picks rows by class numbers, not columns.

neuron.py MODIFY 10 lines changed

Edit this file in ./

"""neuron.py — a neuron built by hand from raw tensor math. No nn.Module."""

import torch

from fashion import CLASSES, load_fashion


def neuron(x: torch.Tensor, w: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
    """One neuron per column of w: a weighted sum of the inputs plus a bias."""
    return x @ w + b


def sigmoid(z: torch.Tensor) -> torch.Tensor:
    """Squash any number into (0, 1): 'how sure is this neuron?'"""
    return 1 / (1 + torch.exp(-z))


def softmax(z: torch.Tensor) -> torch.Tensor:
    """Turn a row of scores into probabilities that sum to 1."""
    e = torch.exp(z - z.max(dim=1, keepdim=True).values)  # subtract max: avoids overflow
    return e / e.sum(dim=1, keepdim=True)


def forward(x: torch.Tensor, W: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
    """Scores for every image, turned into probabilities.

    x is [n, 784], W is [784, 10], b is [10]; the result is [n, 10] and every
    row sums to 1. Use neuron() and softmax() from above.
    """
    return softmax(neuron(x, W, b))


def cross_entropy(probs: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
    """Average -log(probability given to the correct class). 0 = perfect."""
    # HOLE 1: for every image, pick the probability the model gave to its TRUE
    # class. (This placeholder always takes class 0, right for few images.)
    correct = probs[:, 0]
    # HOLE 2: turn those probabilities into a loss: minus the log of each,
    # averaged over the images. (This placeholder is a plain average.)
    return correct.mean()


def accuracy(probs: torch.Tensor, y: torch.Tensor) -> float:
    """The fraction of images whose most probable class is the true label.

    probs is [n, 10], y is [n] of class numbers. Return a plain Python float
    between 0 and 1.
    """
    return (probs.argmax(dim=1) == y).float().mean().item()


def main() -> None:
    torch.manual_seed(42)
    x, y = load_fashion("train", n=10_000)

    # Ten neurons side by side (one per class) = one [784, 10] weight matrix.
    W = torch.randn(784, 10) * 0.01
    b = torch.zeros(10)

    # One neuron is just one column of it: 784 weights and 1 bias.
    z = neuron(x[0], W[:, 0], b[0])
    print(f"one neuron, one image: score={z.item():.4f} sigmoid={sigmoid(z).item():.4f}")

    probs = forward(x, W, b)
    print("probs shape:", probs.shape, "row sums to", probs[0].sum().item())
    print(f"first image: predicted {CLASSES[probs[0].argmax()]}, truth {CLASSES[y[0]]}")

    acc = accuracy(probs, y)
    print(f"untrained accuracy: {acc:.1%}   (chance = {1 / len(CLASSES):.1%})")


if __name__ == "__main__":
    main()

test_neuron.py MODIFY 31 lines changed

Edit this file in ./

"""test_neuron.py — the verdict for neuron.py.

Run it from this folder:   python -m unittest
"""

import unittest

import torch

import neuron as nr
from fashion import load_fashion


class TestNeuron(unittest.TestCase):
    def test_one_neuron_one_image(self):
        # 1*0.5 + 2*(-1) + 3*2 + 0.25 = 4.75
        x = torch.tensor([1.0, 2.0, 3.0])
        w = torch.tensor([0.5, -1.0, 2.0])
        b = torch.tensor(0.25)
        self.assertAlmostEqual(nr.neuron(x, w, b).item(), 4.75, places=5)

    def test_bias_is_added(self):
        x = torch.zeros(3)  # all-zero input: only the bias is left
        w = torch.ones(3)
        b = torch.tensor(0.7)
        self.assertAlmostEqual(nr.neuron(x, w, b).item(), 0.7, places=5)

    def test_many_neurons_many_images(self):
        # two images, three inputs each; two neurons side by side = [3, 2]
        x = torch.tensor([[1.0, 0.0, 0.0], [0.0, 1.0, 1.0]])
        W = torch.tensor([[1.0, 10.0], [2.0, 20.0], [3.0, 30.0]])
        b = torch.tensor([0.5, -0.5])
        out = nr.neuron(x, W, b)
        self.assertEqual(tuple(out.shape), (2, 2))
        self.assertEqual(out.tolist(), [[1.5, 9.5], [5.5, 49.5]])


class TestSigmoid(unittest.TestCase):
    def test_sigmoid_squashes_into_zero_one(self):
        z = torch.tensor([-100.0, 0.0, 100.0])
        out = nr.sigmoid(z)
        self.assertAlmostEqual(out[1].item(), 0.5, places=6)
        self.assertTrue(bool((out >= 0).all() and (out <= 1).all()))


class TestSoftmax(unittest.TestCase):
    def test_rows_sum_to_one(self):
        z = torch.tensor([[1.0, 2.0, 3.0], [0.0, 0.0, 0.0]])
        sums = nr.softmax(z).sum(dim=1)
        self.assertTrue(torch.allclose(sums, torch.ones(2)))

    def test_known_values(self):
        out = nr.softmax(torch.tensor([[1.0, 2.0, 3.0]]))
        want = torch.tensor([[0.0900, 0.2447, 0.6652]])
        self.assertTrue(torch.allclose(out, want, atol=1e-4), f"got {out.tolist()}")

    def test_big_scores_do_not_overflow(self):
        out = nr.softmax(torch.tensor([[1000.0, 1.0, 2.0]]))
        self.assertFalse(bool(torch.isnan(out).any()), f"got {out.tolist()} - overflow?")
        self.assertAlmostEqual(out[0, 0].item(), 1.0, places=5)

    def test_every_row_is_stabilised_on_its_own(self):
        # row 1 is huge, row 2 is hugely negative: one shared maximum fails row 2
        z = torch.tensor([[1000.0, 1.0, 2.0], [-1000.0, -1001.0, -1002.0]])
        out = nr.softmax(z)
        self.assertFalse(bool(torch.isnan(out).any()), f"got {out.tolist()}")
        want = torch.tensor([0.6652, 0.2447, 0.0900])
        self.assertTrue(torch.allclose(out[1], want, atol=1e-4), f"row 2 is {out[1].tolist()}")


class TestForward(unittest.TestCase):
    def test_forward_gives_probabilities_per_image(self):
        torch.manual_seed(0)
        x = torch.rand(6, 784)
        W = torch.randn(784, 10) * 0.01
        b = torch.zeros(10)
        probs = nr.forward(x, W, b)
        self.assertEqual(tuple(probs.shape), (6, 10))
        self.assertTrue(torch.allclose(probs.sum(dim=1), torch.ones(6)))

    def test_forward_picks_the_class_with_the_big_weight(self):
        # class 1 listens to input 0, class 0 listens to input 1
        W = torch.tensor([[0.0, 5.0], [5.0, 0.0]])
        b = torch.zeros(2)
        x = torch.tensor([[1.0, 0.0], [0.0, 1.0]])
        probs = nr.forward(x, W, b)
        self.assertEqual(probs.argmax(dim=1).tolist(), [1, 0])


class TestAccuracy(unittest.TestCase):
    def setUp(self):
        # four images, three classes; the model's answers are 0, 1, 2, 0
        self.probs = torch.tensor(
            [[0.7, 0.2, 0.1], [0.1, 0.8, 0.1], [0.2, 0.2, 0.6], [0.5, 0.3, 0.2]]
        )

    def test_half_right(self):
        y = torch.tensor([0, 1, 0, 1])  # right, right, wrong, wrong
        self.assertEqual(nr.accuracy(self.probs, y), 0.5)

    def test_all_right_and_all_wrong(self):
        self.assertEqual(nr.accuracy(self.probs, torch.tensor([0, 1, 2, 0])), 1.0)
        self.assertEqual(nr.accuracy(self.probs, torch.tensor([2, 2, 1, 2])), 0.0)

    def test_returns_a_plain_float(self):
        acc = nr.accuracy(self.probs, torch.tensor([0, 1, 2, 0]))
        self.assertIsInstance(acc, float)


class TestCrossEntropy(unittest.TestCase):
    def test_perfect_predictions_cost_nothing(self):
        probs = torch.tensor([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]])
        y = torch.tensor([0, 1])
        self.assertAlmostEqual(nr.cross_entropy(probs, y).item(), 0.0, places=6)

    def test_a_shrug_costs_ln_of_the_number_of_classes(self):
        probs = torch.full((4, 10), 0.1)  # ten classes, no opinion at all
        y = torch.tensor([3, 7, 0, 9])
        self.assertAlmostEqual(nr.cross_entropy(probs, y).item(), 2.302585, places=4)

    def test_it_looks_at_the_probability_of_the_true_class(self):
        probs = torch.tensor([[0.5, 0.25, 0.25], [0.25, 0.5, 0.25]])
        right = nr.cross_entropy(probs, torch.tensor([0, 1])).item()  # -ln(0.5)
        wrong = nr.cross_entropy(probs, torch.tensor([1, 0])).item()  # -ln(0.25)
        self.assertAlmostEqual(right, 0.693147, places=4)
        self.assertAlmostEqual(wrong, 1.386294, places=4)

    def test_it_averages_the_losses_not_the_probabilities(self):
        probs = torch.tensor([[0.5, 0.5], [0.1, 0.9]])
        y = torch.tensor([0, 0])  # losses: -ln(0.5) and -ln(0.1)
        want = (0.693147 + 2.302585) / 2
        self.assertAlmostEqual(nr.cross_entropy(probs, y).item(), want, places=4)

    def test_confidently_wrong_is_punished_hard(self):
        y = torch.tensor([0])
        mild = nr.cross_entropy(torch.tensor([[0.4, 0.6]]), y).item()
        harsh = nr.cross_entropy(torch.tensor([[0.01, 0.99]]), y).item()
        self.assertGreater(harsh, 4 * mild)


class TestUntrained(unittest.TestCase):
    def test_untrained_neurons_guess_at_chance(self):
        torch.manual_seed(42)
        x, y = load_fashion("train", n=10_000)
        W = torch.randn(784, 10) * 0.01
        b = torch.zeros(10)
        acc = nr.accuracy(nr.forward(x, W, b), y)
        # chance is 10%, and random weights scatter around it
        self.assertGreater(acc, 0.03, f"untrained accuracy {acc:.1%} - suspiciously low")
        self.assertLess(acc, 0.20, f"untrained accuracy {acc:.1%} - an untrained model is not THIS good")


if __name__ == "__main__":
    unittest.main()