A Single Neuron (from Scratch)

Code

The first number: an untrained neuron's accuracy

The tests pass — your neuron works. Now run it on real data and find out the number the rest of the course has to beat.

The panel shows the complete file

neuron.py on the right is now the complete version of your file: holes filled, forward and accuracy written. Compare it with yours. Two things were added:

from fashion import CLASSES, load_fashion at the top — fashion.py from lesson 2;
main() at the bottom — what you saw in step 2, only from your code.

test_neuron.py got one test: untrained neurons on 10,000 pictures must land in 3 % – 20 %. That is not "hit 10 %" — you will see why below.

Run it

python neuron.py

one neuron, one image: score=-0.2554 sigmoid=0.4365
probs shape: torch.Size([10000, 10]) row sums to 1.0
first image: predicted Bag, truth Ankle boot
untrained accuracy: 10.2%   (chance = 10.0%)

The same four lines you saw in step 2 — now from your code. The last is the number:

10.2 %. This is what a model with no training reaches on its own. Everything above it will be your work.

We learn on ten classes, each about a tenth of the set (942 – 1,027 out of 10,000), so a random guess has 10 % accuracy. You will call this number the baseline: the answer to "did my model learn anything at all".

But "random" is not one number

Change the seed of the random weights and run it ten times:

3.3%  11.4%  8.0%  10.9%  11.3%  9.2%  15.2%  9.0%  8.8%  6.5%   | mean 9.4%

The same 10,000 pictures, the same code, only the initial weights differ. The result ranged from 3.3 % to 15.2 %.

So:

10.2 % is not "the right answer", but one of many numbers drawn that day. On your machine, with another PyTorch version or seed, you may get 8 % or 12 %, and that is just as normal;
if training lifts accuracy from 10 % to 10.8 %, that means nothing — chance alone is enough for that. We need jumps.

That is why the test limits are wide (3 % – 20 %): it checks that the model is not wrong-headed and not too good, not "did it hit the same number".

In this course we set torch.manual_seed(42) everywhere so that your script gives the same number every time. But the same seed on a different machine or PyTorch version may give a different result — the numbers quoted in this course were obtained with PyTorch 2.14, CPU.

One more thing worth seeing

10.2 % is an overall number. Look at what it is made of. Per class (from the step 2 distribution — it guessed almost only bag and boot):

class        correct
Bag          30.3 %
Ankle boot   70.6 %
all others    0.0 % – 0.9 %

The two "favourite" answers are right for 30 % and 71 % of their classes' pictures, respectively, and the other eight classes — almost never. But that is not recognition yet: the model says "ankle boot" 5,056 times and is right only 706 times (out of 1,000 real boots). It simply says the same thing almost every time, and sometimes hits.

You will see this again in the next step.

neuron.py MODIFY 30 lines changed

Edit this file in ./

"""neuron.py — a neuron built by hand from raw tensor math. No nn.Module."""

import torch

from fashion import CLASSES, load_fashion


def neuron(x: torch.Tensor, w: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
    """One neuron per column of w: a weighted sum of the inputs plus a bias."""
    return x @ w + b


def sigmoid(z: torch.Tensor) -> torch.Tensor:
    """Squash any number into (0, 1): 'how sure is this neuron?'"""
    return 1 / (1 + torch.exp(-z))


def softmax(z: torch.Tensor) -> torch.Tensor:
    """Turn a row of scores into probabilities that sum to 1."""
    e = torch.exp(z - z.max(dim=1, keepdim=True).values)  # subtract max: avoids overflow
    return e / e.sum(dim=1, keepdim=True)


def forward(x: torch.Tensor, W: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
    """Scores for every image, turned into probabilities.

    x is [n, 784], W is [784, 10], b is [10]; the result is [n, 10] and every
    row sums to 1. Use neuron() and softmax() from above.
    """
    return softmax(neuron(x, W, b))


def accuracy(probs: torch.Tensor, y: torch.Tensor) -> float:
    """The fraction of images whose most probable class is the true label.

    probs is [n, 10], y is [n] of class numbers. Return a plain Python float
    between 0 and 1.
    """
    return (probs.argmax(dim=1) == y).float().mean().item()


def main() -> None:
    torch.manual_seed(42)
    x, y = load_fashion("train", n=10_000)

    # Ten neurons side by side (one per class) = one [784, 10] weight matrix.
    W = torch.randn(784, 10) * 0.01
    b = torch.zeros(10)

    # One neuron is just one column of it: 784 weights and 1 bias.
    z = neuron(x[0], W[:, 0], b[0])
    print(f"one neuron, one image: score={z.item():.4f} sigmoid={sigmoid(z).item():.4f}")

    probs = forward(x, W, b)
    print("probs shape:", probs.shape, "row sums to", probs[0].sum().item())
    print(f"first image: predicted {CLASSES[probs[0].argmax()]}, truth {CLASSES[y[0]]}")

    acc = accuracy(probs, y)
    print(f"untrained accuracy: {acc:.1%}   (chance = {1 / len(CLASSES):.1%})")


if __name__ == "__main__":
    main()

test_neuron.py MODIFY 13 lines changed

Edit this file in ./

"""test_neuron.py — the verdict for neuron.py.

Run it from this folder:   python -m unittest
"""

import unittest

import torch

import neuron as nr
from fashion import load_fashion


class TestNeuron(unittest.TestCase):
    def test_one_neuron_one_image(self):
        # 1*0.5 + 2*(-1) + 3*2 + 0.25 = 4.75
        x = torch.tensor([1.0, 2.0, 3.0])
        w = torch.tensor([0.5, -1.0, 2.0])
        b = torch.tensor(0.25)
        self.assertAlmostEqual(nr.neuron(x, w, b).item(), 4.75, places=5)

    def test_bias_is_added(self):
        x = torch.zeros(3)  # all-zero input: only the bias is left
        w = torch.ones(3)
        b = torch.tensor(0.7)
        self.assertAlmostEqual(nr.neuron(x, w, b).item(), 0.7, places=5)

    def test_many_neurons_many_images(self):
        # two images, three inputs each; two neurons side by side = [3, 2]
        x = torch.tensor([[1.0, 0.0, 0.0], [0.0, 1.0, 1.0]])
        W = torch.tensor([[1.0, 10.0], [2.0, 20.0], [3.0, 30.0]])
        b = torch.tensor([0.5, -0.5])
        out = nr.neuron(x, W, b)
        self.assertEqual(tuple(out.shape), (2, 2))
        self.assertEqual(out.tolist(), [[1.5, 9.5], [5.5, 49.5]])


class TestSigmoid(unittest.TestCase):
    def test_sigmoid_squashes_into_zero_one(self):
        z = torch.tensor([-100.0, 0.0, 100.0])
        out = nr.sigmoid(z)
        self.assertAlmostEqual(out[1].item(), 0.5, places=6)
        self.assertTrue(bool((out >= 0).all() and (out <= 1).all()))


class TestSoftmax(unittest.TestCase):
    def test_rows_sum_to_one(self):
        z = torch.tensor([[1.0, 2.0, 3.0], [0.0, 0.0, 0.0]])
        sums = nr.softmax(z).sum(dim=1)
        self.assertTrue(torch.allclose(sums, torch.ones(2)))

    def test_known_values(self):
        out = nr.softmax(torch.tensor([[1.0, 2.0, 3.0]]))
        want = torch.tensor([[0.0900, 0.2447, 0.6652]])
        self.assertTrue(torch.allclose(out, want, atol=1e-4), f"got {out.tolist()}")

    def test_big_scores_do_not_overflow(self):
        out = nr.softmax(torch.tensor([[1000.0, 1.0, 2.0]]))
        self.assertFalse(bool(torch.isnan(out).any()), f"got {out.tolist()} - overflow?")
        self.assertAlmostEqual(out[0, 0].item(), 1.0, places=5)

    def test_every_row_is_stabilised_on_its_own(self):
        # row 1 is huge, row 2 is hugely negative: one shared maximum fails row 2
        z = torch.tensor([[1000.0, 1.0, 2.0], [-1000.0, -1001.0, -1002.0]])
        out = nr.softmax(z)
        self.assertFalse(bool(torch.isnan(out).any()), f"got {out.tolist()}")
        want = torch.tensor([0.6652, 0.2447, 0.0900])
        self.assertTrue(torch.allclose(out[1], want, atol=1e-4), f"row 2 is {out[1].tolist()}")


class TestForward(unittest.TestCase):
    def test_forward_gives_probabilities_per_image(self):
        torch.manual_seed(0)
        x = torch.rand(6, 784)
        W = torch.randn(784, 10) * 0.01
        b = torch.zeros(10)
        probs = nr.forward(x, W, b)
        self.assertEqual(tuple(probs.shape), (6, 10))
        self.assertTrue(torch.allclose(probs.sum(dim=1), torch.ones(6)))

    def test_forward_picks_the_class_with_the_big_weight(self):
        # class 1 listens to input 0, class 0 listens to input 1
        W = torch.tensor([[0.0, 5.0], [5.0, 0.0]])
        b = torch.zeros(2)
        x = torch.tensor([[1.0, 0.0], [0.0, 1.0]])
        probs = nr.forward(x, W, b)
        self.assertEqual(probs.argmax(dim=1).tolist(), [1, 0])


class TestAccuracy(unittest.TestCase):
    def setUp(self):
        # four images, three classes; the model's answers are 0, 1, 2, 0
        self.probs = torch.tensor(
            [[0.7, 0.2, 0.1], [0.1, 0.8, 0.1], [0.2, 0.2, 0.6], [0.5, 0.3, 0.2]]
        )

    def test_half_right(self):
        y = torch.tensor([0, 1, 0, 1])  # right, right, wrong, wrong
        self.assertEqual(nr.accuracy(self.probs, y), 0.5)

    def test_all_right_and_all_wrong(self):
        self.assertEqual(nr.accuracy(self.probs, torch.tensor([0, 1, 2, 0])), 1.0)
        self.assertEqual(nr.accuracy(self.probs, torch.tensor([2, 2, 1, 2])), 0.0)

    def test_returns_a_plain_float(self):
        acc = nr.accuracy(self.probs, torch.tensor([0, 1, 2, 0]))
        self.assertIsInstance(acc, float)


class TestUntrained(unittest.TestCase):
    def test_untrained_neurons_guess_at_chance(self):
        torch.manual_seed(42)
        x, y = load_fashion("train", n=10_000)
        W = torch.randn(784, 10) * 0.01
        b = torch.zeros(10)
        acc = nr.accuracy(nr.forward(x, W, b), y)
        # chance is 10%, and random weights scatter around it
        self.assertGreater(acc, 0.03, f"untrained accuracy {acc:.1%} - suspiciously low")
        self.assertLess(acc, 0.20, f"untrained accuracy {acc:.1%} - an untrained model is not THIS good")


if __name__ == "__main__":
    unittest.main()