Multilayer Network — Programuoki

Code

Run it: rings and clothes

The tests pass. Now — two runs that answer the question this lesson exists for.

1. The rings — here the leap is real

This step creates circles.py: the rings generator and a run with two models — one layer and an MLP with 16 hidden neurons. It is not your task, so the file is complete.

$ python circles.py
--- linear: 6 parameters
epoch  1  loss 0.7084  train accuracy 44.4%
epoch  2  loss 0.6921  train accuracy 51.5%
...
epoch 19  loss 0.6894  train accuracy 52.8%
epoch 20  loss 0.6895  train accuracy 54.3%
linear final train accuracy: 54.3%
--- mlp: 82 parameters
epoch  1  loss 0.6181  train accuracy 77.8%
epoch  2  loss 0.5022  train accuracy 97.4%
epoch  3  loss 0.3604  train accuracy 99.5%
epoch  4  loss 0.2276  train accuracy 100.0%
...
epoch 20  loss 0.0065  train accuracy 100.0%
mlp final train accuracy: 100.0%

(... — lines I left out; the real output has them.)

54.3 % → 100.0 %. One layer — 6 numbers — gets stuck with a loss of 0.689, which is almost ln 2 = 0.693 — "guess by coin toss". An MLP with 82 numbers stops making mistakes after four epochs. No amount of noise explains this away.

2. FashionMNIST — here the leap is modest

$ python train.py --model linear
linear: 7,850 parameters
...
epoch 10  loss 0.4342  train accuracy 85.5%
done in 7.9s

$ python train.py --model mlp
mlp: 101,770 parameters
...
epoch 10  loss 0.3053  train accuracy 89.5%
done in 5.8s

$ python train.py --model mlp --hidden 128 64
mlp: 109,386 parameters
...
epoch 10  loss 0.2877  train accuracy 89.9%
done in 6.4s

model	numbers	training accuracy
linear	7 850	85.5 %
MLP (128)	101 770	89.5 %
MLP (128, 64)	109 386	89.9 %

What the numbers show — honestly

The rings and the clothes are two very different worlds. The rings are impossible without a hidden layer; the clothes — nearly not. One layer already gets 85.5 % there.

+4 points is real, but small. The run "same seed, same code" gives 85.5 → 89.5. But another pair of runs (different shuffling) gave 84.3 → 87.9: +3.6. That is why I say "about +4", not "+4.0".

A single run is noise. The same MLP(128) with the same settings but a different batch order gave 89.5, 87.9 and 86.3 %. A range of a few points from random shuffling alone. So a single "89.5 vs 85.5" is not very firm; the difference of matched pairs is firmer.

The other "improvement" here is smaller than the noise. (128, 64) gives 89.9 vs 89.5: less than the spread between runs of the same model. Do not say "deeper is better" from one run.

Broader context: I ran three times with different seeds and measured accuracy on images the model had not seen. The linear model averaged 81.6 %, MLP(128) 84.3 % — roughly +3 points. In lesson 7 you will learn to measure that yourself.

The panel now shows the filled-in version

As in the previous lesson: model.py and train.py in the panel are filled in. That is the reference to compare with; later lessons change exactly it.

circles.py CREATE

Create this file in ./

"""circles.py — data a straight line cannot separate: a ring around a dot."""

import math

import torch

from model import MLP, LinearClassifier, count_parameters
from train import evaluate, fit, make_loader


def make_circles(n: int = 1000) -> tuple[torch.Tensor, torch.Tensor]:
    """Class 0 sits on an inner ring (radius 0.5), class 1 on an outer one (radius 1)."""
    y = torch.randint(0, 2, (n,))
    radius = 0.5 + 0.5 * y.float()
    angle = torch.rand(n) * 2 * math.pi
    x = torch.stack([radius * torch.cos(angle), radius * torch.sin(angle)], dim=1)
    return x + 0.05 * torch.randn(n, 2), y


def main() -> None:
    torch.manual_seed(42)
    x, y = make_circles(1000)
    loader = make_loader(x, y, batch_size=32, shuffle=True)

    for name, model in [
        ("linear", LinearClassifier(in_features=2, num_classes=2)),
        ("mlp", MLP(in_features=2, hidden=(16,), num_classes=2)),
    ]:
        print(f"--- {name}: {count_parameters(model)} parameters")
        fit(model, loader, epochs=20, lr=0.01)
        _, acc = evaluate(model, loader)
        print(f"{name} final train accuracy: {acc:.1%}")


if __name__ == "__main__":
    main()

model.py MODIFY 2 lines changed

Edit this file in ./

"""model.py — the model as an nn.Module."""

import torch
from torch import nn


class LinearClassifier(nn.Module):
    """The same 10 neurons as lesson 4: scores = x @ W + b."""

    def __init__(self, in_features: int = 784, num_classes: int = 10) -> None:
        super().__init__()
        self.fc = nn.Linear(in_features, num_classes)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.fc(x)  # raw scores (logits); the loss applies softmax itself


class MLP(nn.Module):
    """Linear -> ReLU -> Linear -> ReLU -> ... -> Linear."""

    def __init__(
        self,
        in_features: int = 784,
        hidden: tuple[int, ...] = (128,),
        num_classes: int = 10,
    ) -> None:
        super().__init__()
        layers: list[nn.Module] = []
        width = in_features
        for h in hidden:
            layers += [nn.Linear(width, h), nn.ReLU()]
            width = h
        layers.append(nn.Linear(width, num_classes))
        self.net = nn.Sequential(*layers)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.net(x)


def count_parameters(model: nn.Module) -> int:
    return sum(p.numel() for p in model.parameters())

train.py MODIFY 3 lines changed

Edit this file in ./

"""train.py — the training loop, automated: DataLoader, optimizer, nn.Module."""

import argparse
import time

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

from fashion import load_fashion
from model import MLP, LinearClassifier, count_parameters


def make_loader(
    x: torch.Tensor, y: torch.Tensor, batch_size: int = 64, shuffle: bool = False
) -> DataLoader:
    # num_workers=0 (the default) is the safe choice on Windows.
    return DataLoader(TensorDataset(x, y), batch_size=batch_size, shuffle=shuffle)


@torch.no_grad()
def evaluate(model: nn.Module, loader: DataLoader) -> tuple[float, float]:
    """Return (average loss, accuracy) over every batch of the loader."""
    loss_fn = nn.CrossEntropyLoss(reduction="sum")
    total_loss, correct, count = 0.0, 0, 0
    for xb, yb in loader:
        logits = model(xb)
        total_loss += loss_fn(logits, yb).item()
        correct += (logits.argmax(dim=1) == yb).sum().item()
        count += len(yb)
    return total_loss / count, correct / count


def fit(model: nn.Module, loader: DataLoader, epochs: int, lr: float) -> None:
    loss_fn = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=lr)

    for epoch in range(1, epochs + 1):
        for xb, yb in loader:
            loss = loss_fn(model(xb), yb)  # forward + loss
            optimizer.zero_grad()  # clear old gradients
            loss.backward()  # backward
            optimizer.step()  # W -= lr * W.grad, for every parameter

        train_loss, train_acc = evaluate(model, loader)
        print(f"epoch {epoch:2d}  loss {train_loss:.4f}  train accuracy {train_acc:.1%}")


def build_model(name: str, hidden: tuple[int, ...]) -> nn.Module:
    if name == "linear":
        return LinearClassifier()
    return MLP(hidden=hidden)


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--model", choices=["linear", "mlp"], default="mlp")
    parser.add_argument("--hidden", type=int, nargs="+", default=[128])
    parser.add_argument("--epochs", type=int, default=10)
    parser.add_argument("--lr", type=float, default=0.001)
    args = parser.parse_args()

    torch.manual_seed(42)
    x, y = load_fashion("train", n=10_000)
    loader = make_loader(x, y, batch_size=64, shuffle=True)

    model = build_model(args.model, tuple(args.hidden))
    print(f"{args.model}: {count_parameters(model):,} parameters")
    start = time.time()
    _, acc = evaluate(model, loader)
    print(f"before training: train accuracy {acc:.1%}")

    fit(model, loader, epochs=args.epochs, lr=args.lr)
    print(f"done in {time.time() - start:.1f}s")


if __name__ == "__main__":
    main()