Overfitting & Regularization

Code

Run it: the honest number

The tests pass. For the first time, run the training with all three measures:

$ python train.py --dropout 0.3 --weight-decay 1e-4 --patience 5
mlp: 269,322 parameters
epoch  1  loss 0.5656  train acc 80.2%  |  val loss 0.6024  val acc 78.5%
epoch  2  loss 0.4643  train acc 83.2%  |  val loss 0.5130  val acc 81.6%
...
epoch  9  loss 0.2889  train acc 89.4%  |  val loss 0.3878  val acc 86.0%
epoch 10  loss 0.3128  train acc 88.3%  |  val loss 0.4364  val acc 84.4%
epoch 11  loss 0.2745  train acc 89.7%  |  val loss 0.3953  val acc 85.8%
epoch 12  loss 0.2646  train acc 90.1%  |  val loss 0.4042  val acc 86.0%
epoch 13  loss 0.2645  train acc 89.9%  |  val loss 0.3985  val acc 85.4%
epoch 14  loss 0.2668  train acc 89.7%  |  val loss 0.4197  val acc 85.7%
early stop at epoch 14: best val loss was 0.3878
final: train 89.4%  val 86.0%  gap +3.3%
done in 20.6s

See what happened: training stopped at epoch 14 (the best was at epoch 9, and five in a row did not improve), and the weights were put back to epoch 9's. That is why the final "train 89.4 %" is the training accuracy of the epoch-9 model, not of epoch 14.

One table

regularize.py (a new file — provided, you do not write it) trains the same network five ways and prints one table:

$ python regularize.py
config               epochs   train     val     gap
baseline                 30   97.1%   86.7%  +10.4%
dropout 0.3              30   93.8%   87.0%   +6.8%
weight decay 1e-3        30   92.5%   85.7%   +6.9%
early stopping           12   90.7%   85.8%   +4.9%
all three                14   89.4%   86.0%   +3.3%

The same starting state (seed 42) for every row, the same network, the same 5 000 validation images.

How to read it

The gap narrowed — that is real. +10.4 → +3.3 points.

But look at how it narrowed. Validation accuracy: 86.7 → 86.0. It did not rise. Training: 97.1 → 89.4 — it fell. The gap shrank because we pushed the training number down, not because we raised the real one.

To say "regularisation made the network better" from this table is dishonest. The honest version: regularisation made the training accuracy closer to the real one. That is valuable too — now the training number can almost be trusted again — but it is not the same thing.

The panel now shows the filled-in version

In this step fashion.py and train.py in the panel are filled in. It is the reference to compare your files with; later lessons change exactly it.

fashion.py MODIFY 2 lines changed

Edit this file in ./

"""fashion.py — load FashionMNIST as plain tensors."""

import os

import torch
from torchvision import datasets

# Where torchvision caches the download. Override with the NN_DATA env variable.
DATA_ROOT = os.environ.get("NN_DATA", "downloads")

CLASSES = [
    "T-shirt",
    "Trouser",
    "Pullover",
    "Dress",
    "Coat",
    "Sandal",
    "Shirt",
    "Sneaker",
    "Bag",
    "Ankle boot",
]


def load_fashion(
    split: str = "train", n: int | None = None
) -> tuple[torch.Tensor, torch.Tensor]:
    """Return (x, y). x is [n, 784] float32 in 0..1, y is [n] int64 labels.

    split: "train" = images 0..49 999 of the training file,
           "val"   = images 50 000..59 999 (never trained on, used to judge),
           "test"  = the official 10 000 test images (touch at the very end).
    """
    ds = datasets.FashionMNIST(DATA_ROOT, train=(split != "test"), download=True)
    x, y = ds.data, ds.targets  # x: [N, 28, 28] uint8
    if split == "train":
        x, y = x[:50_000], y[:50_000]
    elif split == "val":
        x, y = x[50_000:], y[50_000:]
    if n is not None:
        x, y = x[:n], y[:n]
    return x.reshape(len(x), -1).float() / 255.0, y

train.py MODIFY 44 lines changed

Edit this file in ./

"""train.py — the training loop, automated: DataLoader, optimizer, nn.Module."""

import argparse
import copy
import time

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

from fashion import load_fashion
from model import MLP, LinearClassifier, count_parameters


def make_loader(
    x: torch.Tensor, y: torch.Tensor, batch_size: int = 64, shuffle: bool = False
) -> DataLoader:
    # num_workers=0 (the default) is the safe choice on Windows.
    return DataLoader(TensorDataset(x, y), batch_size=batch_size, shuffle=shuffle)


@torch.no_grad()
def evaluate(model: nn.Module, loader: DataLoader) -> tuple[float, float]:
    """Return (average loss, accuracy) over every batch of the loader."""
    model.eval()  # switch dropout off while measuring
    loss_fn = nn.CrossEntropyLoss(reduction="sum")
    total_loss, correct, count = 0.0, 0, 0
    for xb, yb in loader:
        logits = model(xb)
        total_loss += loss_fn(logits, yb).item()
        correct += (logits.argmax(dim=1) == yb).sum().item()
        count += len(yb)
    return total_loss / count, correct / count


def fit(
    model: nn.Module,
    loader: DataLoader,
    epochs: int,
    lr: float,
    val_loader: DataLoader | None = None,
    weight_decay: float = 0.0,
    patience: int | None = None,
    verbose: bool = True,
) -> list[dict]:
    """Train; return one dict of metrics per epoch.

    patience: stop when val loss has not improved for this many epochs,
    and put the best weights back.
    """
    loss_fn = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(
        model.parameters(), lr=lr, weight_decay=weight_decay
    )
    history: list[dict] = []
    best_loss, best_state, waited = float("inf"), None, 0

    for epoch in range(1, epochs + 1):
        model.train()  # switch dropout on while learning
        for xb, yb in loader:
            loss = loss_fn(model(xb), yb)  # forward + loss
            optimizer.zero_grad()  # clear old gradients
            loss.backward()  # backward
            optimizer.step()  # W -= lr * W.grad, for every parameter

        train_loss, train_acc = evaluate(model, loader)
        row = {"epoch": epoch, "train_loss": train_loss, "train_acc": train_acc}
        line = f"epoch {epoch:2d}  loss {train_loss:.4f}  train acc {train_acc:.1%}"

        if val_loader is not None:
            val_loss, val_acc = evaluate(model, val_loader)
            row.update(val_loss=val_loss, val_acc=val_acc)
            line += f"  |  val loss {val_loss:.4f}  val acc {val_acc:.1%}"
            if val_loss < best_loss:
                best_loss, waited = val_loss, 0
                best_state = copy.deepcopy(model.state_dict())
                row["best"] = True
            else:
                waited += 1

        history.append(row)
        if verbose:
            print(line)
        if patience is not None and waited >= patience:
            if verbose:
                print(
                    f"early stop at epoch {epoch}: best val loss was {best_loss:.4f}"
                )
            break

    if patience is not None and best_state is not None:
        model.load_state_dict(best_state)
    return history


def build_model(name: str, hidden: tuple[int, ...], dropout: float) -> nn.Module:
    if name == "linear":
        return LinearClassifier()
    return MLP(hidden=hidden, dropout=dropout)


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--model", choices=["linear", "mlp"], default="mlp")
    parser.add_argument("--hidden", type=int, nargs="+", default=[256, 256])
    parser.add_argument("--epochs", type=int, default=30)
    parser.add_argument("--lr", type=float, default=0.001)
    parser.add_argument("--n-train", type=int, default=10_000)
    parser.add_argument("--dropout", type=float, default=0.0)
    parser.add_argument("--weight-decay", type=float, default=0.0)
    parser.add_argument("--patience", type=int, default=None)
    args = parser.parse_args()

    torch.manual_seed(42)
    x, y = load_fashion("train", n=args.n_train)
    xv, yv = load_fashion("val", n=5_000)
    loader = make_loader(x, y, batch_size=64, shuffle=True)
    val_loader = make_loader(xv, yv, batch_size=256)

    model = build_model(args.model, tuple(args.hidden), args.dropout)
    print(f"{args.model}: {count_parameters(model):,} parameters")
    start = time.time()
    history = fit(
        model,
        loader,
        epochs=args.epochs,
        lr=args.lr,
        val_loader=val_loader,
        weight_decay=args.weight_decay,
        patience=args.patience,
    )
    _, train_acc = evaluate(model, loader)
    _, val_acc = evaluate(model, val_loader)
    print(f"final: train {train_acc:.1%}  val {val_acc:.1%}  gap {train_acc - val_acc:+.1%}")
    print(f"done in {time.time() - start:.1f}s")


if __name__ == "__main__":
    main()

regularize.py CREATE

Create this file in ./

"""regularize.py — one model, four ways of training it, one honest table."""

import torch

from fashion import load_fashion
from model import MLP
from train import evaluate, fit, make_loader

CONFIGS = [
    # name,            dropout, weight_decay, patience
    ("baseline", 0.0, 0.0, None),
    ("dropout 0.3", 0.3, 0.0, None),
    ("weight decay 1e-3", 0.0, 1e-3, None),
    ("early stopping", 0.0, 0.0, 3),
    ("all three", 0.3, 1e-4, 5),
]


def main() -> None:
    x, y = load_fashion("train", n=10_000)
    xv, yv = load_fashion("val", n=5_000)
    loader = make_loader(x, y, batch_size=64, shuffle=True)
    val_loader = make_loader(xv, yv, batch_size=256)

    print(f"{'config':<20}{'epochs':>7}{'train':>8}{'val':>8}{'gap':>8}")
    for name, dropout, weight_decay, patience in CONFIGS:
        torch.manual_seed(42)  # same starting weights for every config
        model = MLP(hidden=(256, 256), dropout=dropout)
        history = fit(
            model, loader, epochs=30, lr=0.001, val_loader=val_loader,
            weight_decay=weight_decay, patience=patience, verbose=False,
        )
        _, train_acc = evaluate(model, loader)
        _, val_acc = evaluate(model, val_loader)
        gap = train_acc - val_acc
        print(f"{name:<20}{len(history):>7}{train_acc:>8.1%}{val_acc:>8.1%}{gap:>+8.1%}")


if __name__ == "__main__":
    main()