Convolutional Networks (CNN)

Code

Train the CNN and see what it really bought

All 13 tests pass — so the CNN exists. Time to train it.

What changes in `train.py`

Almost nothing. fit, evaluate and the loop have no idea about pictures; they only know "model → logits". All that is needed is for build_model to be able to create a CNN:

from model import CNN, MLP, LinearClassifier, count_parameters
...
    if name == "cnn":
        return CNN(dropout=dropout)
...
    parser.add_argument("--model", choices=["linear", "mlp", "cnn"], default="mlp")

In the right panel you see three files in our versions: model.py with the CNN filled in, conv_by_hand.py with the holes filled and pooling written, and train.py with this change. If yours passes the tests, keep yours — these are only for comparison.

Run it

python train.py --model cnn --epochs 15 --dropout 0.3

With our random seed (torch.manual_seed(42)) and 10,000 training images we got:

cnn: 20,490 parameters
epoch  1  loss 0.6056  train acc 76.5%  |  val loss 0.6426  val acc 74.5%
epoch  2  loss 0.4953  train acc 82.4%  |  val loss 0.5401  val acc 80.8%
epoch  3  loss 0.4497  train acc 83.6%  |  val loss 0.5035  val acc 81.7%
   ...
epoch 13  loss 0.2729  train acc 90.8%  |  val loss 0.3562  val acc 87.5%
epoch 14  loss 0.2728  train acc 90.4%  |  val loss 0.3651  val acc 87.4%
epoch 15  loss 0.2769  train acc 89.9%  |  val loss 0.3647  val acc 87.4%
final: train 89.9%  val 87.4%  gap +2.5%

The duration depends on your CPU: our measurements ranged from ~20 seconds (a quiet CPU) to 66 seconds (with three other runs sharing it).

Your numbers will differ by roughly 1–2 points. The same machine with the same seed reproduces numbers exactly, but another machine or another PyTorch version does not quite. What matters is the order, not the third digit.

What it means

model	parameters	train	val	gap
MLP (256, 256), 30 ep. (lesson 7)	269,322	97.1 %	86.7 %	+10.4
MLP (256, 256) + dropout + weight decay + early stopping	269,322	89.4 %	86.0 %	+3.3
CNN (16, 32), dropout 0.3, 15 ep.	20,490	89.9 %	87.4 %	+2.5

Read it carefully, because it is easy to overstate:

The CNN beat the MLP by 0.7 points in one seed. Over three seeds (0, 1, 2), on average: MLP 86.2 %, MLP with regularisation 86.5 %, CNN 87.5 %; the seed-to-seed spread is ~0.6 points. That is roughly +1 to +1.5 points — real, but not a jump.
And it did it with 13 times fewer parameters (20,490 vs 269,322).
The train-to-validation gap is 2.5 points instead of 10.4: a smaller network has less to memorise.

Equal budget — this is where the difference is

The fair comparison is the same parameter count. An MLP with a single hidden layer of 25 neurons has 19,885 parameters, nearly the same as the CNN:

python train.py --model mlp --hidden 25 --epochs 15
python train.py --model cnn --epochs 15

mlp: 19,885 parameters
   ...
final: train 87.2%  val 84.1%  gap +3.1%

cnn: 20,490 parameters
   ...
final: train 91.0%  val 87.2%  gap +3.8%

(Both without dropout, 15 epochs, seed 42.) That is +3.1 points on validation for the same number of parameters. This is where structure pays off.

What to take away

On this task the CNN is not magic: 28 × 28 grayscale clothes are not where it shines. But it wins when the budget is limited, and it is practically unavoidable when pictures are large (224 × 224 × 3 = 150,528 inputs — a dense layer with 100 neurons would have over 15 million weights). Those are exactly the pictures we move to in lessons 9 and 10.

model.py MODIFY 14 lines changed

Edit this file in ./

"""model.py — the model as an nn.Module."""

import torch
from torch import nn


class LinearClassifier(nn.Module):
    """The same 10 neurons as lesson 4: scores = x @ W + b."""

    def __init__(self, in_features: int = 784, num_classes: int = 10) -> None:
        super().__init__()
        self.fc = nn.Linear(in_features, num_classes)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.fc(x)  # raw scores (logits); the loss applies softmax itself


class MLP(nn.Module):
    """Linear -> ReLU -> (Dropout) -> ... -> Linear."""

    def __init__(
        self,
        in_features: int = 784,
        hidden: tuple[int, ...] = (128,),
        num_classes: int = 10,
        dropout: float = 0.0,
    ) -> None:
        super().__init__()
        layers: list[nn.Module] = []
        width = in_features
        for h in hidden:
            layers += [nn.Linear(width, h), nn.ReLU(), nn.Dropout(dropout)]
            width = h
        layers.append(nn.Linear(width, num_classes))
        self.net = nn.Sequential(*layers)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.net(x)


class CNN(nn.Module):
    """conv -> ReLU -> pool, twice, then one Linear layer on the feature maps.

    Input: [N, 1, 28, 28] images, or flat [N, 784] rows (forward reshapes them).
    Layers, in order:
      Conv2d(1, c1, kernel_size=3, padding=1) -> ReLU -> MaxPool2d(2)    28 -> 14
      Conv2d(c1, c2, kernel_size=3, padding=1) -> ReLU -> MaxPool2d(2)   14 -> 7
      Flatten -> Dropout(dropout) -> Linear(c2 * 7 * 7, num_classes)
    With the defaults (16, 32) that is 20,490 parameters.
    """

    def __init__(
        self,
        num_classes: int = 10,
        channels: tuple[int, int] = (16, 32),
        dropout: float = 0.0,
    ) -> None:
        super().__init__()
        c1, c2 = channels
        self.features = nn.Sequential(
            nn.Conv2d(1, c1, kernel_size=3, padding=1),  # [N, 1, 28, 28] -> [N, c1, 28, 28]
            nn.ReLU(),
            nn.MaxPool2d(2),  # -> [N, c1, 14, 14]
            nn.Conv2d(c1, c2, kernel_size=3, padding=1),  # -> [N, c2, 14, 14]
            nn.ReLU(),
            nn.MaxPool2d(2),  # -> [N, c2, 7, 7]
        )
        self.head = nn.Sequential(
            nn.Flatten(), nn.Dropout(dropout), nn.Linear(c2 * 7 * 7, num_classes)
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = x.reshape(-1, 1, 28, 28)  # flat 784-pixel rows -> 1-channel images
        return self.head(self.features(x))


def count_parameters(model: nn.Module) -> int:
    return sum(p.numel() for p in model.parameters())

conv_by_hand.py MODIFY 11 lines changed

Edit this file in ./

"""conv_by_hand.py — a convolution written with plain loops."""

import torch


def conv2d_by_hand(image: torch.Tensor, kernel: torch.Tensor) -> torch.Tensor:
    """Slide kernel over image (no padding, stride 1) and return the feature map.

    image: [H, W]   kernel: [kh, kw]   result: [H - kh + 1, W - kw + 1]
    Every output number is the sum of (patch * kernel), where patch is the
    kh x kw piece of the image that lies under the kernel at that position.
    """
    h, w = image.shape
    kh, kw = kernel.shape
    out = torch.zeros(h - kh + 1, w - kw + 1)
    for i in range(out.shape[0]):
        for j in range(out.shape[1]):
            # the kh x kw patch whose top-left corner is (i, j)
            patch = image[i : i + kh, j : j + kw]
            # multiply patch and kernel element by element, add it all up
            out[i, j] = (patch * kernel).sum()
    return out


def max_pool2d_by_hand(x: torch.Tensor, size: int = 2) -> torch.Tensor:
    """Keep the largest number in every size x size block (stride = size).

    x: [H, W]   result: [H // size, W // size]
    A leftover row or column that does not fill a whole block is dropped,
    exactly as nn.MaxPool2d does it.
    """
    h, w = x.shape
    out = torch.zeros(h // size, w // size)
    for i in range(out.shape[0]):
        for j in range(out.shape[1]):
            block = x[i * size : (i + 1) * size, j * size : (j + 1) * size]
            out[i, j] = block.max()
    return out

train.py MODIFY 4 lines changed

Edit this file in ./

"""train.py — the training loop, automated: DataLoader, optimizer, nn.Module."""

import argparse
import copy
import time

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

from fashion import load_fashion
from model import CNN, MLP, LinearClassifier, count_parameters


def make_loader(
    x: torch.Tensor, y: torch.Tensor, batch_size: int = 64, shuffle: bool = False
) -> DataLoader:
    # num_workers=0 (the default) is the safe choice on Windows.
    return DataLoader(TensorDataset(x, y), batch_size=batch_size, shuffle=shuffle)


@torch.no_grad()
def evaluate(model: nn.Module, loader: DataLoader) -> tuple[float, float]:
    """Return (average loss, accuracy) over every batch of the loader."""
    model.eval()  # switch dropout off while measuring
    loss_fn = nn.CrossEntropyLoss(reduction="sum")
    total_loss, correct, count = 0.0, 0, 0
    for xb, yb in loader:
        logits = model(xb)
        total_loss += loss_fn(logits, yb).item()
        correct += (logits.argmax(dim=1) == yb).sum().item()
        count += len(yb)
    return total_loss / count, correct / count


def fit(
    model: nn.Module,
    loader: DataLoader,
    epochs: int,
    lr: float,
    val_loader: DataLoader | None = None,
    weight_decay: float = 0.0,
    patience: int | None = None,
    verbose: bool = True,
) -> list[dict]:
    """Train; return one dict of metrics per epoch.

    patience: stop when val loss has not improved for this many epochs,
    and put the best weights back.
    """
    loss_fn = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(
        model.parameters(), lr=lr, weight_decay=weight_decay
    )
    history: list[dict] = []
    best_loss, best_state, waited = float("inf"), None, 0

    for epoch in range(1, epochs + 1):
        model.train()  # switch dropout on while learning
        for xb, yb in loader:
            loss = loss_fn(model(xb), yb)  # forward + loss
            optimizer.zero_grad()  # clear old gradients
            loss.backward()  # backward
            optimizer.step()  # W -= lr * W.grad, for every parameter

        train_loss, train_acc = evaluate(model, loader)
        row = {"epoch": epoch, "train_loss": train_loss, "train_acc": train_acc}
        line = f"epoch {epoch:2d}  loss {train_loss:.4f}  train acc {train_acc:.1%}"

        if val_loader is not None:
            val_loss, val_acc = evaluate(model, val_loader)
            row.update(val_loss=val_loss, val_acc=val_acc)
            line += f"  |  val loss {val_loss:.4f}  val acc {val_acc:.1%}"
            if val_loss < best_loss:
                best_loss, waited = val_loss, 0
                best_state = copy.deepcopy(model.state_dict())
                row["best"] = True
            else:
                waited += 1

        history.append(row)
        if verbose:
            print(line)
        if patience is not None and waited >= patience:
            if verbose:
                print(
                    f"early stop at epoch {epoch}: best val loss was {best_loss:.4f}"
                )
            break

    if patience is not None and best_state is not None:
        model.load_state_dict(best_state)
    return history


def build_model(name: str, hidden: tuple[int, ...], dropout: float) -> nn.Module:
    if name == "linear":
        return LinearClassifier()
    if name == "cnn":
        return CNN(dropout=dropout)
    return MLP(hidden=hidden, dropout=dropout)


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--model", choices=["linear", "mlp", "cnn"], default="mlp")
    parser.add_argument("--hidden", type=int, nargs="+", default=[256, 256])
    parser.add_argument("--epochs", type=int, default=30)
    parser.add_argument("--lr", type=float, default=0.001)
    parser.add_argument("--n-train", type=int, default=10_000)
    parser.add_argument("--dropout", type=float, default=0.0)
    parser.add_argument("--weight-decay", type=float, default=0.0)
    parser.add_argument("--patience", type=int, default=None)
    args = parser.parse_args()

    torch.manual_seed(42)
    x, y = load_fashion("train", n=args.n_train)
    xv, yv = load_fashion("val", n=5_000)
    loader = make_loader(x, y, batch_size=64, shuffle=True)
    val_loader = make_loader(xv, yv, batch_size=256)

    model = build_model(args.model, tuple(args.hidden), args.dropout)
    print(f"{args.model}: {count_parameters(model):,} parameters")
    start = time.time()
    history = fit(
        model,
        loader,
        epochs=args.epochs,
        lr=args.lr,
        val_loader=val_loader,
        weight_decay=args.weight_decay,
        patience=args.patience,
    )
    _, train_acc = evaluate(model, loader)
    _, val_acc = evaluate(model, val_loader)
    print(f"final: train {train_acc:.1%}  val {val_acc:.1%}  gap {train_acc - val_acc:+.1%}")
    print(f"done in {time.time() - start:.1f}s")


if __name__ == "__main__":
    main()

Train the CNN and see what it really bought

What changes in train.py

Run it

What it means

Equal budget — this is where the difference is

What to take away

What changes in `train.py`