Transfer Learning — Programuoki

Code

Natural photographs — where the jump is real

On the stand-in dataset transfer learning looked modest. Is it really that weak? No — our stand-in dataset was a poor one to show it on: 28 × 28 gray clothes are far from ImageNet. Let us check on natural colour photographs.

Two files

Both are complete — experiment tools, not an exercise:

make_cifar.py saves four CIFAR-10 classes (cat, dog, horse, deer), 200 colour 32 × 32 pictures each, into cifar_data/ — the same folder layout as your photographs. The first time it downloads CIFAR-10 (~170 MB).
cifar_compare.py runs four experiments on the same 640 / 160 split: a small CNN from scratch, the same with augmentation, a frozen ResNet, and a ResNet with layer4. It uses your make_loaders, fit and build_resnet.

python make_cifar.py
python cifar_compare.py

cifar_compare.py takes a few minutes (ours: from ~40 s for the CNN to ~2 min for the ResNet; longer on a busy CPU). Chance is 25 %.

Our result

scratch CNN, 32px, 30 epochs               trainable     35,780  train 100.0%  val 49.4%
scratch CNN + augmentation, 30 epochs      trainable     35,780  train 71.6%   val 56.2%
resnet18 frozen, 224px, 5 epochs, lr .01   trainable      2,052  train 89.7%   val 66.9%
resnet18 + layer4, 224px, 5 epochs, lr .001 trainable  8,395,780  train 100.0%  val 74.4%

	trainable params	train	val
CNN from scratch	35,780	100.0 %	49.4 %
+ augmentation	35,780	71.6 %	56.2 %
ResNet, frozen	2,052	89.7 %	66.9 %
ResNet + `layer4`	8,395,780	100.0 %	74.4 %

Here the jump is real: from 49.4 % to 74.4 % — 25 points, the same code, the same 640 pictures.

What it shows

The CNN from scratch memorises (100 % train, 49 % validation — lesson 7). 640 photographs are too few for it to learn the difference between a cat and a dog on its own.
Here augmentation helps (+6.8 points), unlike on the clothes. The reason is the same one that explained the clothes' tie: augmentation helps where the network overfits badly. But it is one seed and one run.
Frozen features alone — +17.5 points over the CNN from scratch, training only 2,052 parameters. Features learned on other photographs transfer.
Fine-tuning added another +7.5, but the network again memorises the training set (100 %).

How solid is that number?

An earlier run of ours — the same code, but where the model's initial values were not seeded (the model was built before torch.manual_seed) — gave: 52.5 → 57.5 → 65.0 → 79.4 %. The current one, with the seed set before the model: 49.4 → 56.2 → 66.9 → 74.4 %. Differences of up to 5 points, while one picture in 160 is 0.6. Only the order is solid: from scratch ≪ frozen < fine-tuned.

That is why cifar_compare.py sets the seed before building the model. It is not a detail: without it "the same seed" does not mean "the same number" — the weights at the first epoch already differ.

What to take to your own project

With natural photographs and ~600 training examples transfer learning is the main road, not an alternative.
The photographs here are only 32 × 32, blown up to 224 — and it still works. Phone photographs shrunk to 224 will do better.
If your data resembles the clothes (small, single-colour, drawings), do not expect transfer to help more than a small CNN does.

make_cifar.py CREATE

Create this file in ./

"""make_cifar.py — four natural-photo classes from CIFAR-10, as image folders.

    python make_cifar.py

The first run downloads CIFAR-10 (about 170 MB) into the same cache folder
fashion.py uses. Result: cifar_data/<class>/NNNN.png, 200 colour 32x32 images of
cat, dog, horse and deer each - the same layout as your photographs.
"""

from pathlib import Path

from PIL import Image
from torchvision import datasets

from fashion import DATA_ROOT

PICK = {3: "cat", 5: "dog", 7: "horse", 4: "deer"}  # CIFAR-10 label ids
PER_CLASS = 200


def main() -> None:
    ds = datasets.CIFAR10(DATA_ROOT, train=True, download=True)
    counts = {name: 0 for name in PICK.values()}
    for pixels, label in zip(ds.data, ds.targets):
        name = PICK.get(label)
        if name is None or counts[name] >= PER_CLASS:
            continue
        folder = Path("cifar_data") / name
        folder.mkdir(parents=True, exist_ok=True)
        Image.fromarray(pixels).save(folder / f"{counts[name]:04d}.png")
        counts[name] += 1
    print(counts)


if __name__ == "__main__":
    main()

cifar_compare.py CREATE

Create this file in ./

"""cifar_compare.py — does transfer learning pay off on natural colour photos?

    python cifar_compare.py [folder]        (default: cifar_data)

Four runs on the same 640 train / 160 val split, seed 42:
  1. a small CNN trained from scratch at 32 px,
  2. the same CNN with augmentation,
  3. resnet18, backbone frozen, 224 px,
  4. resnet18, layer4 fine-tuned, 224 px.
Takes several minutes on a CPU.
"""

import sys
import time

import torch
from torch import nn

from data import make_loaders
from train import fit
from transfer import build_resnet, count_trainable


def scratch_cnn() -> nn.Module:
    # the lesson-8 idea, with 3 input channels and 32 x 32 images
    return nn.Sequential(
        nn.Conv2d(3, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Flatten(), nn.Dropout(0.3), nn.Linear(64 * 8 * 8, 4),
    )


def run(root, name, make_model, size, epochs, lr, augment=False) -> None:
    torch.manual_seed(42)  # seed FIRST: the model's random start must be seeded too
    model = make_model()
    train_loader, val_loader, _ = make_loaders(
        root, 32, size=size, rgb=True, augment=augment
    )
    start = time.time()
    history = fit(model, train_loader, epochs, lr, val_loader, verbose=False)
    trainable, _ = count_trainable(model)
    last = history[-1]
    print(
        f"{name:<42} trainable {trainable:>10,}  train {last['train_acc']:.1%}"
        f"  val {last['val_acc']:.1%}  {time.time() - start:.0f}s",
        flush=True,
    )


def main() -> None:
    root = sys.argv[1] if len(sys.argv) > 1 else "cifar_data"
    run(root, "scratch CNN, 32px, 30 epochs", scratch_cnn, 32, 30, 1e-3)
    run(root, "scratch CNN + augmentation, 30 epochs", scratch_cnn, 32, 30, 1e-3, augment=True)
    run(root, "resnet18 frozen, 224px, 5 epochs, lr .01", lambda: build_resnet(4), 224, 5, 0.01)
    run(
        root,
        "resnet18 + layer4, 224px, 5 epochs, lr .001",
        lambda: build_resnet(4, True),
        224,
        5,
        0.001,
    )


if __name__ == "__main__":
    main()