Evaluation & Deployment

Code

One image: predict.py

Until now the model evaluated folders. Now somebody brings one photo. The road from file to answer is short, but every step can go wrong — and the mistake will not announce itself.

One image, step by step

This is that road with the real model and one test/ image (real output):

PIL image: (28, 28) L
after convert: (28, 28) RGB
tensor: (3, 224, 224) min -2.12 max 2.62
with batch: (1, 3, 224, 224)
logits: (1, 4) [1.34, 5.91, -3.7, -4.63]
probs: [0.01, 0.99, 0.0, 0.0] sum 1.0
classes: ['coat', 'pullover', 'sneaker', 'tshirt']

Read it in order:

The file opens as a 28×28 grey (L) picture.
.convert("RGB") — ResNet expects 3 channels. A grey image gets three identical ones.
The same transforms as in training: size 224×224 and ImageNet normalisation (which is why the values run from −2.12 to 2.62, not 0…1).
unsqueeze(0) — the network expects a batch, so one image becomes a batch of one: [1, 3, 224, 224].
The output is four scores (logits), one per class. softmax turns them into probabilities that sum to 1.
The class names come from the same model.pt, where we saved the list.

Your job

Create predict.py and test_predict.py (on the right). Both are complete — there are no holes, because this file is glue, not an exercise. Look at one line:

tf = build_transform(False, IMAGENET_MEAN, IMAGENET_STD, size, rgb=True)

It is the same function that make_loaders and make_eval_loader use to prepare images. Not "similar" — the same one. That is how a difference between the two paths (evaluation and prediction) becomes impossible.

Try it:

python predict.py test/coat/0000.png test/sneaker/0003.png test/tshirt/0050.png

test/coat/0000.png: pullover 99%, coat 1%, sneaker 0%
test/sneaker/0003.png: sneaker 100%, coat 0%, pullover 0%
test/tshirt/0050.png: tshirt 100%, coat 0%, pullover 0%

The first image is a coat, and the model says pullover and is 99 % sure. It is one of those 11 coats the model called a pullover. The softmax probability only shows the ratio of the scores, not the real probability of being right: 99 % is no protection against a mistake.

Three ways to get it wrong that nobody will tell you about

test_predict.py holds the most important test: agrees_with_the_evaluation_loader. It takes the same files and sends them two ways: through make_eval_loader and through predict_image. If the paths differ by even one step, the numbers disagree.

That catches all three:

1. Normalize forgotten.

AssertionError: False is not true : …/ant/0.png: predict_image gives [0.501939594745636, 0.498060405254364], evaluation gives [0.5009315013885498, 0.4990684986114502]

The difference is small (the test uses a randomly initialised network), but real. On the real model: the same coat image gets pullover 99 %, coat 1 % with normalisation and pullover 93 %, coat 4 % without; and over 100 test/ images (25 per class) the accuracy without Normalize is 25.0 %. That is exactly the level of random guessing for four classes.

2. Resize forgotten (28 px straight into the network).

AssertionError: False is not true : …/ant/0.png: predict_image gives [0.30359408259391785, 0.6964059472084045], evaluation gives [0.5009315013885498, 0.4990684986114502]

On the real model that is 25.0 % on the same 100 images, too.

3. .convert("RGB") forgotten.

RuntimeError: output with shape [1, 64, 64] doesn't match the broadcast shape [3, 64, 64]

This one at least shouts. (Without Normalize as well, the error would be a different but equally loud one: expected input[1, 1, 224, 224] to have 3 channels, but got 1 channels instead.)

Two of the three mistakes are silent. That is why the test does not compare with a "right answer" (a new image may not have one) but with another path you have already verified.

A practical rule: if accuracy suddenly becomes exactly 1 / number of classes (here 25 %), suspect the preparation — size, normalisation, channels — before you suspect the model.

predict.py CREATE

Create this file in ./

"""predict.py — use the saved model on images it has never seen."""

import argparse

import torch
from PIL import Image

from data import IMAGENET_MEAN, IMAGENET_STD, build_transform
from evaluate import load_model


@torch.no_grad()
def predict_image(
    model: torch.nn.Module, classes: list[str], size: int, image: Image.Image
) -> list[tuple[str, float]]:
    """Return every class with its probability, best first."""
    # The SAME preprocessing as training: same size, same normalisation.
    tf = build_transform(False, IMAGENET_MEAN, IMAGENET_STD, size, rgb=True)
    x = tf(image.convert("RGB")).unsqueeze(0)  # add a batch dimension: [1, 3, H, W]
    probs = model(x).softmax(dim=1)[0]
    return sorted(zip(classes, probs.tolist()), key=lambda cp: -cp[1])


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("images", nargs="+")
    parser.add_argument("--model", default="model.pt")
    args = parser.parse_args()

    model, classes, size = load_model(args.model)
    for path in args.images:
        ranked = predict_image(model, classes, size, Image.open(path))
        top = ", ".join(f"{name} {p:.0%}" for name, p in ranked[:3])
        print(f"{path}: {top}")


if __name__ == "__main__":
    main()

test_predict.py CREATE

Create this file in ./

"""test_predict.py — predict_image must see an image exactly as evaluation does."""

import os
import tempfile
import unittest

import torch
from PIL import Image

from data import make_eval_loader
from predict import predict_image
from transfer import build_resnet


class TestPredictImage(unittest.TestCase):
    def setUp(self):
        torch.manual_seed(0)
        self.tmp = tempfile.TemporaryDirectory()
        self.paths = []
        for cls in ("ant", "bee"):  # two noisy 28x28 grey images per class
            os.makedirs(os.path.join(self.tmp.name, cls))
            for k in range(2):
                pixels = (torch.rand(28, 28) * 255).to(torch.uint8).numpy()
                path = os.path.join(self.tmp.name, cls, f"{k}.png")
                Image.fromarray(pixels).save(path)
                self.paths.append(path)
        self.model = build_resnet(2, pretrained=False)
        self.model.eval()
        self.classes = ["ant", "bee"]

    def tearDown(self):
        self.tmp.cleanup()

    def test_probabilities_are_sorted_and_sum_to_one(self):
        ranked = predict_image(self.model, self.classes, 64, Image.open(self.paths[0]))
        probs = [p for _, p in ranked]
        self.assertEqual(sorted(probs, reverse=True), probs)
        self.assertAlmostEqual(sum(probs), 1.0, places=5)

    def test_agrees_with_the_evaluation_loader(self):
        # The same files, once through the evaluation loader and once through
        # predict_image. Any preprocessing difference shows up as a mismatch.
        loader, _ = make_eval_loader(self.tmp.name, size=64)
        with torch.no_grad():
            expected = torch.cat([self.model(xb).softmax(dim=1) for xb, _ in loader])
        for path, want in zip(self.paths, expected):
            ranked = predict_image(self.model, self.classes, 64, Image.open(path))
            got = torch.tensor([dict(ranked)[c] for c in self.classes])
            self.assertTrue(
                torch.allclose(got, want, atol=1e-5),
                f"{path}: predict_image gives {got.tolist()}, evaluation gives {want.tolist()}",
            )


if __name__ == "__main__":
    unittest.main()