Data & Augmentation — Programuoki

Code

Your photographs — prepare.py and our dataset

Your photographs do not look like 28 × 28 gray PNGs. A phone gives a 4000 × 3000 colour JPEG, with a "rotated" flag on top. Before anything else they have to be put into the folder layout that data.py reads.

What `prepare.py` does

photos/                       data/
  category_a/  IMG_001.jpg        category_a/  0000.png  0001.png ...
  category_b/  IMG_002.jpg  →     category_b/  0000.png  0001.png ...

For every photograph it:

opens any format Pillow understands (jpg, png, webp …);
applies the phone's "rotation" flag (exif_transpose) — otherwise half of the pictures would lie on their side;
converts to grayscale;
cuts a square from the middle and shrinks it to 28 × 28 — nothing is stretched;
saves it as 0000.png, 0001.png …

It is complete and ready — a tool, not an exercise. Create prepare.py (see the right panel).

Run it

python prepare.py

With no arguments it reads photos/ and writes to data/. You can name other folders too: python prepare.py photos_test data_test (your separate test folder). An example of what it prints (our trial with two folders of 6 JPEGs and one text file in each):

skipped (not an image): photos\cat\notes.txt
skipped (not an image): photos\dog\notes.txt
cat: 6 images
dog: 6 images

A file that is not a picture is skipped and you are told — it will not blow up later with a confusing error.

What you lose — honestly

Colour and fine detail. 28 × 28 in shades of gray is very little. If your categories differ by colour (say "ripe / unripe"), this step throws that difference away. It is a deliberate compromise: our CNN takes 1 channel and 28 × 28. In lesson 10, when we use a pretrained network, we read pictures from photos/ straight from the originals (colour, 224 × 224).

If 28 × 28 seems too small, you may change SIZE — but then the CNN (the reshape to 28 × 28 and the Linear input c2 · 7 · 7) has to change too. In this course we stay with 28.

Before your own photographs — reproduce our numbers

You do not have our stand-in dataset, but you can make it. A second file, make_standin.py, takes four FashionMNIST classes you already downloaded in lesson 2 and saves them as PNG folders with exactly the same layout as your photographs:

python make_standin.py

standin_data {'tshirt': 200, 'pullover': 200, 'coat': 200, 'sneaker': 200}
standin_test {'tshirt': 100, 'pullover': 100, 'coat': 100, 'sneaker': 100}
standin_imbalanced {'tshirt': 200, 'pullover': 200, 'coat': 20, 'sneaker': 200}

That is ~1,800 small files, so it can take a couple of minutes. Three folders:

standin_data — 4 classes of 200 (the lesson's numbers rest on this);
standin_test — 100 new pictures per class (we use it in lesson 11);
standin_imbalanced — like standin_data, but coat has only 20 (the trap in step 7).

Work with standin_data first, so that you can see your code produce our numbers. Only then move on to your own data/.

What next

You have data/<category>/*.png. In the next step you write the code that reads it, splits it and normalises it.

prepare.py CREATE

Create this file in ./

"""prepare.py — turn your photographs into the data/<class>/*.png layout.

    python prepare.py [photos_dir] [data_dir]

Reads photos/<class>/* (any format Pillow can open: jpg, png, webp, ...) and
writes data/<class>/0000.png, 0001.png, ... as 28x28 grayscale. Each picture is
centre-cropped to a square first, so nothing is stretched.
"""

import sys
from pathlib import Path

from PIL import Image, ImageOps

SIZE = 28


def prepare(src: Path, dst: Path, size: int = SIZE) -> dict[str, int]:
    """Convert every picture of every class folder; return images written per class."""
    counts: dict[str, int] = {}
    for class_dir in sorted(p for p in src.iterdir() if p.is_dir()):
        out_dir = dst / class_dir.name
        out_dir.mkdir(parents=True, exist_ok=True)
        n = 0
        for path in sorted(class_dir.iterdir()):
            try:
                img = ImageOps.exif_transpose(Image.open(path))  # honour the phone's rotation
            except OSError:
                print(f"skipped (not an image): {path}")
                continue
            img = ImageOps.fit(img.convert("L"), (size, size))  # centre-crop, then shrink
            img.save(out_dir / f"{n:04d}.png")
            n += 1
        counts[class_dir.name] = n
    return counts


def main() -> None:
    src = Path(sys.argv[1] if len(sys.argv) > 1 else "photos")
    dst = Path(sys.argv[2] if len(sys.argv) > 2 else "data")
    for name, n in prepare(src, dst).items():
        print(f"{name}: {n} images")


if __name__ == "__main__":
    main()

make_standin.py CREATE