Capstone Project — Programuoki

Code

Choose on validation — write it yourself

You have at least two runs and two .json files. Now comes the one new function of this lesson, the one that carries out the first rule: decide on validation.

It looks trivial: max over one field. But this is exactly where it is easiest to slip, because every summary.json already holds both numbers, and test_accuracy is one word away from val_accuracy.

Pseudocode

choose_on_validation(runs):

if runs is empty: ValueError
best ← the first run
for each other run:
    if its val_accuracy is STRICTLY greater than best.val_accuracy:
        best ← this run
return best

table_lines(runs, chosen):

lines ← [header]
for each run (in the same order):
    line ← name, number of epochs, val_accuracy %, test_accuracy %
    if it is chosen: append "<- chosen on validation" to the line
    lines.append(line)
return lines

Two small things the test checks:

A tie. If two runs have the same validation number, the first wins, no matter which one has the better test. Dull, but otherwise test creeps back in — as a "tie-breaker".
Both numbers in every line. The table shows all runs with test, not only the winner. If you showed test only for the winner, you could not see whether the other runs were better or worse.

Your job

Create choose.py and test_choose.py (on the right). In choose.py, load_runs and main are ready (they read the files and print); two functions with raise NotImplementedError are missing.

test_choose.py uses the four rows from the table in the last step and a deliberately chosen trap: the row with the highest test (noaug_5ep, 94.0 %) is not the one to choose (noaug_stop2, 95.6 % on validation).

Run before you write

$ python -m unittest test_choose
...
Ran 9 tests in 0.002s
FAILED (errors=9)

(All nine are NotImplementedError.)

When it does not pass

AssertionError: 'noaug_5ep' != 'noaug_stop2' in …highest_validation_accuracy_wins and …cannot_change_the_choice — you pick by test_accuracy, not val_accuracy. The highest test is noaug_5ep.
AssertionError: 'b' != 'a' in …tie_goes_to_the_first_run… — two equal val, and you took the second: most likely test slipped in as a tie-breaker, e.g. key=(val, test). Keep only val; max itself takes the first of equals.
AssertionError: Lists differ: … in …does_not_modify_the_runs — your function changed the list (e.g. runs.sort(…)). Sort only a copy.
AssertionError: '89.0%' not found in 'aug_stop2 4 90.6% ' — a line of the table has no test number. A line must show both.
ValueError with an empty list needs no separate line: max([]) raises ValueError itself. But now you know why that test exists.

Run it

python choose.py noaug.json aug.json

For our stand-in:

run            epochs     val    test
noaug               6   95.6%   93.8%  <- chosen on validation
aug                 4   90.6%   89.0%

the test folder was evaluated 2 times; report that number

And with all four runs from the previous step:

run            epochs     val    test
aug_stop2           4   90.6%   89.0%
aug_8ep             8   93.1%   92.2%
noaug_stop2         6   95.6%   93.8%  <- chosen on validation
noaug_5ep           5   93.8%   94.0%

Notice the last row: it is better on test (94.0 %), and we did not choose it. The difference from the winner is one image out of 400. That is how it should be: not choosing what looks prettier on test is the whole rule.

One more line: "the test folder was evaluated 4 times". Take that number to your report.

choose.py CREATE

Create this file in ./

"""choose.py — pick a configuration on VALIDATION accuracy, then look at TEST.

Usage: python choose.py aug.json noaug.json ...
Each file is a summary written by `capstone.py --summary <file>`.
"""

import json
import sys
from pathlib import Path


def load_runs(paths: list[str]) -> list[dict]:
    """Read the summary files; each run is named after its file."""
    runs = []
    for path in paths:
        run = json.loads(Path(path).read_text())
        run["name"] = Path(path).stem
        runs.append(run)
    return runs


def choose_on_validation(runs: list[dict]) -> dict:
    """Return the run with the highest val_accuracy.

    Contract:
      - decide from val_accuracy ONLY. test_accuracy must not influence the
        choice in any way - not even as a tie-breaker;
      - on a tie the run that comes first in the list wins;
      - do not modify the runs;
      - raise ValueError if the list is empty.
    """
    raise NotImplementedError


def table_lines(runs: list[dict], chosen: dict) -> list[str]:
    """The comparison table, as a list of lines.

    One header line, then one line per run, in the order given:
    the run's name, epochs_run, val_accuracy and test_accuracy as percentages
    with one decimal (like 95.6%). EVERY run shows BOTH numbers. The chosen
    run's line ends with the text "<- chosen on validation".
    """
    raise NotImplementedError


def main() -> None:
    if len(sys.argv) < 2:
        sys.exit("usage: python choose.py summary1.json summary2.json ...")
    runs = load_runs(sys.argv[1:])
    chosen = choose_on_validation(runs)
    print("\n".join(table_lines(runs, chosen)))
    print(f"\nthe test folder was evaluated {len(runs)} times; report that number")


if __name__ == "__main__":
    main()

test_choose.py CREATE

Create this file in ./

"""test_choose.py — the verdict for choose.py. Run: python -m unittest test_choose -v"""

import copy
import unittest

from choose import choose_on_validation, table_lines

# The four runs from the lesson's experiment (numbers from the recorded runs).
RUNS = [
    {"name": "aug_stop2", "epochs_run": 4, "val_accuracy": 0.906, "test_accuracy": 0.89},
    {"name": "aug_8ep", "epochs_run": 8, "val_accuracy": 0.931, "test_accuracy": 0.9225},
    {"name": "noaug_stop2", "epochs_run": 6, "val_accuracy": 0.9563, "test_accuracy": 0.9375},
    {"name": "noaug_5ep", "epochs_run": 5, "val_accuracy": 0.9375, "test_accuracy": 0.94},
]


class TestChoose(unittest.TestCase):
    def test_the_highest_validation_accuracy_wins(self):
        self.assertEqual(choose_on_validation(RUNS)["name"], "noaug_stop2")

    def test_it_wins_even_though_another_run_has_a_better_test_number(self):
        best_on_test = max(RUNS, key=lambda r: r["test_accuracy"])
        self.assertEqual(best_on_test["name"], "noaug_5ep")  # the trap
        self.assertNotEqual(choose_on_validation(RUNS)["name"], best_on_test["name"])

    def test_the_test_numbers_cannot_change_the_choice(self):
        scrambled = copy.deepcopy(RUNS)
        for run, value in zip(scrambled, [0.99, 0.10, 0.50, 0.97]):
            run["test_accuracy"] = value
        self.assertEqual(choose_on_validation(scrambled)["name"], "noaug_stop2")

    def test_a_tie_goes_to_the_first_run_even_if_the_second_has_better_test(self):
        a = {"name": "a", "epochs_run": 3, "val_accuracy": 0.9, "test_accuracy": 0.80}
        b = {"name": "b", "epochs_run": 3, "val_accuracy": 0.9, "test_accuracy": 0.95}
        self.assertEqual(choose_on_validation([a, b])["name"], "a")

    def test_it_does_not_modify_the_runs(self):
        before = copy.deepcopy(RUNS)
        choose_on_validation(RUNS)
        self.assertEqual(RUNS, before)

    def test_an_empty_list_is_an_error(self):
        with self.assertRaises(ValueError):
            choose_on_validation([])


class TestTable(unittest.TestCase):
    def setUp(self):
        self.chosen = choose_on_validation(RUNS)
        self.lines = table_lines(RUNS, self.chosen)

    def test_a_header_and_one_line_per_run(self):
        self.assertEqual(len(self.lines), 1 + len(RUNS))

    def test_every_run_shows_both_numbers(self):
        for run, line in zip(RUNS, self.lines[1:]):
            self.assertIn(run["name"], line)
            self.assertIn(f"{run['val_accuracy']:.1%}", line)
            self.assertIn(f"{run['test_accuracy']:.1%}", line)

    def test_only_the_chosen_line_is_marked(self):
        marked = [l for l in self.lines if "<- chosen on validation" in l]
        self.assertEqual(len(marked), 1)
        self.assertIn("noaug_stop2", marked[0])


if __name__ == "__main__":
    unittest.main()