Theory

The honest-metrics rubric

A self-check before you write the report. Not one row here says "high accuracy": the rubric measures whether what you got is stated honestly.

The rubric

# what the report must say good weak
1 What you measure — how many images, how many classes, how many in data/ and test/ "400 new images, 100 each from four classes" "accuracy 94 %" with no image count
2 Which number you publish the new-image one, with ± and the image count the one that looks best, or the validation one
3 What you decided on validation only; it is shown that test was not used for the decision "I chose the one that gave the best result" (which?)
4 How often test/ was looked at a concrete number, equal to the number of runs not mentioned
5 What you tried — every run, including those that did not help a table of all of them; the run that did not help is mentioned only the winner shown
6 Where it goes wrong — a confusion matrix and at least three mistakes described matrix + 3 files with an explanation only the average accuracy
7 What it matters for — which mistake costs you more (precision or recall, for which class) one concrete sentence "both matter"
8 What is unresolved — what the model cannot do, what is unknown a concrete limit ("no 'other' class", "20 photos per class") "I will keep improving it"

Five common distortions

None of them is deception; each is an ordinary accident that "improves" the number:

  1. Publishing the validation number instead of the new-image one. In lesson 11: 95.6 % against 93.5 %.
  2. The best epoch, picked on the same set you publish. On validation it will always look good.
  3. test/ used for choosing. Even "I just looked once to be sure" — if you changed anything afterwards, it is no longer a test.
  4. The best of many runs, without saying there were many. In lesson 7 six runs with different seeds gave 87.2 % validation for the best, while their mean was 86.2 %.
  5. A single number without an error. 160 validation images give ±3.2 points; 400 new ones, ±2.4.

How to read your own results

  • If validation and test differ by less than the error — you have established no difference; say: "the difference fits inside the error".
  • If augmentation (or anything else) made a difference smaller than the error — do not rank. Choose by validation, but do not claim it is significant.
  • If a class has only 20 test/ images, then when its recall is about 90 % the error is ±13 points (the drills of lesson 11).
  • If the model does everything badly — that is also a report. Only with a matrix that shows where.

Ask yourself

If somebody who has not seen my data read only my report, could they repeat what I did, and would they know what to believe?