Theory
The honest-metrics rubric
A self-check before you write the report. Not one row here says "high accuracy": the rubric measures whether what you got is stated honestly.
The rubric
| # | what the report must say | good | weak |
|---|---|---|---|
| 1 | What you measure — how many images, how many classes, how many in data/ and test/ |
"400 new images, 100 each from four classes" | "accuracy 94 %" with no image count |
| 2 | Which number you publish | the new-image one, with ± and the image count |
the one that looks best, or the validation one |
| 3 | What you decided on | validation only; it is shown that test was not used for the decision |
"I chose the one that gave the best result" (which?) |
| 4 | How often test/ was looked at |
a concrete number, equal to the number of runs | not mentioned |
| 5 | What you tried — every run, including those that did not help | a table of all of them; the run that did not help is mentioned | only the winner shown |
| 6 | Where it goes wrong — a confusion matrix and at least three mistakes described | matrix + 3 files with an explanation | only the average accuracy |
| 7 | What it matters for — which mistake costs you more (precision or recall, for which class) | one concrete sentence | "both matter" |
| 8 | What is unresolved — what the model cannot do, what is unknown | a concrete limit ("no 'other' class", "20 photos per class") | "I will keep improving it" |
Five common distortions
None of them is deception; each is an ordinary accident that "improves" the number:
- Publishing the validation number instead of the new-image one. In lesson 11: 95.6 % against 93.5 %.
- The best epoch, picked on the same set you publish. On validation it will always look good.
test/used for choosing. Even "I just looked once to be sure" — if you changed anything afterwards, it is no longer atest.- The best of many runs, without saying there were many. In lesson 7 six runs with different seeds gave 87.2 % validation for the best, while their mean was 86.2 %.
- A single number without an error. 160 validation images give ±3.2 points; 400 new ones, ±2.4.
How to read your own results
- If validation and
testdiffer by less than the error — you have established no difference; say: "the difference fits inside the error". - If augmentation (or anything else) made a difference smaller than the error — do not rank. Choose by validation, but do not claim it is significant.
- If a class has only 20
test/images, then when its recall is about 90 % the error is ±13 points (the drills of lesson 11). - If the model does everything badly — that is also a report. Only with a matrix that shows where.
Ask yourself
If somebody who has not seen my data read only my report, could they repeat what I did, and would they know what to believe?