Reading the Numbers

You are reading a scorecard and it says κ 0.085, ECE 0.32, F1 66.7%. Whether those numbers praise the model or warn you off is pure reading skill, and this reference is that skill, one metric at a time. Every entry follows the same four lines: intuition (the plain-English one-liner and which cells of the grid it reads), formula and worked toy (the display equation, numbered and consistent across the whole course), when it lies (the failure mode that catches people), and how to read it (the rule the benchmark holds itself to). The audience is deliberately the software engineer integrating a judged pipeline — each entry opens with the deployment decision the metric answers, not the statistics. If you already think in confusion matrices, this page is your lookup layer: jump straight to the entry you need.

Everything on this page is arithmetic on four numbers. One frozen toy runs the whole reference: a censor screening files for secrets judged 20 files in one labeled run — 10 carrying secrets, 10 clean. It flagged 8 files, of which 6 were real. It is a teaching toy: the benchmark's real, interval-carrying numbers live in the lessons, and S01E05's report is where they land.

The one grid everything reads from

The glossary defines a confusion matrix as the four buckets a binary judgment lands in against truth: hits (true positive), false alarms (false positive), misses (false negative), clean passes (true negative). Every metric below is a different fraction of these four counts — nothing richer ever arrives at a metric, and every dispute between two summary metrics resolves by walking back here:

Table 1. The 2×2 grid this reference is built on. TP + FP + FN + TN = 20.
Actual: secret (positive)Actual: clean (negative)
Flagged (predicted positive)True Positive (TP) = 6False Positive (FP) = 2
Passed (predicted negative)False Negative (FN) = 4True Negative (TN) = 8
Actual: secret (+) Actual: clean (−) Flagged (predicted +) Passed (predicted −) TP = 6 a hit FP = 2 a false alarm FN = 4 a miss TN = 8 a clean pass Precision TP + FP = 8 reads the flagged row Recall TP + FN = 10 reads the actual-secret column Accuracy (6 + 8) of 20 reads the outlined diagonal κ uses the same four counts, read against the agreement chance would produce — the κ entry below carries the full reading
Figure 1. How the three headline cuts read the same grid: precision's denominator is the flagged row of the matrix, recall's is the actual-secret column, and accuracy's is the diagonal.

The counting is the cheap half; the ground-truth column (who actually carried a secret) is the expensive one — the labeling work that makes every metric here possible. Figure 1 lays out the grid's three headline reads; every entry below walks back to it. Four definitions already live in the glossary and are not redefined below: κ (Cohen's kappa), Brier score, the reliability curve, and PR-AUC for threshold-free reading. Equations carry the numbers (1)–(12), and S01E05 re-displays the same numbers: a citation travels between this page and the lessons intact — one number, many surfaces. The numbers are stable names, not a reading order: a course page may cite (12) before (7), because the lessons meet formulas in their own narrative order — the number is the identity.

Confusion matrix

The four buckets a binary judgment lands in against truth: hits (true positive), false alarms (false positive), misses (false negative), clean passes (true negative). Every metric is an arithmetic view of these four counts.

N = TP + FN + FP + TN.(1)

Worked toy: on Table 1, 6 + 4 + 2 + 8 = 20 — the equation is just the grid audited against the dataset size.

When it lies: it doesn't. The matrix is raw ground truth, not a summary; when two summary metrics contradict each other, look back at the four numbers to see what actually happened.

How to read it: print the 2×2 beside any summary anyone hands you. Every dispute resolves at the counts — it is the only table on this page that is evidence rather than arithmetic on evidence.

Accuracy

The share of answers in the right bucket: (hits + clean passes) / everything. All four buckets count alike, so accuracy rewards the lazy judge on a lopsided workload — pair it with recall or precision before believing it.

Accuracy = (TP + TN) / N.(2)

Worked toy: the two correct quadrants hold 6 + 8 = 14 of 20, so accuracy = 14 / 20 = 0.70. Flattened twice: on the balanced 10/10 grid a flag-nothing censor and a flag-everything censor each read exactly 0.50 — one for missing everything, one for crying wolf on every clean file.

The deployment decision it answers: none, safely, by itself. If 90% of your test files are clean, a censor hard-coded to say "clean" achieves 90% accuracy while doing zero work — it would pass your accuracy gate outright. The class balance is a fact about your workload, not the model.

When it lies: imbalanced workloads reward the lazy or biased judge.

How to read it: never alone; on guardrail-shaped workloads (few true positives), read κ and precision/recall instead — and remember the keep-all signature.

Precision

Of everything flagged, how much was real: hits / (hits + false alarms). It charges every cry of wolf; alone it cannot see the misses — a judge that flags one thing and skips the rest scores 100%. Reads the flagged column of the grid — the flagged pool is the denominator's cost.

Precision = TP / (TP + FP).(3)

Worked toy: the flagged column holds 8 files and 6 of them were real, so precision = 6 / (6 + 2) = 0.75. The denominator is the flag pool — the quadrant the censor itself fills — which is why precision falls as the censor over-flags and rises as it goes quiet.

The deployment decision it answers: alert fatigue. If an engineer gets 10 censor alerts a day and 8 are false alarms, they stop trusting the alert queue — low precision is the number that forecasts that.

The trap: precision is recall-blind. A censor that flags exactly one file — and happens to get it right — scores a perfect 1.00 while missing the other nine secrets. Precision never sees the quadrant it did not fill.

How to read it: report it beside recall, with the flag count named. A precision claim without its denominator (the flagged pool) proves nothing about coverage.

Recall

Of everything real, what the judge caught: hits / (hits + misses). It charges every miss; alone it rewards the censor that flags everything. Reads the actual-positive row of the grid — the missed-pool is the denominator's cost.

Recall = TP / (TP + FN).(4)

Worked toy: the actual-secret row holds 10 files and the censor caught 6, so recall = 6 / (6 + 4) = 0.60. The denominator is the true-positive pool, so the clean column is invisible to it by construction.

The deployment decision it answers: leaked secrets. In security (or compliance) review the false negative is the catastrophic class — the secret that ships. Low recall is the number that forecasts that breach.

The trap: recall rewards over-prediction. A censor that flags every file reads a perfect 1.00 while filling the false-positive quadrant; flag-nothing reads 0.00. Neither number says whether the system is operable.

When it lies: aggregate recall hides per-kind failure. S01E05's report found one arm blind to a whole secret kind (auth tokens) while its overall recall looked acceptable — keep per-kind recall where the truth matters at the kind level.

How to read it: the pair (precision, recall) travels together, never either alone.

F1

The harmonic mean of precision and recall at one cut — a judge's usefulness in one number that punishes either side falling to zero. Reads the same flagged column and positive row as precision and recall.

F1 = 2TP / (2TP + FP + FN).(5)

Worked toy: both forms on the grid agree — from the counts, F1 = 2 · 6 / (12 + 2 + 4) = 12 / 18 ≈ 0.667; from the pair, 2 · 0.75 · 0.60 / (0.75 + 0.60) = 0.90 / 1.35 ≈ 0.667. That agreement is the harmonic mean's whole point: a judge strong on one side and idle on the other is held to the idle side (100% precision with 0% recall scores 0, not 0.50).

When it lies: it hides the cut choice. S01E05's worked case: an arm's keep-scores sat under the 0.5 cut so hard F1 read 50% there; the threshold sweep rescued 70% at 0.35; after the answer definitions were corrected the scores separated on their own — 73.1% at the original cut, nothing tuned.

How to read it: report F1 at the cut WITH the sweep over cuts — or the operating point is a guess wearing a number.

Cut

The threshold where a continuous answer becomes a decision: answers at or above it count as "yes". The model owns the probabilities; the cut is yours, and moving it trades recall against precision along the sweep.

flagi ⇔ scorei ≥ t.(6)

where i runs over a judged run and t is the threshold you own.

Worked toy: eight of the 20 scored answers sit at or above the 0.5 cut and get flagged. Push the cut to 0.7: only two true positives remain above it while both false positives now sit below — flags 2, hits 2, precision 1.00, recall 0.20. Nothing about the censor changed between the two reads; only the cut moved, and with it the whole trade you own.

The deployment decision it answers: your guardrail thresholds. S01E03's routing thresholds and abstention bands are this line — if you tuned one against a build, the receipt habit (S01E04) protects the match between cut and build.

How to read it: never report a cut-bound number without its cut. Sweep before believing any operating point — a sweep is a re-read of captured scores, not a re-run of the benchmark (capture-first makes it free).

Confidence interval (95%)

The range the same measurement lands in on 95% of repetitions of the data — bootstrapped by resampling the labeled items with replacement, no distribution assumed. Read the interval first: the point is a guess, the interval is its honest size.

CI95 = [ κ̂ at the 2.5th percentile , κ̂ at the 97.5th percentile ].(7)

where κ̂ is the κ recomputed on one bootstrap resample.

Worked toy: resample the toy run's 20 labels at random 10,000 times, recompute κ on each resample, sort, and read the 250th and 9750th values: toy κ = 0.40 lands in [0.15, 0.65]. A model scoring 0.85 on 20 evaluation samples might behave anywhere between roughly 0.60 and 0.95 in deployment — that is what small n means.

The deployment decision it answers: how much to trust a short eval before you build on it. Interval width scales with 1/√n: the same κ on 500 items tightens about five-fold. A wide interval is not failure; it is small n speaking — and the honest report prints it.

When it lies: narrow intervals from cherry-picked subsets are the dishonest version; a wide one printed plainly is honest.

How to read it: never compare points without intervals — and never declare Model A "better" than Model B unless their intervals are fully disjoint. The tie rule below fixes "disjoint" as arithmetic.

Tie rule

The pre-committed comparison reading: overlapping intervals are ties; only a disjoint interval supports a separation claim. Fixed before the data arrives, so "who won" cannot be re-litigated after.

separate ⇔ A's high end < B's low end, or B's high end < A's low end.(8)

where A and B are two judges' intervals on the same task.

Worked toy: judge A reads 0.40 [0.25, 0.55], judge B reads 0.43 [0.28, 0.58]. A's high end (0.55) sits above B's low end (0.28): the intervals overlap and the reading is a tie — however big the 0.03 gap in B's favor looks in a headline.

When it lies: it cannot — what it can do is disappoint. Wide-interval ties answer less than a vendor's launch post wants; that is the rule working, not failing.

How to read it: commit the comparison reading before the data arrives (the benchmark's pre-registered rules); the family matters too — S01E05's S1 intervals all overlap, so its arms are a family, not a ranking, and no purchase decision can be priced off the point estimates.

ECE (expected calibration error)

The weighted gap between claimed confidence and achieved hit rate, binned over the model's own claims. "80% sure" ten times should hit about 8 — ECE prices that distance; it says nothing about accuracy.

ECE = Σb (nb / N) · |confb − hitb|.(9)

where nb is the count in confidence bin b, N the total answers, confb that bin's average claim, and hitb its achieved hit rate.

Worked toy: one judge answers 20 times. Ten answers claim 0.8 and seven hit — that bin over-claims by |0.8 − 0.7| = 0.1; ten claim 0.3 and four hit — a 0.1 gap again. Each bin pays its gap, weighted by its share of the answers: the toy reads ECE = 0.10, systematic over-prediction of confidence.

The deployment decision it answers: is the band router load-bearing? S01E03's routing bands act on the model's own claims — if the claims do not predict the hits, the band routes on nothing and every threshold you set on confidence is fiction. The benchmark gates band routing on a pre-registered bar: ECE ≤ 0.15 on both answer types, resolved conservatively (both must pass).

The real caution, for calibration of your reading: S01E05's quiet-confabulation finding — wrong answers that still claimed 0.43–0.54 confidence on every arm. ECE below the bar is necessary, never sufficient: wrong-and-confident is the failure mode a band router trusts most.

When it lies: few bins or few items make ECE noisy — read it beside the reliability curve; and it prices the claims' gap only, never correctness (a precise-but-wrong judge scores 0 while being useless).

How to read it: gate the band router on it; keep the Brier score as the companion (both live in the glossary).

MAE (mean absolute error) and bias

MAE — the average size of an error, ignoring direction, on scale answers. Absolute errors cannot cancel: two opposite errors still pay full price. Bias — the signed average error: positive leans consistently over, negative consistently under. MAE prices the size; bias names the direction of the lean.

MAE = (1/n) · Σ |ŷi − yi|.(10)

where ŷi is prediction i, yi its truth, and n the number of scale answers.

Bias = (1/n) · Σ (ŷi − yi).(11)

Worked toy: a three-item urgency toy with truths (1, 1, 1). Judge A answers (2, 0, 1): errors +1, −1, 0 — the signed average would cancel to 0 and hide all of it, the absolute average reads 2/3. Judge B answers (2, 2, 1): errors +1, +1, 0, leaning +0.67 — over-predicting urgency on every item. Same MAE, opposite deployments.

The deployment decision it answers: which countermeasure to ship. A leaning judge tightens downward; an eager one (positive bias) needs thresholds raised, a sluggish one (negative) lowered — the sign chooses the fix; MAE alone cannot even name the problem.

The real case, for calibration of your reading: S01E05's urgency column printed both — 0.53/+0.37 on an eager small arm, 0.65/−0.39 on a sluggish one, 0.36/+0.22 on the hosted reference. Same magnitude, opposite fixes.

When it lies: a near-zero bias can hide two opposite sub-populations that cancel — break the groups before trusting the aggregate.

How to read it: the pair travels together: MAE for size, bias for direction, never one instead of the other.

κ (Cohen's kappa) and the keep-all signature

κ — rater agreement beyond chance — is the standard instrument for judge-vs-human agreement on binary verdicts; the keep-all signature is the fingerprint of a judge that filters nothing: pass-rate at the workload's shape with κ at exactly 0, because chance agreement absorbs an unfiltered funnel.

κ = (po − pe) / (1 − pe).(12)

where po is observed agreement and pe what chance alone would agree to given the workload's shape.

Worked toy: the toy's flag-nothing censor keeps every file. Of its 20 judgments the 10 correct keeps are its only agreements: po = 10/20 = 0.50, and chance explains exactly 0.50 of it — the subtraction leaves κ = 0.000 while accuracy read 50% and looked like a judge doing its job.

The deployment decision it answers: does the funnel filter at all? A curation funnel that keeps everything has no operating cost advantage over no funnel; a pass rate alone cannot tell you — a low keep-rate could equally mean a tight, useful sieve or an over-eager one.

The real case, for calibration of your reading: on S01E05's 23-keep/29-split workload, one small arm kept all 52 items: observed agreement 23/52 ≈ 0.44, chance agreement 0.44, κ = 0.000 — the signature in the wild, one ten-thousandth from its definition. Another arm's κ read 0.085 beside a 47-of-52 keep rate: barely past the funnel, not judgment.

When it lies: conversely — a near-zero κ on an imbalanced workload is the signature's neighborhood, not automatically its verdict; read the keep-rate next to it before calling a model lazy.

How to read it: pass-rate and κ together, on imbalanced workloads instead of accuracy. The pair, one arithmetic step apart, is the lazy-judge detector.

Assembling a scorecard row

A scorecard row is not a list of six numbers; it is a production order. Read it in four passes, and let the pass structure — not the number order — decide what you believe:

Table 2. Assembling a scorecard row: the question each column answers and its reading rule.
columnthe question it answersits reading rule
κ with 95% CIjudgment past the workload's shape?interval first; overlaps tie
accuracyhow often right, raw?never alone; pair with recall or precision
precision / recall (or F1)flags mean something / catches the real ones?always paired; at the chosen cut
ECE (Brier companion)probabilities load-bearing?the band-as-router rule stands here
MAE / biasscale error size and direction?sign = deployment posture
unknown-rate / conf-wrongabstains where it should; confidence honest?the honesty pair, read together

The first pass (κ + CI) decides whether the row is readable at all: a keep-all signature or a family of overlapping intervals means there is no ranking to read further down, only a description. Only a row that survives that pass earns the per-task precision/recall reading in second place; the calibration pair (ECE + honesty) gates what you may automate; the MAE/bias pair sets your countermeasure.

Beyond two rows. The toy grid is binary — every entry above divides one flagged axis and one true axis in half. The season's S1 task is already one grade up: three classes (act, review, unknown), which the benchmark reads the multi-class way — the same precision/recall arithmetic run once per class, then averaged equally across classes (macro-averaging), so a large healthy class cannot drown a small collapsing one. The reading rules survive the upgrade unchanged: the average is the summary, the per-class lines are the finding, which is why S3's weak-cell column, not its aggregate, tells you which harm a particular arm misreads in production.

🧭 Practice

A censor flags 8 of 10 real secrets and also flags 3 of 20 clean files. Compute precision, recall, accuracy, and F1 by hand from this reference — then name the reading rule each number needs.

Step-by-step solution (expand only after trying it yourself)
  1. Fill the grid first. The dataset is 30 files: 10 secrets, 20 clean. Flagged-and-real: TP = 8. Flagged-and-clean: FP = 3. Real-but-passed: FN = 10 − 8 = 2. Clean-and-passed: TN = 20 − 3 = 17. Check: 8 + 3 + 2 + 17 = 30. (Every metric is arithmetic on these four counts.)
  2. Precision (3) = TP / (TP + FP) = 8 / (8 + 3) = 8/11 ≈ 0.73. Reading rule: report it beside recall, with the flag count named.
  3. Recall (4) = TP / (TP + FN) = 8 / (8 + 2) = 0.80. Reading rule: never alone — pair it with precision.
  4. Accuracy (2) = (TP + TN) / N = (8 + 17) / 30 = 25/30 ≈ 0.83. Reading rule: never read alone — two-thirds of the files are clean, so the workload leans in the censor's favor; check κ.
  5. F1 (5) = 2TP / (2TP + FP + FN) = 16 / (16 + 3 + 2) = 16/21 ≈ 0.76. Reading rule: report at the chosen cut, with the sweep over cuts.

Sources & further reading

  • Cohen's κ (1960) Wikipedia · encyclopedia article · verified 2026-10-16

    A coefficient of agreement for nominal scales — the original paper, JASA 55(282); the Wikipedia page links it.

  • McNemar's test (1947) Wikipedia · encyclopedia article · verified 2026-10-16

    The paired-flip statistic for before/after judgments; also carried by the glossary.

  • Guo et al. (2017), On Calibration of Modern Neural Networks arXiv · preprint · verified 2026-10-16

    ECE and the reliability diagram, the calibration pair used here.

  • Brier (1950) Wikipedia · encyclopedia article · verified 2026-10-16

    Verification of forecasts expressed in terms of probability — the Brier score's origin (Monthly Weather Review 78(1)).

See also