Jev Glossary
📚 Course home · llms.txt (lesson order, glossary, and citation terms).
Reference for the Jev course; figures verified 2026-10-19. The compressed vocabulary below is what every lesson reads from.
The model
- System One model — gives up text generation; returns typed, calibrated decisions about a supplied state (Kahneman’s fast judgment as an API)
- state — the shared context one request carries; every question sees exactly this text
- system — the exact input document a question ships with (the standardized input every judge reads in full)
- question / answer — one typed ask (noul, choice, score); answers keyed by question name
- criteria — the option/level/true-false definitions you author at request time; the model never invents labels
- type safety — shape guarantee, not a truth guarantee
The primitives
- Noul — probability a statement is true, 0–1; no separate confidence; near 0.5 = does not know
- Choice — one picked option from your option map, plus the full distribution
- Score — position on your ordered levels, 0-indexed; can land between levels
- confidence — certainty in the top answer; the calibration promise: tracks actual accuracy
- probabilities — full distribution over all options, sums to 1
- forced choice / abstention — a menu forces an answer even when the true answer is absent; your schema creates the escape hatch
Calibration and measurement
- calibration — stated probability matches observed accuracy; measured with a reliability curve
- RLCD — Reinforcement Learning for Calibrated Decisions; the vendor’s training objective for Jev
- abstention band — Noul 0.30–0.70; the uncertainty range routed to human review; the fraction landing there is a reported metric. For Choice and Score the equivalent is a predicate: low confidence, thin margin, or a torn distribution near a threshold — any one routes to review
- margin — a Choice’s winner probability minus the runner-up’s; the coin-flip detector. A decisive 0.85/0.15 and a torn 0.51/0.49 share the same winner label; the margin tells them apart. A recurring tie is a schema metric: the menu overlaps or lacks an escape hatch
- torn answer — an answer whose distribution is split across options (Choice near-tie) or adjacent levels (Score landing between levels): the model is genuinely unsettled, the top answer is arbitrary, and the routing decision belongs to review or the safe default
- reliability curve — predicted probability buckets vs observed accuracy; diagonal = perfectly calibrated
- Brier score — proper scoring rule: mean squared difference between stated probability and outcome; punishes confident wrong answers
- κ (Cohen’s kappa) — rater agreement beyond chance; the standard metric for judge-vs-human agreement on binary verdicts under balanced classes (collapses on rare-positive traffic: the κ paradox)
- MCC (Matthews correlation coefficient) — imbalance-robust correlation between prediction and truth over all four confusion cells; the agreement instrument for rare-positive (guardrail-shaped) traffic, with PR-AUC (precision-recall area under the curve), plus precision and recall at your operating threshold, where κ alone misleads
- OOD drift — probabilities quietly shift as inputs change; countermeasure: periodic re-measurement on labeled samples
- confusion matrix — the four buckets a binary judgment lands in against truth: hits (true positive), false alarms (false positive), misses (false negative), clean passes (true negative). Every metric is an arithmetic view of these four counts
- accuracy — The share of answers in the right bucket: (hits + clean passes) / everything. All four buckets count alike, so accuracy rewards the lazy judge on a lopsided workload — pair it with recall or precision before believing it
- precision — Of everything flagged, how much was real: hits / (hits + false alarms). It charges every cry of wolf; alone it cannot see the misses — a judge that flags one thing and skips the rest scores 100%
- recall — Of everything real, what the judge caught: hits / (hits + misses). It charges every miss; alone it rewards the censor that flags everything and burns the reader’s time
- F1 — The harmonic mean of precision and recall at one cut — a judge’s usefulness in one number that punishes either side falling to zero. Read it with the threshold sweep: F1 at the wrong cut hides a judge the right cut rescues
- cut — The threshold where a continuous answer becomes a decision: answers at or above it count as “yes”. The model owns the probabilities; the cut is yours, and moving it trades recall against precision along the sweep
- confidence interval (95%) — The range the same measurement lands in on 95% of repetitions of the data — bootstrapped by resampling the labeled items, no distribution assumed. Read the interval first: the point is a guess, the interval is its honest size
- tie rule — The pre-committed comparison reading: overlapping intervals are ties; only a disjoint interval supports a separation claim. Fixed before the data arrives, so “who won” cannot be re-litigated after
- ECE (expected calibration error) — The weighted gap between claimed confidence and achieved hit rate, binned over the model’s own claims. “80% sure” ten times should hit about 8 — ECE prices that distance; it says nothing about accuracy
- MAE (mean absolute error) — The average size of an error, ignoring direction, on scale answers. Absolute errors cannot cancel: two opposite errors still pay full price
- bias — The signed average error — positive leans consistently over, negative consistently under. MAE prices the size; bias names the direction of the lean
- keep-all signature — The fingerprint of a judge that filters nothing: pass-rate at the workload’s shape with κ at exactly 0, because chance agreement absorbs an unfiltered funnel. Near-zero κ on a lopsided workload is the signature’s neighborhood, not automatically its verdict
- Reading the Numbers — the page carrying these metric entries with worked formulas and reading rules; see /courses/jev-fundamentals/references/reading-the-numbers/
- arm — one model under one fixed serving condition (the venue, the context window, the pin); the controlled-comparison unit, from clinical-trial design: more than the weights, the model plus its serving conditions, so comparisons vary only the treatment
- silver labels — consensus labels from a panel of independent judge models, used as ground truth where owner labels are sparse; stronger than one model’s opinion, weaker than an expert’s read, and the gap against the owner’s own labels reports beside every number the labels feed
- pre-registration — the written plan committed before the first model call: tasks, models, serving conditions, pass bars, exclusion rules, and what counts as a failure; after the data arrives, judge the plan against it and keep the plan as written
- recorded skip — a planned model call that did not run, logged as a row in the capture cache with its full cause (the server error text, the byte count, the band): data about the model’s physical limit, not a missing value to impute; skips and hard failures both publish
- decision band — see abstention band; the general form: a pair of thresholds on the model’s own probabilities — answers inside the band route to human review, answers outside act; for Choice and Score the band is a predicate (low confidence, thin margin, torn distribution)
Patterns
- parallel fan-out — many independent questions in one request (13→1: 12.2× cheaper, 10.0× faster, vendor cookbook verified 2026-09-23)
- composite scoring — independent atomic questions combined with weights in code
- threshold routing — the same probabilities feed pass/review/block/support decisions under thresholds your code owns (vendor guardrails cookbook)
- capture-first — cache every raw answer before any analysis, so analysis can improve later without re-running the model
- interleaved timing (ABAB) — alternate the two conditions per measured unit so time-varying noise hits both equally and cancels
- decorrelation — two models making different mistakes; the architecture-diversity hypothesis behind a mixed judge panel
- receipt — the run’s own record of what actually served it: the log line beside every answer, carrying the versioned model ID from the response’s
modelfield, the run’s timestamp, and any routing metadata the venue exposes; comparisons are valid only between runs whose receipts agree. On local venues the receipt adds amodelDigest(a content hash standing in for the identity the versioned ID cannot pin) — the pin’s echo, second form - sweep — one pass of an instrument over a labeled item set. Two senses, both this course’s: (1) the collection sweep — one full pass collecting receipts, what a benchmark run does; (2) the threshold sweep — re-reading the same collected answers at many cut settings without re-asking the judge (S01E03). The context names which; collection sweeps cost model calls, threshold sweeps cost arithmetic
- wire contract — the request/response shape two programs agree on, independent of who serves them:
POST /v1/systemonewith astateand namedquestions; one contract, several servers (hosted API, aggregator, cloud gateways, local runtimes)
Sources
- Primitives TypeSafe AI docs · docs page · verified 2026-09-23
The concrete API surface the lessons compress: question types, response fields, fan-out economics, abstention guidance.
If an AI assistant is part of your workflow, the site’s llms.txt lists the
markdown twin of this glossary alongside every lesson; ask it to flashcard you
until every term restates cleanly.