S01E04: Pins and Receipts (Keeping Your Setup Stable)

  • Level L2 (intermediate)|~20 min
  • prerequisites: S01E02S01E03
  • published 2026-10-04|verified 2026-10-04
  • jevpinningreceiptsportabilityrouting

A judge pipeline has a property no other service has: you cannot control what served. Remember the expense-report judge from S01E02, the one that approves or routes expense lines under thresholds you tuned against one build. If the platform answers with a different model, that judge keeps saying compliant, and you may be waving policy violations through: the name you send is not a promise, it is a request. This lesson covers what can change under that name: aliases that resolve at request time, limits that move with platform load, aggregators and gateways that may answer with a different model than the one you named. The habit that answers all of it has two halves: pin the versioned build, and log a receipt from every call. By the end you can pin a judge, verify what actually served every request, and carry both across venues without rewriting the request.

Aliases move: the name you send is not the model you get

An alias is a model name the server resolves to a concrete build at request time. A versioned ID names one build. The vendor's alias table lists jev-latest and jev-preview, both currently resolving to jev-1.13.0, and it states the mechanism in its own words: "an alias moves when a new release ships, so the answers behind it can change without a change on your side" (TypeSafe AI docs).

A model name can carry two different contracts. An alias is a pointer: jev-latest names whatever the vendor currently prefers, and the vendor repoints that pointer whenever a release ships, so the answers behind it can change with no change on your side. A versioned ID names one build, and only that build. One of the two names is your floor: with the alias, your code stands on a pointer the vendor moves at release time; with the versioned ID, the build only changes when you change it. And the vendor's ID answers "which build did I send" in one direction only: nothing your code can compute proves what answered, which is exactly why the verifying half lives in the response, not in your math.

The vendor's pinning advice reads exactly parallel: "if you have tuned confidence thresholds against a specific version, pin that version's ID instead of the alias and move to the new one on your own schedule." The reason is S01E03's: a threshold and a sweep are facts about the build that produced the scores. Applied silently to a newer build, the operating point stops matching reality, and nothing in your repo changes to tell you.

Pin and echo: the receipt habit

Pinning is the request half: the model field carries the versioned ID, so the request names the build you want. The echo is the verification half: the response's model field "reports the versioned ID that answered, so you can log which model produced each result."

🏷️ New term receipt· glossary

The run's own record of what actually served it: the log line beside every answer, carrying the versioned model ID from the response's model field, the run's timestamp, and any routing metadata the venue exposes when present. Comparisons are valid only between runs whose receipts agree.

The expense judge S01E02 built answers under the same keys, and the receipt's raw evidence sits beside them:

{
  "answers": {
    "receipt_referenced": 0.86,
    "within_cap": 0.93,
    "lane": { "value": "auto_approve", "confidence": 0.71,
              "probabilities": { "auto_approve": 0.71, "manager_review": 0.21,
                                 "finance_audit": 0.05, "other": 0.03 } },
    "policy_risk": { "value": 1.5, "confidence": 0.44,
                     "probabilities": [0.08, 0.44, 0.41, 0.07] }
  },
  "usage": {"input_tokens": 412},
  "model": "jev-1.13.0"
}

The request could have carried the alias jev-latest; the model field answers with the build that actually served. Whatever sits between your code and the weights, this field is where the serving truth lives; the next sections walk the places that can move it.

Log the receipt beside every answer, and compare only runs whose receipts agree. That rule earns its place because every measurement in this course is a comparison between runs: S01E02's repeat-run flip counts, S01E03's reliability curve and Brier score; none of them means anything unless one build produced both sides. A logged receipt is what makes that checkable after the fact, and it is one line of code.

YOUR CODE model: "jev-1.13.0" the pin: one versioned ID, not the alias POST /v1/systemone hosted API limits adjust dynamically aggregator load balance + fallbacks gateway / router evaluation fallback, lists RESPONSE answers + usage model: "jev-1.13.0" what actually served, in one field assert model = pin? match one request, any one venue mismatch: route like torn, then re-check the pin RECEIPT LOG the echo, one line beside every answer: #41 · "jev-1.13.0" · gateway
Figure 1. The pin-and-echo flow: one request, three possible venues, the assert gate on the model field, and the receipt log receiving the echo on match.

The same picture holds at every venue on the path: whatever sits between your code and the weights (a hosted API, an aggregator, a gateway, a router), the response always comes back with the one field that names what served. That field, logged beside the answer, is the receipt lane in Figure 1. The rest of this lesson walks the venues one by one; the lane never changes.

The page moves under your feet: dynamic limits

On the same models page, in the vendor's own words: "rate limits are adjusting dynamically," and "the limits above can change without notice." Hosted APIs share capacity across thousands of callers, and when traffic spikes, your rate limits drop. The consequence lands on measurement: p95 latency (the 95th-percentile request time) and the share of rate-limited responses (the 429s) are properties of the platform's load as much as of your code. A run measured on Monday and one measured on Friday can differ in ways your own diff never touches.

The countermeasure is one line of logging plus the standard retry discipline. A 429 is transient: honor retry-after (the response header that says how long to wait) and retry; a 401 is not: fix the request. And log the timestamp of every call, so a run is tied to the limits it faced: the timestamp rides the same receipt line as the model ID.

💡 Key insight

A measurement is only as good as the setup it holds fixed, and the setup is bigger than the model: build, limits, venue, and your own question wording all sit inside it. Receipts do not hold those parts still; they make each one visible. That is the next best thing, and the only realistic one.

Same name, different counter: aggregator providers

An aggregator buys breadth with one integration: one API key and one wire contract, many vendors' models behind it. The price is a second resolver on the path: the aggregator decides which provider answers the request. The aggregator's own docs concede the variance: default routing "load balance[s] requests across the top providers to maximize uptime"; the models fallback list triggers on errors including "rate-limiting, downtime" and moderation refusals. When a fallback fires, "requests are priced using the model that was ultimately used, which will be returned in the model attribute of the response body."

Read that as contract language: the platform reserves the right to answer with a different model than the one you named, and the response's model field is where it tells you. For a judge pipeline the consequence is direct: two identical requests can be served by different providers and builds, the answers can differ, and the only ground truth is the receipt your code does or does not log.

The same clause prices the fallback: a cheap decision model falling away under load means an expensive frontier model serving your judge, and the first place the swap surfaces is the bill. The receipt is what makes that traceable.

Portability: one wire contract, new counters

🏷️ New term wire contract· glossary

The request/response shape two programs agree on, independent of who serves them: POST /v1/systemone, a state, named questions: answered by answers, usage, and the response's model field. One contract, several servers. The contract covers the request shape and the one receipt field; the hop detail (provider routing metadata, pipeline metadata, fallback mechanics) is a per-venue extension returned on top of it: read the venue's docs for that part, and moving venues adds a metadata reader without rewriting how the contract itself is parsed.

Because the wire contract holds across venues, one judge request moves by re-pointing the base URL and the credential and keeping the request. Two receipts make the pattern concrete before Table 1 generalizes it. On the hosted API the expense judge logs "model": "jev-1.13.0". The same request on a local runtime (Ollama 0.35, the /v1/systemone endpoint on your own machine since 2026-09-29) logs "model": "nimble": a decision model from the local catalog, named in the same one field. The base URL and the credential changed; the request shape and the receipt did not. The full venue reference, per each venue's docs:

Table 1. The venues and their receipts: what each venue's docs show and how the response's model field reports what served there.
VenueWhat the docs showThe receipt there
TypeSafe hosted APIThe origin contract; the venue every lesson in this course works againstResponse model field: the versioned ID that answered
OpenRouterAdapted serving; fallbacks across models and providers (§4)Response model attribute names the model ultimately used; pipeline metadata per request (§6)
Cloudflare Workers AItypesafe/jev in the catalog; env.AI.run with state and questionsExample responses carry "model": "jev-1.13.0": the resolved build, in the one field the generic name never fills
Vercel AI GatewayTypeSafe-compatible endpoint; re-point the SDK base URL + credential, keep the requestResponse model + provider_metadata.gateway.routing; can also re-run an evaluation with a fallback model on a confidence condition you set, and names the model that served
Ollama 0.35 (local)The same /v1/systemone endpoint from your own machine (since 2026-09-29); decision models such as nimble in the catalogResponse model field carries through ("model": "nimble"); the receipt survives the loss of the network hop

Two rows carry teeth. Cloudflare's examples show the name you send (typesafe/jev) and the name that answers (jev-1.13.0) as different strings, so the receipt is the only place the resolved build appears. And Vercel documents something stronger than serving variance: a fallback evaluation, re-run by the gateway with another model when the primary's confidence misses a condition you set: the gateway names that model when it served. Both strengthen the same discipline, and it stays small: the first response from a new venue gets its receipt read before its answers act.

The platform grew a router

OpenRouter added a router built on Jev itself (listed 2026-09-25): typesafe/jev-router. Jev "reads the conversation and judges the task type, difficulty, and how much a stronger model would help," then routes to "the cheapest candidate that meets that bar," inside include and exclude lists (model IDs) that you pass. With the metadata header enabled, the response carries pipeline metadata naming the resolved models, the lists applied, and the tier served at.

Two list behaviors fail differently, and the difference is the lesson:

🔧 Fail-open versus fail-loud
  • an include list that matches nothing is ignored: the request proceeds on the whole pool, and the pipeline metadata reports list_fallback: "models_ignored": if it is read;
  • an exclude list that removes every candidate fails the request with a 404, instead of routing to a model you banned.

For a list you treat as policy, fail-loud is the half you can trust by itself, and the silent half is why the receipt rule follows the router too: log resolved_models the same way you log the build echo.

The scope line of the whole pattern: the router picks which model answers; your thresholds pick what to do with the answer. It reads as a clean separation, and it is one, as long as the receipt agrees with the pin. A threshold and a band are facts about the build that produced the scores (the reason §1 pins); if the router serves the request with a model other than the one pinned, your code is now comparing its fixed numbers against a different calibration. Re-run under a router, S01E02's expense judge steers policy decisions with thresholds built for another build, and the pin check is what catches it before the bill or the policy ruling does. That is why the receipt rule does not stop at logging: the echo is asserted against the pin, and a mismatch routes the response the way a torn answer does: re-check, or hold the decision. S01E03's band-as-router rule stays in your code; the platform router sits one layer below it, changes the model behind a response, and joins the same rule: any routing decision you did not write is one you verify.

The assertion in code is one if:

if (resp.model !== "jev-1.13.0") {   // the echo, asserted against the pin
  routeTorn();        // metric or hold; re-check the pin before the answers act
}

Verification is becoming infrastructure

Step back one level: the receipt habit is becoming public infrastructure. The UK AI Security Institute (AISI, the government lab that benchmarks frontier models) publishes verified evaluation results through EvalEval, a public registry of Evaluation Cards: what a third party needs to understand and re-run a reported evaluation. The 2026-09-22 release carries results, context, and configuration for five benchmarks, six frontier models, and two cyber evaluations, released alongside AISI's paper on how evaluation protocol and inference compute shape benchmark scores. The launch post (2026-06-11) measured the gap the cards close: of more than 50,000 collected records, 96.5% are missing at least one field needed to re-run them. That is this lesson's phenomenon at ecosystem scale: a score that omits its configuration is a promise without a receipt. And the cost of a missing receipt compounds: a number no third party can re-run cannot be cross-checked, so every downstream judge that adopts it inherits blind the very build drift the pin habit exists to catch.

The habit in six rules

The pin-echo habit compresses to six rules, wired in order.

  1. Pin per judge: the versioned ID in the request; the alias stays in the vendor's examples.
  2. Echo per call, asserted: the response's model field beside every answer, and in code: a check that the echo matches the pin (the build you believe is serving). Logged-and-never-compared is a receipt that only records; asserted-and-acted-on is the verification, and a mismatch is a metric or a hold condition, not a silent field.
  3. Receipt per hop: the aggregator's model attribute, the gateway's routing metadata, the router's pipeline metadata, the local runtime's model field: each hop names what served, and each name is logged.
  4. Gate the venue: the first response from a new venue must come back named by the right build before its answers act. The gate is not the habit; the per-call assertion (rule 2) is, because aggregators and gateways can fall back mid-stream without the venue changing.
  5. Re-verify on drift: any change, wording or input mix (S01E02's repeat-run procedure and S01E03's re-check rule), a new model ID, or a new venue, makes the first response after it evidence, read before the change is treated as safe.
  6. Lists and limits are configuration, and lists are not controls: include lists, exclude lists, thresholds, and band widths are named policies you version. That fail-open behavior (router rules above) means an include list is not the boundary that keeps requests on approved models; the boundary lives in the per-call assertion (rule 2) and the metadata that names resolved models. A failed match must be visible, not survived.

Knowledge check

📖 This knowledge check is an interactive quiz (8 questions). Open this page in a browser to take it; reader mode shows only the prose.

Sources & further reading

  • Models & aliases TypeSafe AI docs · docs page · verified 2026-10-04

    The alias table, the pinning advice, the dynamically-adjusting limits, and the response model field: the primary source behind every section above.

  • Model Fallbacks OpenRouter docs · docs page · verified 2026-10-04

    Provider load balancing, fallback triggers, and the model attribute that names what actually served.

  • Jev Router OpenRouter docs · docs page · verified 2026-10-04

    The router built on Jev, with include/exclude lists and the pipeline metadata (including list_fallback).

  • Jev on Cloudflare Workers AI Cloudflare docs · catalog page · verified 2026-10-04

    typesafe/jev in the catalog; example responses carrying the resolved build in the model field.

  • TypeSafe API with AI Gateway Vercel docs · docs page · verified 2026-10-04

    The TypeSafe-compatible endpoint, routing metadata, and the gateway's evaluation fallbacks.

  • Ollama 0.35 decision models Ollama blog · release post · verified 2026-10-04

    The /v1/systemone endpoint on your own machine since 2026-09-29, with the receipt carrying through.

  • How UK AISI and EvalEval Are Making Benchmark Results Reproducible Hugging Face blog · published 2026-09-22 · verified 2026-10-04

    Verified evaluation results with configuration and provenance, as public infrastructure.

  • Evaluation Cards launch Hugging Face blog · published 2026-06-11 · verified 2026-10-04

    The 96.5% missing-field statistic behind the section above.

🧭 Practice

Your turn. Write the pin policy for one judge your code calls: the pinned versioned ID (not the alias), the receipt that travels with every answer (the response's model field, plus whatever hop metadata sits between your code and the judge), and the list of changes that force a re-verify. Name each one: new wording, new input mix, new model ID, or a new venue. Write the three down as IDs and field names, not adjectives.

If an AI assistant is part of your workflow, Copy page in the header menu hands it this whole lesson as markdown, citation included. Ask it to audit the pin policy you wrote, or to read the receipt rules back against the venues table. The glossary is the canonical vocabulary when terms blur.

An example policy, for self-grading
judge: expense-policy-check (S01E02's expense judge)
pin: model = "jev-1.13.0"; the alias stays in the vendor's docs
receipt: response.model + run timestamp, one line per answer
assert: response.model == pin; mismatch: route torn, re-check the pin
re-verify on: reworded questions / new input mix / new model ID / new venue

See also