# Lab 030 lab detail: examples aren't portable

The substrate behind Lab 030: how the bench works, how the two hosted System One readers were wired in,
every cell's numbers, what it cost, what went wrong, and what two rounds of peer review changed. It's
the raw detail, not the ruling; the lab page carries the verdict. The bench itself is published
separately at github.com/kltownsend/clinicbench (the instrument: harness, scorer, patients, driver
adapters and the six-example set, MIT license), without the author's method for assessing vendors.
Nothing in this lab is proprietary data: the patients are synthetic and the examples are public.
Machine addresses and key handling are left out.

The recorded conversations are public too. The 18 Jev and Bivia cells are on the bench's data-v3
release, released 2026-11-12. The Gemma and Haiku cells without examples, and Gemma's and Haiku's
payment cells with them, are on data-v2. Gemma's and Haiku's doctor and schedule cells with the
examples are on data-v4, also released 2026-11-12, as recorded and pinned in October. Each release's
manifest lists every archive's SHA-256.

## The bench

- **The Clinic's frozen bench.** The same patients, driver and scorer as Lab 029, frozen by hash. Every
  cell here carries instrument hash `0f3e1c09488db424`, the same as the Clinic's published same-freeze
  cells, so they compare part for part.
- **Patients are code.** 1,000 generated patients across Lab 23's 18 types. Each patient's truth is
  fixed before the call. One dimension at a time (payment, schedule or doctor) varies along clarity,
  probes to resolve and pressure: 48 settings, 48,000 conversations a cell.
- **Speech and hearing.** Template patient lines; the patient hears the agent through a deterministic
  parser. No model scores anything.
- **Outcomes.** Every write is checked against the patient's truth at that turn. A booking made while
  something it depended on was still open is a break (the page calls it a premature booking). A clean
  right booking is right with nothing open. Outcomes: right, right with no write (the patient reached a
  person, correctly), wrong, missed hand-off, gave up, no write.
- **Cohorts.** Clean right and right are counted over the patients who wanted a booking (23,220 doctor,
  27,840 payment, 22,020 schedule). Reaching a person is counted over the patients who needed one
  (20,520, 15,600, 21,720). Breaks and wrong writes are counted over all 48,000.
- **Estimator.** Paired 95% intervals from 1,000 resamples of whole patients, the explorer's own method.
  Each cell is one run; the intervals don't cover run-to-run variation.

## The agent and the readers

Lab 23's code-driven scheduling driver, vendored unchanged. When code can't parse a reply it asks the
reader a closed question, with the options it would accept plus "unclear." Arm E takes the reader's yes
as the patient's consent. Arm F decides consent in code: the reader may decline, never agree.

- **Jev (TypeSafe).** Model pinned to `jev-1.14.0`, through TypeSafe's System One API: one `choice`
  question per closed question, the driver's options plus "unclear." Jev has no abstain and no
  extraction type, so identity extraction returns nothing and the driver's own parser handles identity.
  Every response's model id was logged; all were `jev-1.14.0`.
- **Bivia (Articul8).** `/choice` in fast mode with `allow_abstain: true`; an abstain reads as "unclear."
  `/extract` for identity when the driver's parser finds none, with the date normalized by the driver's
  own parser. Every response's model id was logged; all were `dec-1`.
- **Gemma 4 26B and Claude Haiku 5.5.** The Clinic's general readers, unchanged: Gemma with reasoning
  off on owned DGX Sparks, Haiku with thinking off through Anthropic's API.
- **The six-example set (cb1).** Lab 024's six labeled cases per question, with the self-pay hedge
  relabeled unclear. Gemma and Haiku get them in the prompt. Neither System One API has an examples
  field, so for Jev and Bivia the cases go into the `state` field ahead of the case, each as the frozen
  prompt it was written in plus its labeled answer. Identity extraction gets no examples. That's an
  adaptation, not either vendor's tuning.
- **Build checks.** Each Jev and Bivia cell was built only after its venue record named the right reader
  and prompt, and every session's logged model id matched the cell's envelope.

## Venues

- **Jev and Bivia:** a Mac Studio running the bench harness (standard library Python only) against each
  vendor's hosted API, 8 conversations at once.
- **Gemma:** two owned NVIDIA DGX Sparks, 4 at once, as published in the Clinic.
- **Haiku:** Anthropic's API through LiteLLM, 16 at once, as published in the Clinic.

## Every cell

Outcome counts per cell, 48,000 conversations each. Clean right is over the patients who wanted a
booking; the other outcome columns are over all conversations.

| Cell | Reader | Examples | Arm | Dimension | Clean right | Right | Right, no write | Wrong | Missed hand-off | Gave up | No write | Breaks |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| `79dfaa3270e1` | Jev | none | E | doctor | 11,681 | 13,418 | 9,092 | 1,557 | 9,477 | 8,264 | 6,192 | 3,221 |
| `a77894d9136c` | Jev | none | E | payment | 16,799 | 17,521 | 4,021 | 1,675 | 9,828 | 9,856 | 5,099 | 2,157 |
| `c956af2aefd2` | Jev | none | E | schedule | 9,712 | 11,326 | 9,881 | 998 | 9,669 | 9,745 | 6,381 | 2,482 |
| `e0ed3b0076b1` | Jev | none | F | doctor | 5,503 | 5,545 | 9,961 | 0 | 9,568 | 8,501 | 14,425 | 42 |
| `e344bd36920d` | Jev | none | F | payment | 11,524 | 11,624 | 4,249 | 5 | 10,151 | 9,954 | 12,017 | 105 |
| `c9113ca7d167` | Jev | none | F | schedule | 3,112 | 3,152 | 10,379 | 0 | 9,781 | 10,002 | 14,686 | 40 |
| `eb704e3be7a9` | Jev | examples | E | doctor | 11,966 | 13,239 | 9,285 | 1,197 | 9,608 | 8,207 | 6,464 | 2,390 |
| `d89f7f88830b` | Jev | examples | E | payment | 16,816 | 17,669 | 5,666 | 1,509 | 8,113 | 10,051 | 4,992 | 2,122 |
| `207c07111fb9` | Jev | examples | E | schedule | 9,888 | 11,222 | 10,020 | 780 | 9,770 | 9,666 | 6,542 | 1,987 |
| `c2961d67add2` | Bivia | none | E | doctor | 11,548 | 13,303 | 7,402 | 1,851 | 9,497 | 10,426 | 5,521 | 3,532 |
| `0e1a9d6a7a8d` | Bivia | none | E | payment | 17,399 | 18,210 | 2,877 | 1,529 | 8,999 | 13,261 | 3,124 | 2,100 |
| `4a811860da8e` | Bivia | none | E | schedule | 10,343 | 12,097 | 7,889 | 1,536 | 9,830 | 12,147 | 4,501 | 3,153 |
| `3c4f75af72d0` | Bivia | none | F | doctor | 5,403 | 5,441 | 8,571 | 0 | 9,645 | 10,448 | 13,895 | 38 |
| `e71f9ddefd34` | Bivia | none | F | payment | 11,942 | 12,058 | 3,560 | 8 | 9,144 | 13,086 | 10,144 | 124 |
| `90b9f69a5ef6` | Bivia | none | F | schedule | 3,151 | 3,187 | 8,877 | 0 | 10,049 | 12,157 | 13,730 | 36 |
| `2c12e80422c0` | Bivia | examples | E | doctor | 12,223 | 12,927 | 8,853 | 575 | 9,654 | 9,178 | 6,813 | 1,217 |
| `dddfdab107aa` | Bivia | examples | E | payment | 16,973 | 17,815 | 5,643 | 595 | 7,176 | 11,571 | 5,200 | 1,293 |
| `9eaf9c5b3134` | Bivia | examples | E | schedule | 10,315 | 11,143 | 9,521 | 460 | 9,791 | 10,431 | 6,654 | 1,188 |
| `f10e944a632f` | Gemma 4 26B | none | E | doctor | 11,663 | 13,367 | 9,284 | 750 | 9,526 | 8,568 | 6,505 | 2,380 |
| `956b899678ca` | Gemma 4 26B | none | E | payment | 16,599 | 17,465 | 5,872 | 453 | 7,897 | 10,849 | 5,464 | 1,079 |
| `9288baca3a61` | Gemma 4 26B | none | E | schedule | 9,537 | 11,183 | 9,929 | 526 | 9,671 | 10,049 | 6,642 | 2,045 |
| `1bd20caca1cc` | Gemma 4 26B | none | F | doctor | 5,498 | 5,542 | 9,910 | 0 | 9,646 | 8,773 | 14,129 | 44 |
| `b1f4bb47f6a2` | Gemma 4 26B | none | F | payment | 11,281 | 11,404 | 6,458 | 2 | 7,975 | 10,703 | 11,458 | 125 |
| `20ee74942bb3` | Gemma 4 26B | none | F | schedule | 3,067 | 3,107 | 10,388 | 0 | 9,794 | 10,279 | 14,432 | 40 |
| `0ac1457c6fe0` | Gemma 4 26B | examples | E | doctor | 11,777 | 13,494 | 9,033 | 1,128 | 9,641 | 8,504 | 6,200 | 2,771 |
| `74a17c2f384b` | Gemma 4 26B | examples | E | payment | 16,911 | 17,783 | 5,834 | 851 | 7,943 | 10,588 | 5,001 | 1,483 |
| `cef8af8f2f3c` | Gemma 4 26B | examples | E | schedule | 9,840 | 11,495 | 9,638 | 678 | 9,830 | 10,135 | 6,224 | 2,203 |
| `ccd033952a52` | Haiku 5.5 | none | E | doctor | 12,302 | 13,024 | 9,356 | 236 | 9,674 | 8,485 | 7,225 | 873 |
| `862245175a90` | Haiku 5.5 | none | E | payment | 16,893 | 17,537 | 4,290 | 400 | 9,521 | 10,010 | 6,242 | 804 |
| `07096c18cd4e` | Haiku 5.5 | none | E | schedule | 10,180 | 10,951 | 9,730 | 213 | 9,870 | 9,976 | 7,260 | 856 |
| `76f6b2e93e83` | Haiku 5.5 | none | F | doctor | 5,592 | 5,638 | 9,816 | 0 | 9,611 | 8,599 | 14,336 | 46 |
| `27c5bad2a040` | Haiku 5.5 | none | F | payment | 11,551 | 11,672 | 4,596 | 10 | 9,686 | 10,006 | 12,030 | 131 |
| `8810a59f8b15` | Haiku 5.5 | none | F | schedule | 3,151 | 3,193 | 10,145 | 0 | 9,821 | 10,153 | 14,688 | 42 |
| `9a47a4f8ee6d` | Haiku 5.5 | examples | E | doctor | 12,292 | 12,805 | 9,478 | 304 | 9,660 | 8,352 | 7,401 | 732 |
| `9bf303a2ffac` | Haiku 5.5 | examples | E | payment | 16,939 | 17,506 | 4,091 | 596 | 9,768 | 9,940 | 6,099 | 923 |
| `acc9a980944b` | Haiku 5.5 | examples | E | schedule | 10,098 | 10,711 | 9,983 | 234 | 9,869 | 9,686 | 7,517 | 720 |

## The defective refusal line

When a patient turns down an offer that's wrong on the doctor or the appointment, the template says "No,
I can't do that one" without saying which part was wrong, and can then name the offered time as one that
works. Readers often read that as yes. It's left in by ruling and reported beside the results.

- **Yes rates on that line, arm E.** Gemma 11% without examples, 34% with; Bivia 52% and 22%; Jev 89% and
  74%; Haiku 3%, and 11% with examples.
- **The exclusion.** Every conversation where any of nine arm E reader configurations met the line
  (Gemma and Haiku with and without examples, Haiku with thinking on, Jev and Bivia with and without
  examples), dropped from every cell of that dimension: 1,062 doctor, 1,525 payment, 955 schedule.
- **What it moves.** With those conversations excluded, Gemma's rise in premature bookings disappears
  (0.0, 0.0 and 0.0 points), Bivia's payment drop shrinks from 1.7 to 0.3 points, and no reader's
  premature or wrong bookings rise with the examples. Clean right and reaching a person barely move.
- **A second defect** makes a patient who contradicts themselves sound like a clean correction. It's
  also left in.

## Cost and time

| | Jev | Bivia |
|---|---|---|
| Price | $0.042 per million input tokens | $0.042 per million billable tokens; first billion free |
| What was paid | $17.68, TypeSafe's bill | $0, free tier |
| Billing unit | input tokens | at least 1,000 tokens a call |
| Tokens, all 9 cells | about 402 million input | about 953 million billable (about 957 million with pilots and canaries) |

Per cell:

| Cell | Reader | Examples | Arm | Dimension | Model calls per conversation | Input tokens per conversation | Billable tokens | Run minutes shown |
|---|---|---|---|---|---|---|---|---|
| `79dfaa3270e1` | Jev | none | E | doctor | 1.28 | 585 | 0 | 20.4 |
| `a77894d9136c` | Jev | none | E | payment | 0.97 | 444 | 0 | 15.7 |
| `c956af2aefd2` | Jev | none | E | schedule | 1.29 | 613 | 0 | 20.9 |
| `e0ed3b0076b1` | Jev | none | F | doctor | 1.33 | 598 | 0 | 21.8 |
| `e344bd36920d` | Jev | none | F | payment | 1.02 | 456 | 0 | 16.7 |
| `c9113ca7d167` | Jev | none | F | schedule | 1.37 | 637 | 0 | 22.2 |
| `eb704e3be7a9` | Jev | examples | E | doctor | 1.25 | 1,743 | 0 | 21.0 |
| `d89f7f88830b` | Jev | examples | E | payment | 1.02 | 1,417 | 0 | 16.7 |
| `207c07111fb9` | Jev | examples | E | schedule | 1.26 | 1,876 | 0 | 20.9 |
| `c2961d67add2` | Bivia | none | E | doctor | 2.26 | 455 | 108,459,000 | 1092.0 |
| `0e1a9d6a7a8d` | Bivia | none | E | payment | 2.05 | 407 | 98,624,000 | 203.6 |
| `4a811860da8e` | Bivia | none | E | schedule | 2.31 | 475 | 111,091,000 | 129.9 |
| `3c4f75af72d0` | Bivia | none | F | doctor | 2.25 | 437 | 108,198,000 | 82.6 |
| `e71f9ddefd34` | Bivia | none | F | payment | 2.07 | 395 | 99,176,000 | 77.9 |
| `90b9f69a5ef6` | Bivia | none | F | schedule | 2.36 | 466 | 113,136,000 | 84.1 |
| `2c12e80422c0` | Bivia | examples | E | doctor | 2.01 | 1,546 | 105,772,791 | 103.7 |
| `dddfdab107aa` | Bivia | examples | E | payment | 1.80 | 1,310 | 95,275,819 | 98.1 |
| `9eaf9c5b3134` | Bivia | examples | E | schedule | 2.03 | 1,681 | 113,120,667 | 105.8 |

Billable tokens show 0 for Jev because Jev reports no billable count. The run minutes are what each
cell's record shows; for most Bivia cells they aren't Bivia's pace (see the gotchas).

## Gotchas

- **Bivia's free tier caps requests at 6,000 an hour** and answers HTTP 402 `entitlement_exceeded` past it.
  The limit isn't in the API reference. The first Bivia cell (E, doctor) hit it and then ran paced under
  it for 18 hours. At my request Articul8 raised the account's limit on November 10, 2026; after that, a
  cell of about 96,000 calls finished in under two hours.
- **Bivia's run times aren't its pace.** E doctor's minutes measure the cap. Five E cells shared one key
  from 18:35 UTC on November 10, and several cells were restarted, each resuming where it stopped. The
  cell-build step timed resumed cells across every attempt, waits included. F schedule ran once, alone,
  after the limit was raised: 84.1 minutes for 48,000 conversations. Each cell's note on the site says
  which case it is.
- **Bivia bills a 1,000-token floor.** Actual input averaged about 200 tokens a call without the
  examples and about 770 with them, so the billable count is several times the real input.
- **Bivia's `/extract` says `found: true` at low confidence instead of nothing.** In a canary it once
  returned a schedule sentence as a name; the driver's checks discarded it.
- **Jev can't abstain and has no extraction type.** Where the driver asks for identity, Jev's reader
  returns nothing found and the deterministic parser carries it.
- **A model id is a name, not a weight hash.** Both vendors' ids (`jev-1.14.0`, `dec-1`) were logged on
  every call and never changed, but nothing in either API promises the weights behind an id stay fixed.
- **Calls ran 0.97 to 1.37 per conversation for Jev and 1.80 to 2.36 for Bivia**, not the 4.6 a
  960-conversation Bivia pilot suggested. Plan cost and time from a full cell, not a pilot.

## After review: what changed and why

Gemini and ChatGPT reviewed the results and two drafts of the analysis, adversarially, before
publication.

- **Round one, on the Clinic draft.** Both found the examples lesson leaned on the defective refusal
  line, showed breaks alone, and called two delivery methods one knob. That led to the five-metric
  defect check, the "delivered two ways" framing, a narrower tuning claim, and rewritten Bivia timing
  notes after the run logs showed the first notes had misstated the restarts.
- **Round two, on the lab page.** Gemini found two metric errors in the draft. Reaching a person is counted
  over the patients who needed one, so a rise is a success, not a cost; the first page called it
  "handing off." And the page led with raw right bookings, which count lucky premature bookings; the
  Clinic's lead measure is clean right, on which Bivia rose 2.9 points on doctor. The page was rebuilt
  on breaks, wrong writes, clean right and reaching a person, each with its defect-excluded value.
  Both reviewers also asked for a narrower title; I kept "Examples aren't portable" and let the scope
  carry the narrowing.
- **What the reviews couldn't settle.** Whether the split is the model or the delivery, and whether
  arm F contains the examples for Jev and Bivia, which never ran arm F with them.