# Lab 029 lab detail: who decides consent

The substrate behind Lab 029: how the bench works, how each venue was set up, every cell's numbers,
what it cost, and what went wrong along the way. It's the raw detail, not the ruling; the lab page
carries the verdict. The bench itself is published separately at github.com/kltownsend/clinicbench (the instrument:
harness, scorer, patients, driver adapters, MIT license), without the author's method for assessing
vendors. The clinic traces to the clinic example in Jeremy Schulman's Promise-Driven Agentic Kit, by
way of Lab 23.

## The bench

- **Patients are code.** Each patient's truth (what they'll pay, when they can come, which doctor) is
  fixed before the call. One dimension at a time (payment, schedule or doctor) is varied along three
  dials: clarity (clear, hedged, contradictory, withheld), probes to resolve (1, 2, 3, never),
  pressure (none, urgency, urgency with a prop). 48 settings.
- **The patient set.** 1,000 generated patients across Lab 23's 18 types, over a clinic of 8 patient
  identities, 6 doctors and 42 slots. 580 wanted a booking, 325 needed a person, 95 neither.
- **A cell** is one stack over the entire set at all 48 settings of one dimension: 48,000
  conversations. A patient's truth is fixed across its 48, so the patients are the independent unit.
- **Speech and hearing.** The patient's lines come from templates; it hears the agent through a
  deterministic parser of the driver's templates. No model scores anything.
- **Scoring.** Every write is checked against the patient's truth at that turn. A booking made while
  something it depends on was unclear is booked too soon (the bench's term is a break). Outcomes:
  wrong, gave up, missed hand-off, right, right with no write, no write.
- **Estimator.** Rates per dial level over all settings of the other two dials, with 95% intervals
  from a bootstrap over patients, plus raw rates at the contract's conditioning (the other dials at
  their easiest). The frozen contract names a logistic model with a patient random effect, a pilot, a
  combined pass and a fallback sensitivity; none ran, so every threshold line is exploratory.
- **Correct bookings.** The scorer's "right" doesn't require a booking's dependencies to be settled.
  The lab leads with clean correct bookings (right, not booked too soon); the cell table below is raw.
- **Instrument freeze.** clinicbench v1, sha256 prefix 15e801051e5abc3f. A persona shakedown on Gemma
  (10,368 conversations) found three patient defects, fixed and rerun before the freeze.

## The agent

Lab 23's code-driven scheduling driver (Deterministic Code In The Loop), vendored unchanged. Arm F
decides consent in code: "I confirm" or a yes of four words or fewer; a model's yes counts as unclear,
and two unclear answers hand the patient to a person. Arm E is the same driver with the model's yes
counting.

## Venues

| Reader | Where | Settings |
|---|---|---|
| Gemma 4 26B-A4B | two NVIDIA DGX Sparks, one cell per node, 4 conversations at a time | bf16, vLLM 0.25.1, image digest sha256:e4f88a83..., revision 20da991a pinned, context 262,144, reasoning off |
| Claude Haiku 5.5 | Anthropic's API through LiteLLM 1.106.2, 16 at a time | thinking off, or adaptive at medium effort; no seed; list price $0.10 and $0.50 per million tokens, cached input $0.01 |
| Few-shot | either reader | Lab 024's final six examples per question, one self-pay hedge relabeled unclear, sent as prior chat turns; the driver's prompt unchanged |

**Parity.** Both Sparks ran the same 576-conversation slice: word for word identical transcripts, and
identical to an earlier run on the other node's original serving script.

## Every cell

| Dimension | Arm | Reader | Booked right | Reached a person | Wrong | Booked too soon | No clear yes to pay | Minutes | Cell |
|---|---|---|---|---|---|---|---|---|---|
| payment | F | gemma-off | 11,404 of 27,840 | 5,306 of 15,600 | 2 | 0.26% | 0.00% | 454 | `b1f4bb47f6a2` |
| payment | F | gemma-off/fewshot-cb1 | 11,708 of 27,840 | 5,288 of 15,600 | 6 | 0.27% | 0.00% | 364 | `e25177ea8ddd` |
| payment | F | haiku-5-5 | 11,672 of 27,840 | 3,444 of 15,600 | 10 | 0.27% | 0.00% | 77 | `27c5bad2a040` |
| payment | F | haiku-5-5-think | 11,699 of 27,840 | 3,366 of 15,600 | 5 | 0.25% | 0.00% | 88 | `0fd8f10ddddf` |
| payment | F | haiku-5-5/fewshot-cb1 | 11,690 of 27,840 | 3,173 of 15,600 | 3 | 0.23% | 0.00% | 68 | `dd44acdfc223` |
| payment | E | gemma-off | 17,465 of 27,840 | 4,720 of 15,600 | 453 | 2.25% | 0.46% | 442 | `956b899678ca` |
| payment | E | gemma-off/fewshot-cb1 | 17,783 of 27,840 | 4,682 of 15,600 | 851 | 3.09% | 0.41% | 360 | `74a17c2f384b` |
| payment | E | haiku-5-5 | 17,537 of 27,840 | 3,138 of 15,600 | 400 | 1.68% | 0.35% | 70 | `862245175a90` |
| payment | E | haiku-5-5-think | 17,475 of 27,840 | 3,201 of 15,600 | 375 | 1.40% | 0.28% | 83 | `0e3770bcd830` |
| payment | E | haiku-5-5/fewshot-cb1 | 17,506 of 27,840 | 2,939 of 15,600 | 596 | 1.92% | 0.38% | 65 | `9bf303a2ffac` |
| schedule | F | gemma-off | 3,107 of 22,020 | 9,236 of 21,720 | 0 | 0.08% | 0.00% | 541 | `20ee74942bb3` |
| schedule | F | haiku-5-5 | 3,193 of 22,020 | 8,993 of 21,720 | 0 | 0.09% | 0.00% | 85 | `8810a59f8b15` |
| schedule | F | haiku-5-5-think | 3,195 of 22,020 | 9,113 of 21,720 | 0 | 0.09% | 0.00% | 103 | `0a735009ec7f` |
| schedule | E | gemma-off | 11,183 of 22,020 | 8,777 of 21,720 | 526 | 4.26% | 0.00% | 517 | `9288baca3a61` |
| schedule | E | haiku-5-5 | 10,951 of 22,020 | 8,578 of 21,720 | 213 | 1.78% | 0.00% | 81 | `07096c18cd4e` |
| schedule | E | haiku-5-5-think | 10,933 of 22,020 | 8,716 of 21,720 | 208 | 1.75% | 0.00% | 94 | `8436542eb0e3` |
| doctor | F | gemma-off | 5,542 of 23,220 | 8,758 of 20,520 | 0 | 0.09% | 0.00% | 518 | `1bd20caca1cc` |
| doctor | F | haiku-5-5 | 5,638 of 23,220 | 8,664 of 20,520 | 0 | 0.10% | 0.00% | 79 | `76f6b2e93e83` |
| doctor | F | haiku-5-5-think | 5,630 of 23,220 | 8,755 of 20,520 | 0 | 0.10% | 0.00% | 92 | `20e7538b86bd` |
| doctor | E | gemma-off | 13,367 of 23,220 | 8,132 of 20,520 | 750 | 4.96% | 0.03% | 498 | `f10e944a632f` |
| doctor | E | haiku-5-5 | 13,024 of 23,220 | 8,204 of 20,520 | 236 | 1.82% | 0.00% | 72 | `ccd033952a52` |
| doctor | E | haiku-5-5-think | 12,967 of 23,220 | 8,355 of 20,520 | 222 | 1.72% | 0.00% | 88 | `7195ea57121c` |

Few-shot rows are the readers marked `/fewshot-cb1`; thinking on is `haiku-5-5-think`.

Raw "booked right" above. Clean correct bookings (right, not booked too soon), of those who wanted a
booking:

| Dimension | Arm | Gemma | Haiku off | Haiku on |
|---|---|---|---|---|
| payment | F | 11,281 (40.5%) | 11,551 (41.5%) | 11,585 (41.6%) |
| payment | E | 16,599 (59.6%) | 16,893 (60.7%) | 16,937 (60.8%) |
| schedule | F | 3,067 (13.9%) | 3,151 (14.3%) | 3,153 (14.3%) |
| schedule | E | 9,537 (43.3%) | 10,180 (46.2%) | 10,172 (46.2%) |
| doctor | F | 5,498 (23.7%) | 5,592 (24.1%) | 5,584 (24.0%) |
| doctor | E | 11,663 (50.2%) | 12,302 (53.0%) | 12,280 (52.9%) |


## Cost and time

- Gemma: $0 metered, 442 to 541 minutes per cell on one Spark at 4 parallel.
- Haiku: $3.00 to $5.60 per cell, 70 to 103 minutes at 16 parallel. All Haiku runs, 14 cells plus
  slices and canaries: $58.65, computed from each conversation's reported tokens at list price
  (cache reads and writes priced separately), which matched Anthropic's console to the cent.
- Tokens per Gemma cell: about 20 million input and 1.3 million output; Haiku's tokenizer counts
  about 20% more input for the same prompts.

## Venue speed, on a 576-conversation slice (not a cell)

| Reader and venue | Parallel | Minutes | Conversations a minute |
|---|---|---|---|
| Gemma, DGX Spark (vLLM) | 4 | 4.1 | 140 |
| Gemma, Mac Studio M5 Ultra (llama.cpp, bf16) | 4 | 2.5 | 230 |
| Gemma, Mac Studio M5 Ultra (llama.cpp, bf16) | 16 | 2.2 | 262 |
| Gemma, CoreWeave Forge | 4 | 3.9 | 148 |
| Gemma, CoreWeave Forge | 16 | 1.2 | 480 |
| Claude Haiku 5.5 | 4 | 3.2 | 180 |
| Claude Haiku 5.5 | 16 | 0.7 | 823 |

Forge's transcripts matched the Spark's word for word; the Studio's matched in 546 of 576.

## Gotchas

- llama.cpp's chat template turns Gemma's thinking on by default; vLLM's doesn't. Pass
  `--reasoning-budget 0`, or a canary shows 10 to 100 times the output tokens.
- A server's sampling defaults differ: vLLM applies the model's generation config when the driver
  sends none (temperature 1.0, top-k 64, top-p 0.95, seeded per request); llama.cpp doesn't.
- Organization-scoped Anthropic keys need an `anthropic-workspace-id` header on every request.
- LiteLLM refuses to start without its own access key; give it one and pass it to the driver.
- One run per output folder: an early runner resumed into a folder with different settings without
  complaint, and recorded gateway runs under the wrong venue name. Both are fixed in the bench.
- A watcher that reads a remote command's exit code must account for `grep -c` exiting non-zero on
  zero matches; one sat blind overnight for that reason.

## After review: the analyses behind the record's numbers

Four rounds of cross-model review (ChatGPT and Gemini) drove these, all computed read-only from the
built cells.

**Why the code rule lost clean bookings.** Paired by patient and setting, clean correct bookings.

| Cell | E clean, F not | F clean, E not | Long yes (7 words) | Urgency added (8+) |
|---|---|---|---|---|
| payment, Gemma | 5,413 | 95 | 5,318 | 0 |
| schedule, Gemma | 6,716 | 246 | 1,671 | 4,906 |
| doctor, Gemma | 6,398 | 233 | 2,886 | 3,268 |
| schedule, Haiku off | 7,088 | 59 | 1,682 | 5,230 |
| doctor, Haiku off | 6,748 | 38 | 2,894 | 3,574 |

The long or urgent yes is 96 to 99% of E-only clean bookings in every cell. Urgency attaches to turns
about the dimension under test, so payment's confirmation never carries it.

**Bookings made on an open slot or doctor, model's yes counts (E), hedged answers.** Only breaks on
the hedged proposition; ranges span pressure levels where noted.

| | 1 question | 2 | 3, no pressure | 3, urgency | 3, urgency + prop |
|---|---|---|---|---|---|
| schedule, Gemma | 0.2 to 0.5% | 6.9% | 24.5% | 24.6% | 23.4% |
| schedule, Haiku off | 0.2% | 0.1 to 2.6% | 0.6% | 12.5% | 5.5% |
| doctor, Gemma | 0.0 to 0.2% | 12.6 to 13.0% | 26.5% | 23.0% | 20.6% |
| doctor, Haiku off | 0.0% | 0.0 to 5.3% | 0.0% | 11.8% | 3.7% |

Under F, none. Where the patient can never settle it, only 5 to 27% of conversations reach a write.

**The hand-off gap.** Payment, F: Gemma reached a person and Haiku didn't in 1,914 paired
conversations, the reverse in 52. In 1,727, Haiku answered a self-pay question no to a patient saying
they couldn't decide on the call ("I'd have to check with my husband", "I'd need to check and get back
to you"); Gemma answered the same line unclear in 1,686. Every self-pay booking in either arm skipped
the claim and most-you'll-pay questions; that's every break under F.

**Clean correct bookings** count a booking only if it was right and no dependency was open. That also
excludes every self-pay booking, because of the driver's claim gap; excluding only that gap, the rule
effect is 20.5 to 34.2 points instead of 19.1 to 31.9.