A four-word consent rule cost more clean bookings than swapping the model
By Keith Townsend · October 24, 2026
Thread: Where does a deterministic check stop protecting you? · 5 of 5 · ← Lab 025
A patient tells a clinic’s AI scheduling agent, “Yes, that’s right, please go ahead.” Is that a yes? To the code rule I tested, it’s seven words, and the rule only counts four or fewer. So who should decide whether a patient agreed: code, or the model? And does swapping in a frontier vendor’s model change the answer? I ran one agent against 1,000 simulated patients whose answers I controlled, swapped one part of its stack at a time, and scored every booking against what each patient actually meant.
One workflow (clinic scheduling), one code-driven driver (Lab 23’s, frozen), one code consent rule (Lab 23’s four-word rule), one simulated clinic of 8 patient identities, 6 doctors and 42 slots, and patients that are code. They react to which question the agent asks, but speak in templates and repeat their answer when asked again; they don’t shorten or rephrase. The readers are Gemma 4 26B-A4B with reasoning off on owned DGX Sparks and Claude Haiku 5.5 through Anthropic’s API, with thinking off and on. Each cell is 1,000 patient truths at 48 settings; the patients, not the conversations, are the independent observations. These are behaviors measured inside the simulation. They compare the parts here; they don’t show that the ordering holds with real callers, with a code rule that accepts a clear yes of any length, or with a patient who shortens a yes when asked again. Not yet run: that rule, that patient, Gemma with reasoning on, a model running the whole workflow (the next lab), and a model-written patient voice.
Do Test your consent rule against how your callers actually say yes before you shop for a reader. On this bench, letting the model’s yes count instead of a four-word code rule raised clean correct bookings by 19 to 32 points with every reader (paired 95% intervals 16.0 to 34.5): 41% to 60% on payment, 14% to 43% on schedule, 24% to 50% on doctor, for Gemma. Counting every right booking, lucky guesses included, the gain was 21 to 37 points. Swapping Gemma for Haiku moved clean correct bookings by 3 points or less in every cell, in the single runs reported. The rule dominated the booking differences; the reader’s were small. Letting the model decide also brought hundreds of conversations per cell with a wrong write that the code rule didn’t have.
Don’t Don’t let the model’s yes count without testing the reader on slow hedges and on pressed ones; these two failed on different settings. Counting only bookings made while the hedged time or doctor itself was still open, with the model deciding: when one question settled the hedge, both readers did it in 0.5% or fewer of those conversations. Gemma rose with the number of questions at every pressure level, 7 to 13% at two and 21 to 27% at three. Haiku stayed under 6% in 11 of the 12 settings of questions and pressure, and reached 10 to 13% in one: three questions with urgency alone (with urgency and a prop, 3 to 6%). Across each whole cell, Gemma had 526 and 750 conversations with a wrong write, Haiku 213 and 236. Under the code rule, no booking was made on an open time or doctor.
Don’t Don’t send a reader’s no straight to a hang-up. This driver ends the call on no and hands off after unclear, under either consent rule. Asked whether to pay out of pocket, some patients say they can’t decide on this call: “I can’t tell you that right now. I’d have to check with my husband.” or “I honestly don’t know. I’d need to check and get back to you.” On payment under the code rule, Gemma reached a person and Haiku didn’t in 1,914 paired conversations, the reverse in 52. In 1,727 of the 1,914, Haiku had answered a self-pay question no and the driver never handed off; Gemma answered the same line unclear in 1,686 of them. Reached a person: Gemma 34%, Haiku 22%, a gap of 10 to 14 points in every arm and prompt. Thinking on didn’t close it, and borrowed worked examples widened it. Most patients who needed a person missed one under either reader; on schedule and doctor the readers were within 1.1 points.
Don’t Don’t borrow worked examples without measuring them on your own workload. Lab 024’s six examples per question, sent as prior chat turns, moved three things on payment. Clean correct bookings rose a little (Gemma 11,281 to 11,585 under the code rule, 16,599 to 16,911 with the model deciding; Haiku by 32 and 46). On self-pay hedges, where four of five examples say no, Haiku moved further toward no and reached fewer patients who needed a person (22% to 20%), while Gemma stayed put. And with the model deciding, conversations with a wrong write rose on both readers, Gemma 453 to 851, Haiku 400 to 596, of 48,000. A small gain in bookings for a large rise in wrong writes. That’s these examples on payment, not a rule about few-shot.
Do Price the code rule against how your callers say yes. Lab 23’s rule counts “I confirm” or a yes of four words or fewer. A patient who talks at length says “Yes, that’s right, please go ahead.” (seven words); a patient pressed on the time or the doctor adds urgency, as in “Yes, book it. I really need this soon.” (eight). Either way the driver asks again, the patient repeats it, and the driver hands off. Of the conversations the model-decides arm booked cleanly and the code arm didn’t, 96 to 99% went that way in every cell; the code arm booked cleanly where the other didn’t in 15 to 246 per cell. On payment it’s almost all the long form, because urgency attaches to the dimension under test and payment’s confirmation doesn’t carry it; that’s part of why payment’s gap is the smallest. This bench’s patient never shortens on the second ask, so this is the rule’s cost against these patients, not against callers who would just say “Yes.”
Self-funded editorial. The DGX Sparks and the Mac Studio were bought by the practice. Disclosure: Google Cloud is a client of this practice, and Google’s Gemma is the reference reader. The driver’s prompts were written for Gemma in Lab 23, and the pre-freeze shakedown ran on Gemma; neither was tuned for Haiku. Anthropic is not a client; Claude Haiku 5.5 ran on API credits that come with a Claude plan I pay for. The bench was built with Claude Code, an Anthropic tool, which also wrote the patient’s template lines, so an Anthropic model wrote the words an Anthropic model reads. The scoring is deterministic code, never a model. CoreWeave, NVIDIA, Apple and OpenAI are not clients. No vendor saw, directed or paid for this work.
A patient calling a clinic says, "Yes, that's right, please go ahead." Is that a yes? To a person, obviously. To the rule I tested, it's seven words, and the rule only counts a yes of four words or fewer. So the agent asked again, got the same answer, and handed the patient to staff.
That rule came from Lab 23, where it made no forbidden booking in the pressure tests. Lab 23 itself started from the clinic example in Jeremy Schulman's Promise-Driven Agentic Kit, and that clinic is still the one these patients call. But Lab 23's patients were scripts told to confirm explicitly, so it couldn't say what the rule costs when patients don't. This lab asks that on a different bench, with patients who react and words from a different writer. Its numbers stand on their own. It also asks a second question: if you let a model decide whether the patient agreed, does it matter which model?
How do you know what the patient meant?
You make the patient. I built a bench where each simulated patient's truth is fixed in code before the call: what they'll pay, when they can come, which doctor they want. Code decides how clearly they say it, how many questions it takes, and whether they push. Every booking is scored against that truth by code, and I only count a booking as correct if it was right and nothing it depended on was still open. A lucky guess doesn't count, and neither does any self-pay booking, because this agent never asks whether to file a claim. Counting those, the rule's gap below grows by two or three points; the story doesn't change. I ran 1,000 patients, each at 48 settings, across 22 combinations of rule, model and prompt. It's a simulation, so it tells you how the parts compare, not what your call center's numbers will be.
The bench is open. clinicbench, with its patients, its scorer and the harness that runs them, is on GitHub under the MIT license, so you can put your own agent in front of the same 1,000 patients and score it the same way. What isn't in it is how I judge a vendor's placement. That part stays here.
Does the rule or the model matter more?
For bookings, the rule, by a lot. Letting the model's yes count instead of the code rule raised correct bookings by 19 to 32 points with every reader. Swapping Gemma 4 26B-A4B, running on hardware I own, for Claude Haiku 5.5 through Anthropic's API moved them by 3 points or less. Nearly all of the rule's cost, 96 to 99% of the bookings it lost in every run, was a clear yes longer than four words: the polite long form above, or "Yes, book it. I really need this soon." from a patient pressed about the time or the doctor, which is eight.
Those extra bookings weren't free. With the model deciding, every run had hundreds of conversations with a wrong booking, move or cancellation that the code rule didn't have.
There's a caveat, and it's a big one. My simulated patient repeats its answer when asked again. A real caller asked for a plain yes or no might just say "Yes." So this is the cost of this rule against these patients, not a law about code. And each Haiku run happened once, so the 3 points doesn't include how much a second run would differ.
One more caveat on fairness. The agent's prompts were written for Gemma in an earlier lab and never tuned for Haiku, and Claude Code, an Anthropic tool, wrote the patients' lines. Those lean in opposite directions. I haven't separated them.
So is the model irrelevant?
No. It mattered in two places. Once the model decides, it's a safety choice, and these two broke in different places. When one question settled a patient's hedge about the time or the doctor, neither booked on the open question more than 0.5% of the time. When it took two or three questions, Gemma did in 7 to 27%, whatever the pressure. Haiku stayed under 6% in all but one setting: three questions with urgency, where it reached 10 to 13%. When the patient could never settle it, both booked less, mostly because fewer of those calls got to a booking at all.
The other place is routing, and it shows up even when code decides. Asked whether to pay out of pocket, a patient says, "I can't tell you that right now. I'd have to check with my husband." The bench calls that unclear. Gemma almost always did too, and the agent kept asking. Haiku answered more of those no, and this agent ends the call on a no. On payment, Haiku got 22% of the patients who needed a person to one, against Gemma's 34%. In 1,727 of the 1,914 conversations where Gemma got there and Haiku didn't, Haiku had said no to a self-pay question. Most patients who needed a person didn't reach one with either model, and on schedule and doctor the two were within a point. Turning thinking on didn't close the gap. Six worked examples per question borrowed from an earlier lab, four of five of the self-pay ones labeled no, pushed Haiku further toward no. They added a few hundred clean bookings. With the model deciding consent, they also raised the conversations with a wrong booking, move or cancellation on both models, Gemma from 453 to 851 and Haiku from 400 to 596, out of 48,000.
What does it cost?
Not what I expected. Haiku cost $3.00 to $5.60 a run and finished in about 80 minutes, sixteen conversations at a time. Gemma cost nothing per call on hardware I own and took most of a working day at four at a time. Those are two deployments, not a speed test of the models. All of Haiku's runs came to $58.65, and the Sparks' purchase isn't in Gemma's zero. At this scale the price is small either way. What separates them is time, and whether you get to keep and tune the model.
So what?
Test your consent rule against how your callers actually say yes before you shop for a model. If code decides, the model barely moves your bookings, but its no still steers who reaches a person, so check what your code does with each answer. If the model decides, test it on hedges that take several questions and on hedges under pressure, because these two capable models failed in different places. Or skip my numbers and run the bench on your own stack. Which would you rather explain to a patient: the booking they didn't agree to, or the person they never reached?
The numbers
Where each layer belongs
Read for decision authority. Who makes the runtime decision at this layer, and can you see and override it? How to read this table
| Layer | Code decides consent (F) | The model’s yes counts (E) |
|---|---|---|
Layer 3 (+1) · Applications AI Application Layer — The Value Plane Who decides the next step in the scheduling workflow. In both arms code runs the workflow and the model only interprets what code can’t read. | Retained | Retained |
Layer 2B · Runtime Application Runtime & Execution Who decides, at execution, whether a write goes through. The clinic’s tools check slots and records themselves in both arms; neither takes a fact from the model. | Retained | Retained |
Layer 2C · Reasoning Agentic Infrastructure — The Reasoning Plane Who determines that the patient agreed, which here is what “done” means for a booking. Under F, code ignores a model’s yes; only the patient’s own explicit words commit. A model’s no still routes the call: the driver ends it, which is how the reader moves hand-offs under F. Under E, code commits on the model’s yes to a closed question: Delegated within that question. The reader model doesn’t change this placement; it changes how well a Delegated decision goes, and where a no sends the patient. | Retained | Delegated |
Sources
Where this fits
Method and disclosure
Self-funded. The DGX Sparks and the Mac Studio were bought by the practice. No vendor was briefed, reviewed this, or was given advance sight.
Instrument: clinicbench v1, built and run by Claude Code (an Anthropic tool) under the author’s design rulings, frozen by hash before the cells (freeze sha256 prefix 15e801051e5abc3f). Lab 23’s driver was vendored unchanged from its frozen original. The clinic itself traces to the clinic appointment example in Jeremy Schulman’s Promise-Driven Agentic Kit, which Lab 23 started from and expanded; clinicbench ports Lab 23’s expanded world. The measurement contract was frozen before the build and changes before the freeze are logged with reasons; the analysis deviation after the runs is logged in this record (see what this doesn’t prove). A shakedown on Gemma found three persona defects, all fixed and rerun before the freeze. Paired contrasts, clean correct bookings, the breakdown of F’s losses, the hand-off decomposition, the patient lines behind it, and proposition-specific hedged rates by probes and pressure were computed from the built cells after review.
Venues: Gemma 4 26B-A4B (bf16, vLLM 0.25.1, revision 20da991a pinned) on two DGX Sparks, one cell per node; a parity check gave word-for-word identical transcripts on both. Claude Haiku 5.5 through Anthropic’s API via LiteLLM 1.106.2, thinking off or adaptive at medium effort, checked against a local stand-in before any spend. Worked examples: Lab 024’s final set, one self-pay example relabeled, sent as prior chat turns with the driver’s prompt unchanged.
Reviewed before publication by two models from other families (ChatGPT and Gemini, four rounds); their points on the code rule’s confound, lucky-guess bookings, denominators, intervals, conditioning and routing are reflected in the record, several through analyses run after review.
The bench is clinicbench, published open under the MIT license at github.com/kltownsend/clinicbench: the patients, the scorer, the runner and its gates, the code-driven driver with both consent rules, and a public contract with its own freeze. Lab 029’s private freeze stays with this lab. The method for assessing a vendor’s placement is not published.
The essay on this page was drafted by Claude Opus 5.5, an Anthropic model, from the lab’s frozen structured record under this practice’s voice specification, checked by a deterministic gate against every figure in the record, and validated by the author before publication. The record is the canonical surface: where prose and record disagree, the record rules.
Download the raw lab detail (Markdown)