The Clinic · recorded results

Change one part of the agent and compare

One scheduling agent, the same 1,000 simulated patients, one part of its stack changed at a time. Each patient is heard at 48 settings, so every cell is 48,000 recorded conversations. Every number here comes from those recordings. Nothing is generated while you watch, and no model is called.

Now: Code decides · Gemma 4 26B, reasoning off · The question alone · Code-written lines · Deterministic parser · Payment · Entire set: 1,000 patients, 18 types

This stack made 11,281 clean correct bookings in the 27,840 conversations where the patient wanted a booking, move or cancellation, reached a person in 5,306 of 15,600 conversations where the patient needed one, and had a wrong write in 2 of 48,000 conversations.

Clean correct bookings, of 27,840 conversations where the patient wanted a booking, move or cancellation
40.5%
right, nothing open · 11,281
Right, lucky guesses included
41.0%
11,404 of 27,840
Reached a person, of 15,600 conversations where the patient needed one
34.0%
5,306 of 15,600
A write made too soon (a break)
0.3%
of all 48,000 conversations
Conversations with a wrong write
2
of 48,000
Conversations turned away without a person
7,975
of 48,000
Conversations where the patient gave up
10,703
of 48,000

How to read these counts: 1,000 patients, each heard at all 48 settings, make 48,000 conversations, so every count here is of conversations. A patient's truth stays fixed across its 48 conversations, so the patients, not the conversations, are the independent observations. The intervals are bootstrap stability intervals that resample whole patients of this synthetic set. They don't describe a real patient population, and they don't include run-to-run variation in the reader: each cell is one run.

What makes the agent write too soon

How the patient states payment before it's settled: plainly, hedged, contradicting itself, or not at all.

Now: a write too soon
0%5%10%15%20%1 in 100.2%clear0.3%hedged0.3%contradictory0.2%withheld

What happened in the conversations

Every conversation at every setting, split by what the patient needed. Where the patient wanted a booking, move or cancellation, only the right write is right. Where they needed a person, only a hand-off is right; turned away means the agent said no and stopped. The third group wanted neither: a coverage question, or a list of their appointments.

What happened in each group of conversations, as counts
GroupBooked rightReached a personTurned away without a personPatient gave upWanted it, not doneWrong writeConversations
Now, patient wanted a booking, move or cancellation11,404004,97811,458027,840
Now, patient needed a person05,3067,9752,3170215,600
Now, patient wanted neither (a question, a list)01,15203,408004,560

What it costs to run

Measured tokens and pace for this cell. For Gemma, tokens are priced at the managed per-token rates found for the same model on 2026-10-22 (Google's own API lists it as free; the paid rates come from OpenRouter listings and third-party trackers, so treat the range as an estimate), and the run itself used owned hardware at $0. For Haiku, the spend is measured: reported tokens at Anthropic's list price, which matched Anthropic's Console to the cent.

Model calls per conversation1.80
Tokens per conversation, in / out399 / 28
Managed cost per 1,000 conversations (third-party rates)$0.033 to $0.063
Managed cost per right booking (third-party rates)$0.0001 to $0.0003
Conversations per minute (4 at once)106
Model calls per minute190
Run time, owned DGX Spark, 4 at once453.8 min, $0
At the free tier's limits, this cell58 days
At the free tier's limits, the entire set (48,000 conversations)58 days

Free-tier limits are Gemini's account, unverified against AI Studio: 15 requests a minute and 1,500 a day per project, so the daily cap sets the time. That tier may also log prompts and outputs to improve Google's products, which rules it out for real patient messages. These patients are synthetic.

Sample size. Each cell runs 1,000 simulated patients, each at all 48 settings of one dimension (4 levels of clarity, 4 of probes to resolve, 3 of pressure): 48,000 conversations. A patient's truth stays fixed across its 48 conversations, so the 48,000 aren't independent observations; the 1,000 patients are. Every interval on this page resamples whole patients, never conversations.

What this shows, and what it doesn't. These are measured behaviors inside the simulation: how each stack responds to these simulated patients, scored the same way every time. They show how the parts compare against each other here. They don't establish that the same ordering holds with real patients in a production clinic; that would take real calls.

How it's measured. Each simulated patient's truth (what they'll pay, when they can come, who they'll see) is set in code before the call. Code decides every patient act and logs what it disclosed and how clearly. A booking counts as too soon when it's made while something it depends on (the time, the doctor, whether the patient will pay) is still unclear in that log. The bench's own term for it is a break. No model grades anything.

What's simulated. The patient speaks in code-written lines and hears the agent through a deterministic parser. The agent is Lab 23's code-driven scheduling driver. Its reader is Gemma 4 26B on owned NVIDIA DGX Sparks, or Claude Haiku 5.5 through Anthropic's API. Each patient behaves the same at every setting, so the patients, not the settings, set the uncertainty.

Versions. Every run is frozen by hash before it starts. Two runs compare only when the patient and scoring code is identical. An earlier F run, made before a patient fix, is kept and marked superseded.

Independence. Self-funded editorial work by The CTO Advisor. Google Cloud is a client, and Gemma is a Google model. Kamiwaza is a client, and appears among the platform examples. These are named here and aren't clients: Anthropic (Claude Haiku), NVIDIA (the DGX hardware), Alibaba Cloud (the Qwen models), TypeSafe (Jev), ServiceNow, Salesforce, CoreWeave, AMD and Apple. No vendor reviewed or shaped these pages before publication.

How the runs were paid for. The Gemma runs used two NVIDIA DGX Sparks The CTO Advisor owns. The Claude Haiku 5.5 runs used the $200 monthly API credit included in Keith Townsend's paid Anthropic plan; Anthropic provided no special funding or project-specific credits. The managed per-token prices shown for Gemma come from third-party listings (OpenRouter and public price trackers), not from Google.

What this is. A simulation: synthetic patients written in code, one reference scheduling agent, a deterministic scorer. It shows how the parts compare here, not how a production clinic would behave. The platform examples are unbenched.