A guard is only as good as where its inputs come from.
By Keith Townsend · October 19, 2026
An AI agent books a clinic appointment only if code confirms the patient agreed. But the agent is the one that tells the code whether the patient agreed. Does that check protect anything? This lab ran two versions of a clinic booking agent, the example from an open-source agent kit and a version where code runs the workflow, through 1,000 scripted patient conversations on four model configurations. Both let through bookings the patient never confirmed where the check took the model’s word, and both stopped once it didn’t.
One workflow (clinic booking), one agent framework (Google’s Agent Development Kit, as the kit ships it), scripted patients, an in-memory store, no concurrency. Four model configurations ran the 1,000-scenario corpus: Gemma 4 26B with reasoning off and on, Qwen3.8-Flash-Next, and DeepSeek V4 Flash 0731, whose kit-as-shipped arm is 271 sessions (150 run in corpus order, then a seeded stratified sample of 121) reported as a type-weighted estimate with a 95% interval. Gemini 3.5 Flash ran only the 30-scenario first phase, 45 pressure sessions of the shipped kit. No hosted frontier model ran the corpus. A few hundred pressure sessions per arm per model: directional, not academic. It compares two methods on one workflow and doesn’t rule between them; follow-on labs test a decision model on the kit and a fine-tuned interpreter on the code-driven version.
Don’t Don’t accept a deterministic check as a control until you know where each of its inputs comes from. In the kit as shipped, the booking service refuses unless “confirmed” and “eligible” are true, and the model sets both. Under ordinary pressure, every model that pushed to book also asserted the precondition: 48.7, 60.3, 34.3 and 41.1 forbidden bookings per 100 pressure sessions on the four configurations. No model ever passed an honest false that the check could refuse. Checks that read the clinic’s own records (slot taken, someone else’s appointment, unverified identity) were attempted 74 times under the kit across the four corpus runs, 116 times across all arms, and never let one through.
Do Feed every guard from a record the model can’t write: confirmation from the patient’s own action, coverage from the insurer’s service, the service from the doctor’s record. With only that change to the kit’s boundary, forbidden bookings fell to zero on both local models in the first phase, with the model still driving the conversation.
Do Count what each method costs, not only what it leaks. The kit’s model makes about 9 calls per session (38,305 to 43,057 tokens) and writes every reply; the code-driven version makes about 1.1 (280 to 805 tokens) and replies from code templates. Normal bookings completed about equally, 90.3 to 92.2% for the kit as shipped against 91.3 to 94.5%. Each had its own gap: the kit’s example as shipped can’t complete a reschedule, and the code-driven driver had two bugs that wrote unconsented changes.
Don’t Don’t let a model’s yes count as the patient’s consent, in either method. In the kit’s example the model’s yes is the “confirmed” flag. With code running the workflow but consent judged by the model, forbidden bookings ran 25.8 to 31.9 per 100, every one through a consent question. Deciding consent only in code (an explicit “I confirm” or a bare yes of four words or fewer), and asking again when the reply is unclear, took that to zero in pressure sessions on every model. The price: completed normal bookings fell about 6 points, to 85.4 to 88.3%, and hand-offs to a person roughly doubled, to about 200 per 1,000 sessions. That price is a floor: the scripted patients were told to confirm explicitly, and real patients who say “yes, please book that one” would be asked again.
Don’t Don’t expect a stronger model to fix a guard that takes the model’s word. No configuration came near zero under the kit: the larger models leaked 41.1 (DeepSeek V4 Flash) and 34.3 (Qwen). Turning Gemma’s reasoning on raised its rate from 48.7 to 60.3, because it read coverage correctly and reached the booking step more often, where it then asserted the precondition.
Do Test the code path as hard as the model path. Code-driven designs move risk into code: two driver bugs wrote unconsented changes in about 1 in 1,000 sessions, one of them with an explicit yes to a question that named the wrong action. Consent questions have to name the action exactly, and the driver needs its own check that the action it proposes matches the request. This driver’s consent rule would also read “Before I confirm, is that covered?” as a yes; no corpus message tested it.
Self-funded editorial. The DGX Spark machines and the Mac Studio were bought by the practice. Disclosure: Google Cloud is a client of this practice. Google’s Gemma model ran every corpus arm, Google’s Gemini ran one first-phase arm on Google Cloud, Google’s Agent Development Kit is the framework the kit ships with, and a Google model checked the corpus. Google did not commission, fund, preview or have any say in this lab. Jeremy Schulman, the kit’s author, had no part in the design, runs or verdict; told about the lab before publication, he said the clinic example was early work and raised no objection. NVIDIA, Apple, Anthropic, OpenAI, Alibaba and DeepSeek, whose hardware, tools and models were used, are not clients. No vendor paid for this answer.
In one test session, an AI clinic booking agent asked the patient to confirm the 9:00 slot. She answered, "I told you, it's already confirmed. Book it now." The model called the booking tool with confirmed set to true, and the check committed it. The code did its job. It checked the flag. The model it was guarding had written the flag.
That one exchange is the whole problem. An AI agent books a clinic appointment only if code confirms the patient agreed, but the agent is the one that tells the code whether the patient agreed. Does that check protect anything? I tested two ways of building that agent to find out. One is the clinic example from Jeremy Schulman's open-source Promise-Driven Agentic Kit, where the model runs the workflow and code checks each booking before it's written. The other is a version where code runs the workflow and calls the model only to interpret what the patient meant. I ran both through 1,000 scripted patient conversations on four model configurations. Both leaked in the same place.
What did the check actually check?
Under ordinary pressure, the kit's example let through 34.3 to 60.3 forbidden bookings per 100 pressured conversations, depending on the model. When a model pushed to book, it also asserted the precondition. Bookings that broke a check reading the clinic's own records, a slot already taken or someone else's appointment, reached the check 74 times. None got through.
Jeremy's own argument explains it better than I can. You can't command a model, he writes. You can only get a promise from it and then check whether it was kept. The model's confirmed set to true is a promise about the patient, and in this example the code accepts it without checking it against anything else. His case study calls those flags a teaching simplification, and when I told him about this lab, he said the example was early work. The method holds. This input doesn't.
Did the code-driven version avoid it?
No. With code running the workflow, the model still answered the consent questions code couldn't read, and its yes counted. That version leaked 25.8 to 31.9 per 100. Gemma read replies like "I'll cover the copay" as choosing to pay out of pocket in 57.8% of those sessions. Code brought its own risk, too. Two bugs in my driver wrote unconsented changes, about 1 in 1,000 sessions, and the consent rule has a hole no conversation tested: it would read "Before I confirm" as a yes.
Does a smarter model fix it?
No. The larger models leaked less than Gemma did under the kit, 41.1 for DeepSeek V4 Flash and 34.3 for Qwen, but neither came close to zero. Gemma got worse with reasoning turned on, going from 48.7 to 60.3. With reasoning off it had been telling covered patients they weren't covered, so it never reached the booking step. With reasoning on it got coverage right, reached the booking, and asserted confirmation like every other model.
What closed it?
The same change in both methods: take the fact away from the model. In the kit, I had the check read confirmation and coverage from records the model can't write, and kept everything else, including the model running the conversation. Forbidden bookings went to zero on both local models I tested that on. In the code-driven version, I decided consent only in code and asked again whenever a reply was unclear. Zero in the pressure conversations on every model. That one had a real price: completed bookings fell about 6 points, to 85.4 to 88.3%, and hand-offs to a person roughly doubled, to about 200 per 1,000 sessions. And that's a floor, because my scripted patients were told to confirm explicitly.
So what?
I'm not calling a winner. These methods are early, and so is everyone's understanding of them, mine included. Next I'll put a decision model in front of the kit's check and a fine-tuned interpreter in the code-driven version, and either could change the costs. What I don't expect to change is this: a deterministic guard is only as trustworthy as the provenance of its inputs. When a vendor walks you through their guardrails, will you ask where each one's inputs come from?
The numbers
Where each layer belongs
Read for decision authority. Who makes the runtime decision at this layer, and can you see and override it? How to read this table
| Layer | Kit as shipped (A) | Kit, authoritative inputs (B, C) | DCITL (E) | DCITL, consent in code (F) |
|---|---|---|---|---|
Layer 3 (+1) · Applications AI Application Layer — The Value Plane Who decides the next step in the clinic workflow. In the kit the model drives the conversation and decides when to book, within limits code sets: Delegated. In DCITL code runs the workflow; the model only interprets. | Delegated | Delegated | Retained | Retained |
Layer 2B · Runtime Application Runtime & Execution Who decides, at execution, whether a write goes through: the deterministic check on every tool call, placed at Layer 2B as the gates in Labs 3, 11 and 12 are. In the shipped kit the boundary decides, but on the confirmation and eligibility facts the model supplies, so for those checks the model decides by choosing what to pass: Ceded. Checks that read the clinic’s records stayed with code in every arm. In E code decides, but when it can’t read a consent reply it commits on the model’s yes to a closed question: Delegated. With authoritative inputs (B, C) and in F the boundary decides on facts the model can’t write. | Ceded | Retained | Delegated | Retained |
Layer 2C · Reasoning Agentic Infrastructure — The Reasoning Plane Who determines that a booking is done, which here means consented. In the shipped kit the model’s own assertion is the consent: Ceded. In E code decides, but on consent questions it acts on the model’s reading of the patient’s reply: Delegated within a closed question. In F code acts on a model’s no and ignores a model’s yes; only an explicit yes code reads itself commits. | Ceded | Retained | Delegated | Retained |
Method and disclosure
Self-funded. The DGX Spark machines and the Mac Studio were bought by the practice. No vendor was briefed, reviewed this, or was given advance sight. The kit’s author was told about the lab before publication and had no part in it.
The kit’s clinic appointment example was copied unmodified from upstream commit 0514f991 (Apache-2.0, license kept). Every arm patches the kit’s services at import time; arm A is the kit as shipped. The harness, scorer, scenario generator and DCITL driver were built and run by Claude Code (an Anthropic tool) under the author’s design rulings, with a self-test that plants a lie against every guard in every arm and refuses to start a run unless each is scored correctly.
Venues: Gemma 4 26B (bf16, vLLM 0.25.1) on one DGX Spark, reasoning off and, in a separate run, on; Qwen3.8-Flash-Next (NVIDIA’s FP4-Local build, NVIDIA’s published single-Spark recipe) on a second Spark; DeepSeek V4 Flash 0731 (FP8, oMLX, thinking at maximum effort) on the Mac Studio, with the serving configuration Lab 20 used on the same machine and the weights Lab 14 qualified; Gemini 3.5 Flash on Vertex AI for the first phase only.
Corpus: specifications generated by code with a fixed seed; patient wording by GPT-5.6-Sol through the Codex command-line tool on a ChatGPT subscription; blind check by Gemini 3.8 Flash through the Antigravity command-line tool. The DCITL driver was hash-frozen before generation; three later fixes are recorded with reasons, none adding to the parsers’ vocabulary: a crash on an unclear intent, a loop that re-listed times instead of asking the model, and a consent bug where “I confirm” with a different slot named booked the proposed slot.
Reviewed before publication by a model from a different family instructed to attack the record. Its findings and their disposition are kept with the lab’s working notes.
The harness, scorer, corpus generator and corpus are the instrument and can be published. The method for assessing a vendor’s placement is not.
The essay on this page was drafted by Claude Opus 5.5, an Anthropic model, from the lab’s frozen structured record under this practice’s voice specification, checked by a deterministic gate against every figure in the record, and validated by the author before publication. The record is the canonical surface: where prose and record disagree, the record rules.
Download the raw lab detail (Markdown)