Examples aren’t portable. Validate them per model.

What this covers: four readers in one simulated scheduling workflow that code runs, one set of six examples per question, one run per cell. The examples went into the prompt for Gemma and Haiku and into the state field for Jev and Bivia, so this measures the same cases delivered through the integration each reader offers, not examples alone. Premature and wrong bookings are counted over all 48,000 conversations, clean right bookings over the patients who wanted a booking, and hand-offs over the patients who needed a person. The open question: does a probability bar in code make the examples matter less?

Self-funded editorial. Jev ran on my own TypeSafe account; TypeSafe billed $17.68. Bivia ran on Articul8’s free tier. Articul8 is a past sponsor of The CTO Advisor’s work; it didn’t pay for these runs, but at my request it raised the account’s request limit partway through them, on November 10, 2026. Google Cloud is a client, and Gemma is a Google model. None of TypeSafe, Articul8, Google or Anthropic reviewed this page.

Articul8 calls Bivia a drop-in replacement for Jev, or for any System One API. Say you take that at its word. You swap the reader, and you carry over the examples your team wrote for the old one. Do they still do the same job?

That's the question this lab asks, and my bench's answer is no, not reliably. Not because one model is worse. Because the same examples did different things to different models.

What's a System One reader?

A System One model doesn't write text. You ask it a typed question, a choice or a yes or no, and it hands back an answer and a probability. TypeSafe's Jev and Articul8's Bivia both work this way, hosted behind an application programming interface (API). That shape fits a job the Clinic was built to test: the reader in a workflow that code runs. On the Clinic's bench, code drives a scheduling conversation with 1,000 synthetic patients, each heard at 48 settings, 48,000 conversations a cell. When code can't parse what a patient said, it asks the reader a closed question and takes the answer.

I put Jev and Bivia in that seat beside the two general models already there, Gemma 4 26B and Claude Haiku 5.5. Then I turned one knob. In their public APIs I found no way to fine-tune either model and no field for examples, so examples are the knob a buyer has. I gave all four readers the same six labeled cases per question. Gemma and Haiku got them in the prompt. Jev and Bivia got them as text in the state field, ahead of the case, because their APIs have no field for examples. That's my adaptation, not either vendor's tuning.

What did the same examples do?

I ran it in arm E, where the reader's yes counts as the patient's consent. That's the arm where a reader's reading reaches the booking.

The cleanest result is between the two readers that got the examples the same way. On schedule, Gemma's clean right bookings rose 1.4 points of the patients who wanted one. Haiku's fell 0.4. A clean right booking is a correct one with nothing left open. The bench leads with it because a correct booking made on a guess isn't a decision anyone made. The same split shows on the patients who needed a person: the share who reached one fell 1.3 points for Gemma and rose 1.2 for Haiku. Same six cases, same delivery, opposite directions, and every interval clears zero.

Bivia moved the most. Its premature bookings fell 1.7 to 4.8 points, its wrong bookings fell on every dimension, and 7 to 18 points more of the patients who needed a person reached one. That reads like a win, and on those measures it is. But look at the other side of the clinic. Another 5.6 to 9.8 points of the patients who wanted a booking ended without one, and once the defective line I'll get to is out, its clean right bookings on payment fell too. The examples made Bivia more careful, and careful has a cost you only see if you count it.

That's why the verdict says validate on every outcome, not one. If I'd only watched premature bookings, Bivia would look like the examples fixed it and Gemma would look like they broke it.

Isn't that just the delivery?

It's a fair hit, and it's the objection I'd raise. Gemma and Haiku got the examples in a prompt. Jev and Bivia got text in a state field. Those are two interventions, and the biggest split in my results follows that line. Both System One readers cut premature bookings 1.0 to 4.8 points on doctor and schedule, while the prompt-delivered readers moved under a point either way. Jev and Bivia also started from a worse place. Without examples they booked prematurely in 6.7% and 7.4% of doctor conversations, Haiku in 1.8%. Haiku had less room to fall.

So I don't claim the state field acts like a prompt. I claim something narrower, and I think more useful: this is what a buyer actually gets, the same cases through the integration each reader offers. And inside one delivery, Gemma and Haiku still went opposite ways. The ruling doesn't say examples never transfer. It says you can't assume they do.

What did the bench get wrong?

One of my patient lines is broken, and I left it in on purpose. When a patient turns down an offer that's wrong on the doctor or the appointment, the line says "No, I can't do that one" without saying which part was wrong, and it can name the offered time as one that works. Readers often hear that as yes. The examples changed how often: Gemma's yes rate on that line went from 11% to 34%, Bivia's from 52% to 22%.

That defect carries all of Gemma's increase in premature bookings and most of Bivia's drop on payment. With every conversation that met it excluded, no reader's premature or wrong bookings rose with the examples. The clean-booking and hand-off changes barely moved. So the defect inflated the dramatic number and left the useful ones standing. That's its own lesson. Before you credit or blame your examples for a move in premature bookings, check your inputs.

Where does the yes belong?

There's a second way to read all this. In arm F, code decides consent and the reader may decline but never agree. Gemma and Haiku ran there with and without the examples, and the examples moved their premature and wrong bookings by 0.04 points or less. The placement kept the examples from reaching the booking.

That isn't free. Every reader made about a third fewer clean right bookings on payment and half or more fewer on doctor and schedule than in arm E. And Jev and Bivia only ran arm F without examples, so I haven't tested whether it contains them the same way.

What I didn't prove matters as much. Each cell is one run, and my intervals resample patients, not reruns. I read the vendors' public documentation and didn't ask them about tuning. I left the probability bar, abstain, deep mode and the option lists at their defaults. Every System One answer here came with a probability the driver ignored. Whether a bar in code, with "ask again" below it, makes the examples matter less is the next thing I want to know.

So treat an examples set like code you wrote for one model. It isn't a spec you hand to the next one. Test it where it runs, on every outcome you care about, and if you can't, keep the yes in code.

The numbers

Where each layer belongs

Sources

Where this fits

Method and disclosure

Self-funded editorial. The bench is the Clinic’s, frozen by hash; every cell here carries the same instrument hash as the Clinic’s published cells. The Gemma cells ran on two DGX Sparks I own and the Haiku cells through Anthropic’s API. The Jev and Bivia readers are clients I wrote for this lab, standard library only, and ran from a Mac Studio against each vendor’s hosted API. Neither API has a field for examples, so the six-example set went into the state field ahead of each case: my adaptation, not either vendor’s tuning. Each cell is one run; its intervals resample the 1,000 patients, not reruns. The defect-excluded values drop every conversation where any reader met the defective refusal line and use the same paired intervals. Gemini and ChatGPT peer-reviewed the results and two drafts of this analysis, and the page was revised against their findings.

The essay on this page was drafted by Claude Opus 5.5, an Anthropic model, from the lab’s frozen structured record under this practice’s voice specification, checked by a deterministic gate against every figure in the record, and validated by the author before publication. The record is the canonical surface: where prose and record disagree, the record rules.