# Examples aren’t portable. Validate them per model.

> Lab 030 · Editorial lab · Status: published · Author: Keith Townsend · November 11, 2026  
> Published by The CTO Advisor LLC (The Advisor Bench) · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/system-one-readers · Structured: https://labs.layer2c.com/labs/system-one-readers.json

**Finding:** In one code-run scheduling workflow where the reader’s yes counts as consent, the same six labeled examples moved four readers in different directions: on schedule, Gemma 4 26B’s clean correct bookings rose 1.4 points of the patients who wanted one while Claude Haiku 5.5’s fell 0.4, and the share of patients who needed a person and reached one fell 1.3 points for Gemma while it rose 1.2 for Haiku and 7.5 for Articul8’s Bivia.

**Question:** Articul8 calls Bivia a drop-in replacement for Jev or any System One API. That’s a claim about the API; swapping readers also means carrying your examples across. Jev and Bivia answer a typed question with an answer and a probability, and in their public APIs I found no fine-tuning or examples interface, so the examples went in as text in the state field. Given the same six cases, four readers moved in different directions on clean bookings and on getting patients who needed a person to one. On Bivia, fewer premature bookings also meant fewer bookings made.

**Load:** The Clinic’s frozen bench: Lab 23’s code-driven scheduling driver and 1,000 synthetic patients of 18 types, each heard at 48 settings of one unclear part of the request (payment, schedule or doctor), 48,000 conversations a cell. The reader, the model the driver asks when code can’t parse a reply, is Jev, Bivia, Gemma 4 26B or Claude Haiku 5.5, each with and without the six-example set (Lab 024’s six labeled cases per question, called cb1 in the run records). Arm E takes the reader’s yes as consent; arm F decides consent in code.

**Raw lab detail:** https://labs.layer2c.com/downloads/system-one-readers-lab-detail.md
**Vendors:** TypeSafe (model), Articul8 (model), Google (model), Anthropic (model), NVIDIA (hw), Apple (hw), OpenAI (bench)
**Layers:** Layer 2C (Reasoning), Layer 3 (+1) (Applications)
**Themes:** prompt-vs-weights, authority-placement, vendor-claim-scrutiny

## Verdict

**Scope:** What this covers: four readers in one simulated scheduling workflow that code runs, one set of six examples per question, one run per cell. The examples went into the prompt for Gemma and Haiku and into the state field for Jev and Bivia, so this measures the same cases delivered through the integration each reader offers, not examples alone. Premature and wrong bookings are counted over all 48,000 conversations, clean right bookings over the patients who wanted a booking, and hand-offs over the patients who needed a person. The open question: does a probability bar in code make the examples matter less?

- **DON'T:** Don’t carry an examples set from one reader model to the next and assume it does the same job. On schedule, the same six cases, given the same way in the prompt, raised Gemma’s clean right bookings 1.4 points and cut Haiku’s 0.4, and got fewer of the patients who needed a person to one on Gemma and more on Haiku.
- **DO:** Validate examples per model on every outcome, not one. On Bivia they cut premature bookings 1.7 to 4.8 points and got 7 to 18 points more of the patients who needed a person to one, and they also left 5.6 to 9.8 points more of the patients who wanted a booking without one.
- **DON'T:** Don’t credit or blame the examples for a move in premature bookings until you’ve checked your inputs. One defective patient line carries all of Gemma’s increase and most of Bivia’s drop on payment.
- **DO:** If you can’t validate the examples, keep the yes in code. With consent decided in code, the same examples moved Gemma’s and Haiku’s premature and wrong bookings by 0.04 points or less. That containment costs bookings: under it, clean right bookings fall by about a third on payment and by half or more on doctor and schedule.

**Independence and disclosure:** Self-funded editorial. Jev ran on my own TypeSafe account; TypeSafe billed $17.68. Bivia ran on Articul8’s free tier. Articul8 is a past sponsor of The CTO Advisor’s work; it didn’t pay for these runs, but at my request it raised the account’s request limit partway through them, on November 10, 2026. Google Cloud is a client, and Gemma is a Google model. None of TypeSafe, Articul8, Google or Anthropic reviewed this page.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Bivia, premature bookings | −4.8, −1.7, −4.1 points | Doctor, payment, schedule, with the examples minus without, arm E; points of all 48,000 conversations. Intervals [−5.5, −4.3], [−2.5, −0.9], [−4.6, −3.6]. With the defective line’s conversations excluded: −4.2 [−4.7, −3.7], −0.3 [−0.6, 0.0], −3.4 [−3.8, −3.0]. |
| Bivia, wrong bookings | −2.7, −1.9, −2.2 points | Doctor, payment, schedule, with the examples minus without, arm E; points of all 48,000 conversations. Intervals [−3.2, −2.2], [−2.8, −1.2], [−2.7, −1.8]. With the defective line’s conversations excluded: −2.0 [−2.2, −1.7], −0.5 [−0.9, −0.2], −1.5 [−1.8, −1.3]. |
| Bivia, clean right bookings | +2.9, −1.5, −0.1 points | Doctor, payment, schedule, with the examples minus without, arm E; points of the patients who wanted a booking (23,220, 27,840, 22,020). Intervals [+2.3, +3.6], [−3.1, +0.1], [−1.2, +0.9]. With the defective line’s conversations excluded: +2.8 [+2.2, +3.4], −1.9 [−3.5, −0.2], −0.4 [−1.5, +0.7]. |
| Bivia, patients who needed a person and reached one | +7.1, +17.7, +7.5 points | Doctor, payment, schedule, with the examples minus without, arm E; points of the patients who needed a person (20,520, 15,600, 21,720). Intervals [+5.4, +8.7], [+15.0, +20.5], [+6.0, +9.0]. With the defective line’s conversations excluded: +6.8 [+5.3, +8.6], +17.5 [+14.8, +20.7], +7.5 [+6.0, +9.0]. |
| Bivia, patients who wanted a booking and got none | +5.6, +7.5, +9.8 points | Doctor, payment, schedule, with the examples minus without, arm E; points of the patients who wanted a booking. From the cells’ outcome counts (5,521 to 6,813, 3,124 to 5,200, 4,501 to 6,654); no interval computed. |
| Jev, premature bookings | −1.7, −0.1, −1.0 points | Doctor, payment, schedule, with the examples minus without, arm E; points of all 48,000 conversations. Intervals [−2.0, −1.5], [−0.4, +0.2], [−1.3, −0.8]. With the defective line’s conversations excluded: −1.6 [−1.9, −1.4], 0.0 [−0.3, +0.3], −0.9 [−1.0, −0.7]. |
| Jev, wrong bookings | −0.8, −0.3, −0.5 points | Doctor, payment, schedule, with the examples minus without, arm E; points of all 48,000 conversations. Intervals [−0.9, −0.6], [−0.7, −0.1], [−0.6, −0.3]. With the defective line’s conversations excluded: −0.6 [−0.7, −0.5], −0.2 [−0.6, 0.0], −0.3 [−0.3, −0.2]. |
| Jev, clean right bookings | +1.2, +0.1, +0.8 points | Doctor, payment, schedule, with the examples minus without, arm E; points of the patients who wanted a booking (23,220, 27,840, 22,020). Intervals [+1.0, +1.5], [−0.1, +0.3], [+0.6, +1.0]. With the defective line’s conversations excluded: +1.2 [+1.0, +1.5], 0.0 [−0.2, +0.2], +0.8 [+0.5, +1.0]. |
| Jev, patients who needed a person and reached one | +0.9, +10.5, +0.6 points | Doctor, payment, schedule, with the examples minus without, arm E; points of the patients who needed a person (20,520, 15,600, 21,720). Intervals [+0.3, +1.6], [+9.2, +12.0], [−0.1, +1.4]. With the defective line’s conversations excluded: +0.9 [+0.3, +1.6], +10.6 [+9.2, +12.1], +0.6 [−0.1, +1.4]. |
| Gemma, premature bookings | +0.8, +0.8, +0.3 points | Doctor, payment, schedule, with the examples minus without, arm E; points of all 48,000 conversations. Intervals [+0.3, +1.4], [+0.3, +1.4], [+0.1, +0.6]. With the defective line’s conversations excluded: 0.0 [−0.1, +0.1], 0.0 [0.0, 0.0], 0.0 [−0.1, 0.0]. |
| Gemma, wrong bookings | +0.8, +0.8, +0.3 points | Doctor, payment, schedule, with the examples minus without, arm E; points of all 48,000 conversations. Intervals [+0.3, +1.3], [+0.3, +1.4], [+0.1, +0.6]. With the defective line’s conversations excluded: 0.0 [−0.1, +0.1], 0.0 [−0.1, 0.0], −0.1 [−0.1, 0.0]. |
| Gemma, clean right bookings | +0.5, +1.1, +1.4 points | Doctor, payment, schedule, with the examples minus without, arm E; points of the patients who wanted a booking (23,220, 27,840, 22,020). Intervals [+0.1, +1.0], [+0.3, +2.1], [+0.8, +2.0]. With the defective line’s conversations excluded: +0.5 [0.0, +1.0], +1.1 [+0.3, +2.2], +1.4 [+0.8, +2.0]. |
| Gemma, patients who needed a person and reached one | −1.2, −0.2, −1.3 points | Doctor, payment, schedule, with the examples minus without, arm E; points of the patients who needed a person (20,520, 15,600, 21,720). Intervals [−2.0, −0.7], [−1.0, +0.4], [−2.1, −0.8]. With the defective line’s conversations excluded: −0.8 [−1.4, −0.4], +0.1 [−0.5, +0.6], −1.3 [−2.1, −0.8]. |
| Haiku, premature bookings | −0.3, +0.2, −0.3 points | Doctor, payment, schedule, with the examples minus without, arm E; points of all 48,000 conversations. Intervals [−0.6, 0.0], [−0.1, +0.7], [−0.5, −0.1]. With the defective line’s conversations excluded: −0.5 [−0.7, −0.4], −0.1 [−0.3, −0.1], −0.4 [−0.6, −0.2]. |
| Haiku, wrong bookings | +0.1, +0.4, 0.0 points | Doctor, payment, schedule, with the examples minus without, arm E; points of all 48,000 conversations. Intervals [−0.1, +0.4], [+0.1, +0.8], [0.0, +0.1]. With the defective line’s conversations excluded: −0.1 [−0.1, 0.0], 0.0 [0.0, +0.1], 0.0 [−0.1, 0.0]. |
| Haiku, clean right bookings | 0.0, +0.2, −0.4 points | Doctor, payment, schedule, with the examples minus without, arm E; points of the patients who wanted a booking (23,220, 27,840, 22,020). Intervals [−0.3, +0.1], [0.0, +0.4], [−0.6, −0.1]. With the defective line’s conversations excluded: −0.1 [−0.3, +0.1], +0.1 [0.0, +0.3], −0.4 [−0.6, −0.2]. |
| Haiku, patients who needed a person and reached one | +0.6, −1.3, +1.2 points | Doctor, payment, schedule, with the examples minus without, arm E; points of the patients who needed a person (20,520, 15,600, 21,720). Intervals [−0.1, +1.4], [−2.3, −0.1], [+0.8, +1.5]. With the defective line’s conversations excluded: +0.7 [0.0, +1.4], −1.3 [−2.3, −0.1], +1.2 [+0.9, +1.6]. |
| The defective line’s exclusion | 1,062, 1,525, 955 conversations | Doctor, payment, schedule: every conversation where any of nine arm E reader configurations (Gemma and Haiku with and without examples, Haiku with thinking on, Jev and Bivia with and without examples) met the defective refusal line, dropped from every cell of that dimension. |
| Gemma and Haiku with consent decided in code | 0.04 points or less | The largest change the examples made to premature or wrong bookings in arm F, either reader, any dimension; every interval within 0.08 points of zero. Jev and Bivia didn’t run arm F with examples. |
| What arm F costs in clean right bookings | About 53%, 31%, 70% fewer | Doctor 53%, payment 31%, schedule 70% fewer clean right bookings than arm E without examples, for Jev and Bivia alike (Bivia 11,548 to 5,403, 17,399 to 11,942, 10,343 to 3,151). Gemma and Haiku fall about the same. |
| What the hosted runs cost | $17.68 for Jev; $0 for Bivia | TypeSafe’s bill on my account. Bivia ran on Articul8’s free tier: about 957 million billable tokens by the session records, at a 1,000-token floor per call, while actual input averaged about 200 tokens a call without the examples and about 770 with them. |

## Authority placement (DAPM)

**Reading:** Decision authority: who makes the runtime decision at this layer, and can the enterprise see and override it?

| Layer | Arm E, the reader’s yes counts | Arm F, code decides consent | Basis |
| --- | --- | --- | --- |
| Layer 3 (+1) (Applications) | Retained | Retained | Who decides the next step in the workflow. Code runs the driver in both arms; the reader answers closed questions inside it. |
| Layer 2C (Reasoning) | Ceded | Retained | Who determines that the patient agreed. In E the reader’s yes is the consent, so whatever the examples do to the reader reaches the booking. In F only the patient’s explicit words, read by code, can agree; the reader may decline, never agree. That’s why the examples barely moved Gemma’s and Haiku’s premature and wrong bookings under F: the placement contains what the examples can reach. |

## What this does not establish

- Which reader is better. Four readers, one workflow, one examples set: the lab ranks nothing, and the numbers aren’t a reason to pick a vendor.
- Anything about patient safety or care. The patients are synthetic and “right” means agreement with a scripted answer.
- That the differences would hold on a rerun. Each cell is one run; the intervals resample patients, not runs.
- That either vendor can’t be tuned. I read their public documentation; I didn’t ask them.
- That examples in a state field act like examples in a prompt, or that Articul8’s “drop-in replacement” claim, which is about the API, says anything about behavior.
- What the other System One knobs do. The probability bar, abstain, deep mode and the option lists were left at their defaults.
- That arm F contains the examples for Jev and Bivia. Only Gemma and Haiku ran arm F with them.

## Open questions

- **How much of the split is the model, and how much is the delivery?** Open. These cells can’t separate them. Telling them apart takes the same examples delivered the same way to every reader, or a vendor-supported way to supply examples, and repeated runs to see how much a single cell moves on its own.
- **Does a probability bar in code make the examples matter less?** Open. Every System One answer here came with a probability the driver ignored. A bar code sets, with “ask again” below it, is the design to test next, with the examples turned on top of it.

## Objection handled

**Did you test whether examples transfer, or two different interventions?**

Two interventions, and that’s a fair hit. Gemma and Haiku got the six cases in the prompt. Jev and Bivia got them as text in the state field, because neither API has a field for examples. The biggest split in the results follows that line. Both System One readers cut premature bookings 1.0 to 4.8 points on doctor and schedule and got 10.5 and 17.7 more points of the payment patients who needed a person to one; the two prompt-delivered readers moved premature bookings by under a point either way. Their starting points differ too: without examples, Jev and Bivia booked prematurely in 6.7% and 7.4% of doctor conversations and Haiku in 1.8%, so Haiku had less room to fall.

So read it as what a buyer gets: the same cases, through the integration each reader offers. Inside one delivery the split still shows. On schedule, Gemma and Haiku both had the examples in the prompt. Gemma’s clean right bookings rose 1.4 points and Haiku’s fell 0.4; the share of patients who needed a person and reached one fell 1.3 for Gemma and rose 1.2 for Haiku. All four intervals clear zero, and none of it runs through the defective line. Jev and Bivia moved the same way, by very different amounts.

That’s the ruling’s scope. It doesn’t say examples never transfer. It says you can’t assume they do.

## Sources

- [Articul8: Bivia developer page](https://articul8.ai/bivia)
- [Articul8: Bivia API reference](https://docs.articul8.ai/bivia)
- [TypeSafe: System One API documentation](https://docs.typesafe.ai)
- [The Clinic bench (GitHub)](https://github.com/kltownsend/clinicbench)
- [Run records, data-v3: the 18 Jev and Bivia cells](https://github.com/kltownsend/clinicbench/releases/tag/data-v3)
- [Run records, data-v2: the Gemma and Haiku cells released with Lab 029](https://github.com/kltownsend/clinicbench/releases/tag/data-v2)
- [Run records, data-v4: Gemma’s and Haiku’s examples cells on doctor and schedule](https://github.com/kltownsend/clinicbench/releases/tag/data-v4)

## Summary

A System One model answers a typed question with an answer and a probability, and nothing else. That makes it a candidate for the reader’s seat in a workflow code runs: the driver asks a closed question about what a patient meant, and the reader answers. TypeSafe’s Jev and Articul8’s Bivia are two such models, hosted. On the Clinic’s frozen bench I put each in that seat beside Gemma 4 26B and Claude Haiku 5.5, then gave all four the same six labeled examples per question.

In arm E, where the reader’s yes is the patient’s consent, the examples moved the four readers differently on the bench’s lead measure, clean right bookings, and on getting the patients who needed a person to one. Bivia got far more of those patients to a person and cut premature and wrong bookings, but booked fewer of the patients who wanted a booking. Gemma booked cleanly more often and got fewer patients to a person. Haiku and Gemma, given the examples the same way, moved in opposite directions on both measures on schedule. Each cell is 48,000 conversations, one run, with intervals from resampling the 1,000 patients.

One defect in the patient lines runs through the premature-booking numbers. With it excluded, no reader’s premature or wrong bookings rose with the examples, and Gemma’s increase disappears. The clean-booking and hand-off changes barely move. What survives still differs by model, and that’s the ruling: an examples set is something you validate on the model you run it on.

## Detail

The bench is the Clinic’s, frozen by hash: Lab 23’s code-driven scheduling driver, 1,000 synthetic patients of 18 types, each heard at 48 settings of one unclear part of the request. Code runs every step. When it can’t parse a reply, it asks the reader a closed question, with the options it would accept plus “unclear.” A premature booking, which the bench calls a break, is a booking written while something it depended on, the time, the doctor or whether the patient will pay, was still open in the log. A clean right booking is a correct booking with nothing left open; the bench leads with it because a correct booking made on a guess isn’t a decision anyone made. Nothing grades the reader but the scripted truth.

A System One model has several knobs. Code can set a bar the answer’s probability has to clear. Bivia can abstain, and it has a fast mode and a deep mode. The options the driver offers are a knob too. These cells turn one knob and leave the rest at their defaults: examples. I didn’t fine-tune any reader, Haiku included, and in TypeSafe’s and Articul8’s public documentation, read in November 2026, I didn’t find a way to train either model on your own cases or a field for examples.

So the six-example set went to all four readers two ways. Gemma and Haiku got it in the prompt. Jev and Bivia got it as text in the state field, ahead of the case. That’s my adaptation, not either vendor’s tuning, and it may not act the way prompt examples do.

Bivia moved most. Its premature and wrong bookings fell on every dimension, and 7 to 18 points more of the patients who needed a person reached one. The price was on the other side of the clinic: 5.6 to 9.8 points more of the patients who wanted a booking ended without one. Its clean right bookings rose 2.9 points on doctor and fell on payment, and that payment drop clears zero once the defective line is out. Jev moved the same way, smaller, except on payment, where it got 10.5 more points of the patients who needed a person to one without changing anything else much. Gemma booked cleanly more often on every dimension and got fewer patients to a person on doctor and schedule. Haiku went the other way from Gemma on schedule on both.

One defect in the patient lines runs through the premature-booking numbers. When a patient turns down an offer that’s wrong on the doctor or the appointment, the line says “No, I can’t do that one” without saying which part was wrong, and can then name the offered time as one that works. Readers often hear that as yes, and the examples change how often: Gemma’s yes rate on that line went from 11% to 34%, Bivia’s from 52% to 22%. With every conversation that met the line excluded, Gemma’s increase in premature bookings disappears and no reader’s premature or wrong bookings rise. The clean-booking and hand-off changes barely move. A second defect makes a patient who contradicts themselves sound like a clean correction. Both are left in on purpose; the recorded conversations show them.

In arm F, code decides consent and the reader may decline but never agree. Gemma and Haiku ran there with and without the examples, and the examples moved their premature and wrong bookings by 0.04 points or less: the placement kept the examples from reaching the booking. It costs bookings. Every reader made about a third fewer clean right bookings on payment and half or more fewer on doctor and schedule than in arm E. Jev and Bivia ran arm F without examples only, so I can’t say the same holds for them.

## Method and disclosure

Self-funded editorial. The bench is the Clinic’s, frozen by hash; every cell here carries the same instrument hash as the Clinic’s published cells. The Gemma cells ran on two DGX Sparks I own and the Haiku cells through Anthropic’s API. The Jev and Bivia readers are clients I wrote for this lab, standard library only, and ran from a Mac Studio against each vendor’s hosted API. Neither API has a field for examples, so the six-example set went into the state field ahead of each case: my adaptation, not either vendor’s tuning. Each cell is one run; its intervals resample the 1,000 patients, not reruns. The defect-excluded values drop every conversation where any reader met the defective refusal line and use the same paired intervals. Gemini and ChatGPT peer-reviewed the results and two drafts of this analysis, and the page was revised against their findings.

---
*Layer2C Labs · The CTO Advisor LLC (The Advisor Bench) · labs.layer2c.com*