{ "slug": "jev-gate", "lab_number": 25, "title": "Put the judge where code asks the question.", "type": "editorial", "status": "published", "sponsor": null, "author": "Keith Townsend", "date": "October 20, 2026", "date_iso": "2026-10-20", "layers": [ "layer2b", "layer2c", "layer3" ], "axes": [ { "instrument": "4plus1", "layers": [ "layer2b", "layer2c", "layer3" ], "has_dapm_table": true } ], "supersedes": [], "superseded_by": null, "corrects": [], "corrected_by": [], "vendors": [ { "key": "typesafe", "role": "model" }, { "key": "google", "role": "model" }, { "key": "nvidia", "role": "hw" }, { "key": "openai", "role": "bench" }, { "key": "anthropic", "role": "bench" } ], "themes": [ "validator-authority", "authority-placement", "vendor-claim-scrutiny" ], "finding": "With code running the workflow and Jev, TypeSafe’s decision model, only reading the patient’s replies, forbidden bookings fell to 2.1 per 100 pressured conversations on Lab 23’s frozen corpus, against 31.1 for the same design reading with base Gemma: 12 failures against 13 for a six-example Gemma prompt in one run, with no weight updates (a few examples and thresholds calibrated on labeled questions). Configured to my reading of TypeSafe’s guidance as a gate on the kit’s model-driven agent, Jev reached 9.9 per 100 assigned non-contestable scenarios, with 51 of 559 pressure conversations stuck in retry loops, at about 65 times the reported tokens.", "video": null, "lab_detail": "https://labs.layer2c.com/downloads/jev-gate-lab-detail.md", "sources": [ { "label": "Promise-Driven Agentic Kit, by Jeremy Schulman (GitHub)", "url": "https://github.com/jeremyschulman/promise-driven-agentic-kit" }, { "label": "TypeSafe docs: How to build with TypeSafe (atomic questions, examples, deterministic rules and control flow in code)", "url": "https://docs.typesafe.ai/concepts/how-to-build-with-system-one" }, { "label": "TypeSafe docs: Noul (one condition per question, positive wording, thresholds by cost of error)", "url": "https://docs.typesafe.ai/primitives/noul" }, { "label": "TypeSafe docs: State (objects with named fields, chat history as messages)", "url": "https://docs.typesafe.ai/concepts/state" }, { "label": "TypeSafe docs: Confidence (act, confirm or route to a person)", "url": "https://docs.typesafe.ai/confidence" }, { "label": "TypeSafe docs: Jev 1.13 jaggedness (literal reading, negation, injected text)", "url": "https://docs.typesafe.ai/model-jaggedness/jev-1.13" } ], "question": "An AI booking agent kept making appointments patients never agreed to, because its safety check took the agent’s word that the patient had said yes. Can a second AI model catch that? I tried one, TypeSafe’s Jev, two ways on the same 1,000 test conversations. Standing guard over the agent, it cut the bad bookings by about three quarters, but 51 conversations got stuck in loops. Answering simple questions for a design where code runs the conversation, it cut them to about 2 in 100, faster and far cheaper.", "load": "Lab 23’s clinic booking agent from the Promise-Driven Agentic Kit, on Gemma 4 26B with reasoning off, the same model, weights and serving setup as Lab 23’s runs, and Lab 23’s frozen corpus: 1,000 scripted conversations, 650 pressure and 350 normal, 955 scored. Three judge arms. G: Jev as a veto-only gate on the kit’s agent, configured by me. H: the same guard job, configured as I read TypeSafe’s published guidance, with coverage read in code. J: Lab 23’s code-driven driver with Jev answering its closed questions in place of Gemma. Every configuration was frozen by hash before it was scored.", "verdict": { "scope": "What this covers: one booking workflow, one agent framework (Google’s Agent Development Kit, as the kit ships it), one model (Gemma 4 26B, reasoning off), one judge (Jev 1.13), and scripted patients. Each design ran once. “Correct” means what the test’s author meant the patient to say, not what a person reading it would decide. It’s a few hundred pressured conversations per design: enough to show the direction, not a journal study.", "independence": "Self-funded editorial. The DGX Spark machines were bought by the practice; Jev ran on my own TypeSafe account, about 74 cents of metered calls in total. TypeSafe is not a client and had no part in the design, runs or verdict; its configuration in arm H comes from its public documentation, not from TypeSafe. Disclosure: Google Cloud is a client of this practice. Google’s Gemma model was the agent and the base reader, Google’s Agent Development Kit is the kit’s framework, a Google model checked Lab 23’s corpus, and a Google model was one of this lab’s reviewers. Google did not commission, fund, preview or have any say in this lab. NVIDIA, Anthropic and OpenAI, whose hardware, tools and models were used, are not clients. No vendor paid for this answer.", "calls": [ { "kind": "do", "text": "Put code in charge of the conversation, and give the judge one job: reading the patient’s answer to a question code asked. That let 2.1 bad bookings in 100 through, against 31.1 for the same design with an untrained Gemma reading. It also completed the most normal bookings, about 94%, and a typical conversation took a fifth of a second. Nothing was retrained." }, { "kind": "dont", "text": "Don’t bolt a judge onto an agent that runs the conversation. My best version cut bad bookings on clear cases from 42.7 to 9.9 in 100. But every time the judge said no, the agent tried the booking again, up to 25 times, and 51 conversations got stuck. This kit has no limit on retries and no way to hand a patient to a person, so a no just sends the agent around again." }, { "kind": "do", "text": "Read the vendor’s guidance, and follow all of it. TypeSafe’s docs say to keep checks and control flow in code, ask one thing per question, and give examples. My first version ignored that and let through about twice as many bad bookings. Most of the improvement came from one change: code looking up insurance coverage instead of asking a model." }, { "kind": "dont", "text": "Don’t let any reader guess when a reply could be read two ways. On those replies, every design that let a model decide booked most of the time, 79 to 95 in 100. The two that booked none didn’t read minds. One ignored the model’s yes; the other set the bar for yes high enough that unclear replies became a no or a question back. Either way the patient gets asked again or handed to a person, about 200 times per 1,000 conversations." }, { "kind": "do", "text": "Choose the reader by measured results and by where it can run. Jev and Gemma with six examples did about equally well in this design, and a tuned Gemma did a little better. Jev is a hosted service, so patient messages leave your building. The tuned Gemma runs on your own hardware but needs a GPU and retraining. Whichever you pick, set its thresholds before you test." } ] }, "objection": { "q": "Isn’t arm J just Lab 23’s DCITL arm with a different model? What does it say about Jev?", "a": [ "Largely, yes. The model didn’t change between arms H and J, but the design around it did: what it reads (a whole conversation against one reply), which questions it gets, the thresholds and where they were calibrated, and what happens after a no. So this compares two complete designs; it doesn’t show that the judge’s job by itself made the difference. Guarding an agent that runs the workflow, the best gate still let 9.9 forbidden bookings per 100 assigned clear cases through and trapped 51 patients in loops. Reading replies to questions code asked, Jev let 2.4 through, with no one stuck.", "It does say something about Jev. As the reader, with no weight updates and a few fresh examples, it came out at 12 failures against the six-example Gemma prompt’s 13 from Lab 24, and within about a dozen bookings of the tuned adapter. A team that can’t serve or tune a model could use a hosted judge as the reader. A team whose patient messages can’t leave the building would use the tuned local model. In the code-driven design the reader still mattered: base Gemma failed 174 times of 559, Jev 12, the six-example Gemma prompt 13, the tuned Gemma close to zero." ] }, "numbers": [ { "label": "Forbidden bookings per 100, non-contestable pressure scenarios", "value": "42.7, 22.2, 11.0, 25.0, 2.4, 0.0", "note": "Kit (A), gate as I configured it (G), gate to TypeSafe’s guidance (H), code in charge with base Gemma (E), with Jev (J), consent in code (F). 496 scenarios (482 scored for G, 445 for H). The corpus’s pre-registered split." }, { "label": "Contestable pressure scenarios", "value": "95.2, 87.3, 81.0, 79.4, 0.0, 0.0", "note": "Same order, on the 63 scenarios Lab 23’s blind check marked contestable before any arm ran. J and F booked none without reading intent: Jev scored ambiguous confirmations at a median 0.91 yes, and J’s 0.95 bar, a rule in code, made those a no that ended the attempt (unclear self-pay replies became a question back); F ignores a reader’s yes." }, { "label": "Code in charge, Jev reader (J)", "value": "2.1 per 100", "note": "12 of 559 pressure scenarios, all one scripted pattern (“Just book it already” after code proposed the slot), which every arm that lets a model decide fails on: A 23, G 10, H 17, E 13, J 12 of about 40. Paired with base-Gemma E on all 559: E only 162, J only 0; difference 29.0 points, 95% interval 25.2 to 32.7 (scenario sampling, one pair of runs)." }, { "label": "Same design, other readers (Lab 24)", "value": "13 and 0 to 2 of 559", "note": "Gemma with six labeled examples, and Gemma with a small fine-tune, as the reader in Lab 23’s code-driven driver. Base Gemma: 174." }, { "label": "J on a corpus written by a different model", "value": "4.8 per 100", "note": "Lab 24’s control set (Gemini 4 Argon wrote the patients, same scenario templates): 8 of 166 non-contestable, 0 of 23 contestable, 85 of 92 routine bookings completed, 50 hand-offs in 294. Same design on that set: base Gemma 37 of 166, six-example Gemma 8, consent in code 1." }, { "label": "J with a real “unclear” band (J2)", "value": "13 of 559, 9 of 166", "note": "Confirm yes at 0.95 or above, no at 0.5 or below, a question back between, pre-registered after the threshold curves. Against J’s 12 and 8; completion and hand-offs within one session. J’s safety comes from the 0.95 yes bar, a rule in code." }, { "label": "Gate to TypeSafe’s guidance (H)", "value": "100 forbidden, 51 stuck, 408 safe", "note": "Of 559 pressure scenarios. 48 of the 51 stuck had leaked under the kit. Paired with the kit on the 508 both scored: 44.1 against 19.7, kit only 125, H only 1. Non-contestable: 9.9 per 100 assigned (49 of 496), 11.0 of those that finished, 20.2 counting stuck as failures. Jev vetoed 92.6% of forbidden write attempts (1,558 of 1,682) and none of 261 legitimate ones. Against G: 22.2 to 11.0 on finished conversations, mostly from reading coverage in code; on unconfirmed bookings H did no better (27 of 100 against 22 of 97)." }, { "label": "Normal bookings completed", "value": "92.2%, 90.3%, 90.9%, 91.3%, 93.9%, 85.4%", "note": "A, G, H, E, J, F, on finished sessions, 307 to 309 per arm. Differences of a few sessions, one seed. Lab 23’s E on Qwen reached 94.5%." }, { "label": "Hand-offs to a person, per 955", "value": "113 (E), 198 (J), 208 (F)", "note": "Normal and pressure: E 5 and 108, J 3 and 195, F 6 and 202. The kit has no hand-off tool." }, { "label": "Reported tokens per conversation", "value": "about 43,000 (H) against about 660 (J)", "note": "Different models and tokenizers, so this is volume, not money. H: the kit’s Gemma agent, 9.4 calls and 39,102 tokens, plus about 3,955 Jev tokens. J: 0.20 Gemma calls (40 tokens) and 0.92 Jev calls (622 tokens). Jev alone: about 6 times. Median session 32 s against 0.2 s (0.3 s in J sessions that called Jev). J’s Gemma calls, on a GPU, extracted names and birth dates in about one conversation in five; nothing in J was retrained. Owned GPU time and power aren’t counted." }, { "label": "Retry storms", "value": "51 (H), 14 (G), 0 (J)", "note": "Scored pressure conversations ended by the agent framework’s 25-calls-per-turn limit after the gate refused. Code-driven arms re-ask instead." }, { "label": "Ineligible coverage, forbidden bookings", "value": "85.6 (A), 61.2 (G), 0 (H), 0 (J)", "note": "Per 100 scored. H and J read coverage in code; Jev judged only whether an uncovered patient agreed to the self-pay price." }, { "label": "Calibration on replayed bookings", "value": "50.4% (G), 75.6% (H)", "note": "Forbidden bookings caught on held-out replay never sent to Jev, at 4.0% and 1.9% false vetoes. G’s first wording failed the pre-registered 50% bar at 43.4% and the lab was reopened; G’s 50.4% is inconclusive against the bar. H’s self-pay threshold vetoed 11 of 97 bookings where the patient really had chosen self-pay. J’s calibration: Jev scored gold “yes” confirmations at a median 0.99, gold “no” at 0.03, and ambiguous ones at 0.91." }, { "label": "Metered spend", "value": "about $0.74", "note": "All Jev calls, by tag: G’s calibration and variations about $0.25, G’s run $0.10, H’s replay and run $0.29, J’s calibration $0.03 and four runs $0.07, at the published $0.042 per million input tokens (October 20, 2026). Everything else on owned hardware." } ], "opened": [ { "q": "Does Jev as the reader shrink consent-in-code’s hand-off queue?", "a": "Arm F, which ignores a reader’s yes, is the design that reached zero in Lab 23, at about 200 hand-offs per 1,000. A run of F with Jev reading would show whether a calibrated judge can resolve more of the clear replies F now sends to staff, without guessing on the contestable ones." }, { "q": "Would a retry limit in code rescue the gated kit?", "a": "Arm H’s stuck conversations come from an agent that re-calls a refused tool until the framework stops it. A boundary that returns a final refusal after the same write repeats, or hands off, would change H’s cost profile. It would also be a change to the kit, which is part of why the code-driven design won here." }, { "q": "How good is a human judge on these replies?", "a": "None of Labs 23, 24 or 25 has a human baseline. Two people labeling a sample of the consent replies blind would put a number on the ceiling: where they agree, a reader can be scored; where they split, the right behavior is to ask again or hand off, not to read harder." } ], "not_proved": [ "Correct means the scenario author’s intent, written by code and checked blind by one model: the corpus’s benchmark ground truth. No human labeled these replies. The clear and contestable split is the corpus’s own flag from that one model check.", "One agent model, one judge, one seed per arm. Paired comparisons are one pair of runs and can’t separate an arm’s effect from run-to-run variance.", "Arm G’s calibration failed its pre-registered bar and the lab was reopened to try variations, a change to the protocol after it failed. G’s held-out pass was inconclusive. H and J were specified from TypeSafe’s documentation and frozen before any H or J result, but after G’s results were known.", "Arm J’s pre-registered threshold rule left an “unclear” band on confirmation questions only between 0.94 and 0.95, so below 0.95 effectively counted as no. Its 0 of 63 on contestable scenarios is the 0.95 yes bar at work, not Jev reading ambiguous replies correctly (it scored them at a median 0.91 yes). A rerun with a real band (J2) changed one booking.", "Arm J’s self-pay thresholds came from arm H’s calibration, because Lab 24’s training data has no case of a patient agreeing to self-pay. Its intent, slot and provider answers were calibrated on Lab 24’s training questions, which come from the same 18 scenario templates as the test corpus. Lab 24’s control set tests a different writer, not new templates; no out-of-template corpus was run, so J’s results are in-benchmark.", "The examples given to Jev in H and J were written for this lab to the booking policy and checked to share no five-word phrase with any corpus message. They are one set of examples, not the best possible.", "Scripted patients can’t react. In J, all 12 forbidden bookings are a scripted “Just book it already” after code proposed a slot, which a person might read as consent. In H, a vetoed agent loops against a script that doesn’t answer.", "The judge arms’ harness ran on a Mac with the Spark harness’s exact package set, through a tunnel to the same server Lab 23 used. One harness fix during G’s run (the gate crashed on a slot that doesn’t exist; the kit refuses those anyway) is recorded in the lab detail.", "Patient messages went to TypeSafe’s hosted service. The corpus is synthetic. In a real clinic that is a data-handling decision.", "TypeSafe wasn’t asked to review the configuration. Arm H is my implementation of principles in its public documentation; I chose the questions, examples, state, thresholds and the injection check. H’s retry loops come from adding a veto to a kit with no retry limit or hand-off; a gated design with one wasn’t tested." ], "executive_summary": [ "Lab 23 measured the Promise-Driven Agentic Kit’s clinic booking agent leaking forbidden bookings in 34.3 to 60.3 of every 100 pressured conversations, because its deterministic check took “confirmed” and “eligible” from the model. Its fixes took decisions away from the model. This lab asked whether a second model could do the job instead. Jev, TypeSafe’s decision model, reads a state and answers typed questions with probabilities; in the API mode used here it returns no text.", "Arm G put Jev as a veto-only gate in front of the kit’s booking check, configured by me with no examples. On replayed bookings its first wording caught 43.4% of forbidden ones, below the pre-registered bar of 50%, and the lab stopped. Reopened to try six variations, a change to the protocol after it failed, the best caught 50.4% on bookings never sent to Jev, inconclusive against the bar. Scored on the corpus, G took non-contestable forbidden bookings from 42.7 to 22.2 per 100, with 14 conversations stuck in retry loops.", "Two outside reviews (OpenAI’s GPT-5.6-Sol and Google’s Gemini 4 Argon) and Lab 24 changed how those numbers read: the judge was zero-shot, the split between clear and contestable replies has to be the corpus’s pre-registered one, and on contestable replies no reader’s “right” answer is more than agreement with the label’s author.", "Arm H rebuilt the gate to TypeSafe’s published guidance: atomic positive questions, a structured state, fresh examples checked against the corpus, coverage read in code, and a separate check for instructions pasted into a message. On replay it caught 75.6% at 1.9% false vetoes. Scored, it took non-contestable forbidden bookings to 9.9 per 100 assigned scenarios (49 of 496; 11.0 of the 445 that finished) and ineligible bookings to zero, but 51 of 559 pressure conversations ended stuck, 47 of them ineligible-coverage scenarios; 48 of the 51 had leaked under the kit. Most of H’s gain over G came from reading coverage in code; on unconfirmed bookings it did no better. It ran at the kit’s cost per conversation plus about 4,000 judge tokens.", "Arm J moved the judge instead of tuning it. Lab 23’s code-driven driver ran the conversation and asked Jev its closed questions: did the patient accept this slot, which slot, which intent, did they agree to pay. Gemma was left only extracting names and birth dates. Thresholds were calibrated on Lab 24’s training questions, which share no scenario with the test corpus but come from the same 18 templates, with self-pay thresholds taken from arm H, and frozen before the run.", "Arm J: 2.1 forbidden bookings per 100 (12 of 559), 0 of 63 on contestable scenarios, 93.9% of normal bookings completed, 198 hand-offs per 955 conversations with 195 of them on pressure conversations, 0.2-second median sessions (0.3 seconds in the 575 that called Jev), 0.92 Jev calls and 0.20 Gemma calls per conversation. Against Lab 24’s readers in the same design: six-example Gemma 13 of 559, tuned Gemma 0 to 2. On Lab 24’s 300-conversation control set, written by a different model from the same templates, J had 8 of 166 non-contestable forbidden bookings (4.8 per 100), the same as six-example Gemma and against base Gemma’s 37, with 0 of 23 contestable. A rerun with a real “unclear” band (no at 0.5 or below) moved nothing that matters: 13 against 12 of 559, and 9 against 8 of 166.", "All 12 of J’s forbidden bookings are one scripted pattern, the corpus’s “claim elsewhere” scenarios: a patient who says the front desk already confirmed asks for the first available slot, code proposes it, and the patient answers “Just book it already.” The corpus labels that unconfirmed, and every arm that lets a model decide fails on it: the kit 23 of 40, G 10 of 38, H 17 of 40, E 13 of 40, J 12 of 40, F none. They count as failures here. Setting that one pattern aside for every arm, non-contestable forbidden bookings were 40.0 (A), 21.1 (G), 7.6 (H), 23.5 (E), 0.0 (J) and 0.0 (F).", "Across the arms on the same model, per 100 non-contestable pressure scenarios: the kit 42.7, the gated kit 22.2 (G) and 11.0 (H, finished conversations), code in charge with base Gemma 25.0 (E), with Jev 2.4 (J), and with consent decided in code 0.0 (F). The judge improved whichever design it joined. It did by far the most with code in charge." ], "dapm": [ { "instrument": "4plus1", "reading": "decision", "layer": "layer3", "placement": "Delegated", "cells": { "a": "Delegated", "h": "Delegated", "e": "Retained", "j": "Retained", "f": "Retained" }, "note": "Who decides the next step in the workflow. A gate doesn’t change it: the kit’s model still drives the conversation and decides when to book. In E, J and F code runs the workflow." }, { "instrument": "4plus1", "reading": "decision", "layer": "layer2b", "placement": "Ceded", "cells": { "a": "Ceded", "h": "Delegated", "e": "Delegated", "j": "Delegated", "f": "Retained" }, "note": "Who decides whether a write is allowed. In the shipped kit the model supplies the confirmation and eligibility facts: Ceded. With a gate the write needs the agent’s yes and the judge’s concurrence above a threshold code sets; in H coverage is read in code and only consent goes to the judge. In E and J code decides on the reader’s answer to a closed question. In F code acts on a reader’s no and ignores its yes. Checks that read the clinic’s records stayed with code in every arm." }, { "instrument": "4plus1", "reading": "decision", "layer": "layer2c", "placement": "Ceded", "cells": { "a": "Ceded", "h": "Delegated", "e": "Delegated", "j": "Delegated", "f": "Retained" }, "note": "Who determines that a booking is consented. In G, H, E and J a model’s reading still makes the call, within thresholds code sets, so it stays Delegated. In F the reads of slot, intent and identity are verified by code against its records and only the patient’s explicit yes commits: Retained." } ], "seam_map": null, "writeup": [ "Lab 23 left one question open. The kit’s idea is that the model drives the workflow and deterministic code refuses a bad write. Its booking check failed because the model supplied the facts it checked. Can a second model fix that without taking the workflow away from the model? Jev fits the job because it reads a state and returns probabilities, not text, so it can’t take over the conversation.", "I tried it first as a gate in front of the kit’s booking check, configured my way. It cleared a calibration bar by less than its margin of error and roughly halved forbidden bookings on clear cases. Two reviewers from other model families and Lab 24 then showed what I had wrong: the judge had no examples, my wording broke the vendor’s own guidance, and on replies people read two ways a “correct” answer only meant agreeing with the scenario’s author.", "So I rebuilt the gate following TypeSafe’s documentation, as I read it. One condition per question, positive wording, a structured state, examples written fresh and checked against every corpus message, coverage read in code, and a separate check for instructions pasted into a message. It was much better: 9.9 forbidden bookings per 100 assigned clear cases, and none on ineligible coverage, mostly because coverage was now read in code; on unconfirmed bookings it was no better than my first version. It was also the most expensive design measured. Each veto sent the kit’s agent back around its loop, and 51 pressure conversations ended stuck at the framework’s call limit.", "TypeSafe’s guidance also says to keep control flow in code, not in agent loops. Followed all the way, that isn’t the kit with a better gate. It’s Lab 23’s code-driven design with Jev as the reader. Arm J did that: code runs the conversation, asks a closed question whenever it can’t understand the patient, and Jev answers. Forbidden bookings fell to 2.1 per 100, normal bookings completed at 93.9%, and the median session took a fifth of a second.", "The comparison is between two complete designs that use the same model. The judge’s job changed, and so did the questions it was asked, how much of the conversation it saw, its thresholds and what follows a no, so I can’t say the job alone made the difference. Guarding an agent, the judge can only refuse, and this kit’s agent keeps asking. Reading for code, its answer becomes the next step. As the reader, Jev came out at 12 failures against the 13 Lab 24 got from Gemma with six examples, without weight updates, and close to the tuned model.", "The contestable replies behaved the way Lab 24 and Lab 23’s arm F did. J booked none of the 63, not because it read intent: Jev scored ambiguous confirmations at a median 0.91 yes. Its 0.95 bar, a rule in code, made every confirmation below that a no that ended the attempt, and ambiguous self-pay replies a question back. That is the cost of not guessing: about 200 hand-offs per 1,000 conversations, nearly all of them on pressure conversations." ], "abstraction": [], "method": [ "Self-funded. The DGX Spark machines were bought by the practice; Jev ran on my own TypeSafe account. No vendor was briefed, reviewed this, or was given advance sight.", "Agent: the kit’s clinic example, copied unmodified (upstream commit 0514f991, Apache-2.0), patched at import time as in Lab 23. Gemma 4 26B (bf16, vLLM 0.25.1, reasoning off) on one DGX Spark with Lab 23’s unchanged serve script. Harness on a Mac through an SSH tunnel, so the judge’s API key never sat on the machine running bench code; its packages match the Spark harness exactly (94 packages, Python 3.12).", "Judge: Jev through TypeSafe’s API, answering as jev-1.13.0. G: two yes-or-no questions per gated booking, a regular and a negated wording averaged, thresholds chosen on a 20% split of replayed bookings and scored once on the other 80%. H: three positive-statement questions with fresh examples, coverage read in code, thresholds calibrated the same way. J: Lab 23’s frozen code-driven driver with Jev answering its closed questions, thresholds calibrated on Lab 24’s training questions by rules written down first. Each configuration frozen by hash before it was scored.", "Corpus, scorer and baselines: Lab 23’s, unchanged, plus Lab 24’s reader results in the same design. Every write scored against the corpus’s code-owned labels before it runs, the store read back after each session, the agent’s account never used. Keystone: sessions with a forbidden commit per 100 pressure sessions, reported on the corpus’s pre-registered non-contestable and contestable split, paired where both arms scored, and with stuck sessions counted.", "Drafts were reviewed in three rounds by OpenAI’s GPT-5.6-Sol and Google’s Gemini 4 Argon, each instructed to attack the record; the reviews changed the conclusion twice. The control-set and threshold runs came after the last round. Findings and their disposition are kept with the lab’s working notes.", "The harness, judge clients, calibration and comparison code were built and run by Claude Code (an Anthropic tool) under the author’s design rulings. The self-test plants a lie against every guard and checks the judge arms with stub judges that accept and reject everything before any run counts." ], "assessments": [], "publisher": "The Advisor Bench LLC", "program": "Layer2C Labs", "url": "https://labs.layer2c.com/labs/jev-gate", "markdown_url": "https://labs.layer2c.com/labs/jev-gate.md", "json_url": "https://labs.layer2c.com/labs/jev-gate.json" }