The validator determines done, not the loop

Measured on deterministic coding tasks, where a test harness gives an unfalsifiable pass/fail. The rulings are about where the "determines done" authority sits in an agentic loop. The open edge is domains without an executable evaluator.

Self-funded. No vendor paid for this answer, and the whole run cost about $4.60.

Does a local bug-fix agent need a frontier tier standing behind it? That was the pitch this lab was built to test. I built a three-tier escalation chain on an NVIDIA DGX Spark, gated it with a deterministic test harness, and metered every call. The tasks were real bugs mined from open-source repositories, more-itertools, httpx, PyJWT, h2, and others, by git archaeology: roll the source back to before a fix, then use the fix's own test suite as an unfalsifiable pass/fail gate. The shape is a support desk. An L1 analyst attempts the ticket, but a deterministic escalation rule, not the analyst's self-assessment, decides whether it goes to L2. Here the L1 is a local model on the Spark, the evaluator is a test harness, and the frontier model is the L2 it escalates to.

The evaluator is the part worth staring at, because it is not a model. It runs the same tests against every tier's output and returns the same verdict regardless of which model wrote the code. It cannot be persuaded, anchored, or biased. No model judges its own work or another model's. That property is what makes the chain reliable, and it is the finding the rest of the numbers support.

The loop retried its way to the same wrong answer

The obvious objection is that self-correction loops are the whole point of agents. They are the part that mostly does not pay. The band where a repair loop adds value, where the model understands the fix but makes an implementation error a round of feedback can correct, was about 5% of tasks. Difficulty is bimodal: 42% of tasks were trivially easy and passed on the first attempt, 54% were too hard for a single pass and needed a more capable model, and only the thin band between them is where iteration helps. That is too narrow to build a production strategy around.

And for a deterministic model the loop pays nothing at all. At temperature 0 the local model gives the same output for the same input, so error feedback changes the prompt text but not the answer. On near-miss tasks where six of seven tests passed, all four repair attempts produced byte-identical output: 4x the latency, zero improvement. For a non-deterministic reasoning model the story inverts, and a second try can recover a near-miss, though the specific recovery figure first reported here sits among the statistics the validator audit superseded. The variable that decides whether a repair loop is worth running is not the quality of the feedback. It is whether the model can produce a different output given the same feedback.

Reviewing everything makes it worse, which surprised me more than the loop result. Every configuration that applied a review pass to all outputs degraded accuracy, whether the reviewer was the local model or a frontier governor. The failure was consistent: asked to review a decision, the model promotes correct low-confidence calls to confident wrong ones, because it asks what if this matters more often than what if it does not. A frontier reviewer shown the local model's answer also anchored to it, confirming a call it would have made differently on a fresh look. Review earns its place only when a deterministic trigger says an output is worth challenging. Otherwise skip it.

The audit ate the escalation number

As first reported, tiered escalation looked like the architecture that paid. Local-only cleared 25%, two of the 8 tasks. Local plus a repair loop cleared the same 25%. Three tiers gated by the evaluator cleared 75%. Then I audited the harness itself, and this is the correction the lab is issuing. Nine of its validator checks were invalid, empty or self-contradictory gates that no output could pass, and they had booked escalation events for tasks that were never solved. Corrected, most of the apparent escalation gain dissolves. The tasks the small model could do, it did on its own; the rest were unsolvable on the instrument, not rescued by a bigger model. Those cases reclassify as persistent failures rather than escalation, which changes the reported escalation statistics but not the engineering conclusion: they remain unsolved under the tested strategy. Lab 9 then re-ran the same validator at rigor and put the number where it belongs. A 12B model at Q4 and the 26B mixture-of-experts (MoE) cleared the solvable set, 21 of 39, with zero confirmed false passes across 585 attempts. Code bug-fixing sits inside a lower-class model's reach, not above it. The frontier tier bought throughput, not correct answers, and the honest headline is how much the small model did on its own.

The cost model survives the correction, because it never depended on who got the credit. Each task starts free on the local box and pays for a frontier call only when the evaluator rejects, so cost scales with the local failure rate: about three cents a task at a 75% local pass rate, sixteen cents at the 25% this task set produced, and never more than frontier-only. The whole five-day run cost about $4.60 across 156 API calls. A MoE model in FP8 ran at 48 tokens a second on the GB10 against 1 for a dense model, which is what makes a free local first attempt practical. And published token rates overstated the real frontier bill by four to five times, so paper cost projections are not the real economics.

Two things set the ceiling, and neither is the loop. Capability came first: the MoE model passed tasks that a larger dense model and a coding-specialized model failed across every mode, so architecture amplifies capability but cannot substitute for it. The evaluator came second: the harness is the system's judgment, and a task with a bad test was unsolvable by every tier regardless of capability. The lever is not prompt engineering. It is the quality of the deterministic check.

Who determines done

The honest bounds are tight. The culminating escalation ran on 8 tasks, 26 calibrated, good for an architecture verdict and not a benchmark of pass rates. Two failures were not capability findings: one task had a bad test that passes on buggy code, another was an edge-case blind spot shared across every model family. The third tier did not earn its place, solving nothing the middle tier failed in this set. Scale economics went unmeasured; a better-calibrated local model with a 50 to 70% pass rate would change the cost curve. And the evaluator only checks what the tests cover. A correct fix that fails a bad test, and a wrong fix that passes thin tests, are both invisible to it.

I started by asking where judgment should live in a local-first agentic system. The data made me change the question. It is not where judgment lives. It is who determines done, and the honest answer is that it should not be a model at all. We have built an industry reflex around agents that loop until they are satisfied, and on this hardware, for a model deciding its own done, that loop is theater. Four repair attempts, byte-identical output, nothing gained. The evaluator was the only part ever doing the work. Deterministic Code In The Loop (DCITL) is not a constraint on the agent; it is the part of the system you can actually trust. Two questions stay open. Whether sampling at higher temperatures breaks the convergence and reopens the local repair loop was never tested, because escalation made it unnecessary. And the whole finding rests on coding's executable evaluator. Triage, content generation, and planning do not have one, and the subjective phase of this experiment showed proxy-gated review is fragile. Whether determines-done authority can be made deterministic enough outside code is where DCITL either generalizes or stops.

The numbers

Where each layer belongs

Assessments at the time of the lab

Method and disclosure

Methodology Note: I identified a logic error in nine validator checks that incorrectly recorded escalation events. After correcting the validators, those cases are classified as persistent failures rather than escalation. This changes the reported escalation statistics but does not change the engineering conclusion of the lab: these cases remain unsolved under the tested loop-control strategy.

Self-funded, no sponsor, run over five days on a single NVIDIA DGX Spark (GB10) with real open-source bugs and about $4.60 of frontier API spend. Local models ran on Ollama and vLLM; the frontier tiers were o3 and gpt-5.5.

Tasks were mined from real repositories (more-itertools, httpx, PyJWT, h2, and others) by git archaeology: roll the source back to before a fix, and use the fix’s own test suite as an unfalsifiable pass/fail gate. Each tier ran from a clean workspace with no prior-tier context, and the deterministic harness was the sole authority on accept, reject, and escalation.

The experiment code, the mode definitions, the per-run traces, and the failure-mode catalog are captured in the raw lab detail. The specific task set and the tuning particulars stay in the working notes.

The essay on this page was rendered from the lab’s frozen structured record by a model writing under this practice’s voice specification, with a deterministic gate checking every figure against the record before publication. The record is the canonical surface: where prose and record disagree, the record rules.