{ "slug": "loop-control", "lab_number": 3, "title": "The validator determines done, not the loop", "type": "editorial", "status": "published", "sponsor": null, "author": "Keith Townsend", "date": "July 1, 2026", "date_iso": "2026-07-01", "layers": [ "layer0", "layer2b", "layer2c" ], "axes": [ { "instrument": "4plus1", "layers": [ "layer0", "layer2b", "layer2c" ], "has_dapm_table": true } ], "supersedes": [], "superseded_by": null, "corrects": [], "corrected_by": [ { "lab": "xeon-rematch", "what": "The escalation statistics. Nine invalid validator checks had booked escalation events for unsolved cases; corrected, the local model was clearing the solvable work on its own and the frontier tier bought throughput, not correct answers." } ], "vendors": [ { "key": "nvidia", "role": "hw" } ], "themes": [ "loop-control", "validator-authority", "agentic-repair" ], "finding": "About 5% of tasks benefit from same-tier repair; the control point is the deterministic validator plus escalation policy, not the loop.", "question": "The pitch was that a local bug-fix agent needs a frontier tier to escalate to. I built the three-tier chain on a DGX Spark, gated it with a deterministic test harness, and metered every call. Then I audited the harness. Nine of its checks were invalid, and they had booked escalation events that were really unsolved cases sitting on broken tests. Corrected, the credit moves: the local model was clearing the solvable bug fixes on its own, and the frontier tier bought throughput, not correct answers. The test still determines done. That is the part that got more true.", "load": "Real bug-fix tasks mined from open-source repositories by git archaeology, each with the fix’s own test suite as an unfalsifiable pass/fail gate. Local generation on a DGX Spark, escalation to frontier models only when the evaluator rejected.", "verdict": { "scope": "Measured on deterministic coding tasks, where a test harness gives an unfalsifiable pass/fail. The rulings are about where the \"determines done\" authority sits in an agentic loop. The open edge is domains without an executable evaluator.", "independence": "Self-funded. No vendor paid for this answer, and the whole run cost about $4.60.", "calls": [ { "kind": "do", "text": "run the work on the small model and expect it to clear the solvable share. Code bug-fixing sits inside a lower-class model’s reach, not above it. Lab 9 put this same validator in front of a 12B model at Q4 and it held the pass set across 585 attempts with zero confirmed false passes. Reserve the capable tier for speed, not for judgment." }, { "kind": "do", "text": "make a deterministic evaluator the authority. A test harness decides accept or reject, drives escalation, and supplies the error feedback. No model judges its own work or another model’s. That gate, not the model and not the loop, is what verifies which fixes actually cleared." }, { "kind": "dont", "text": "let the model decide when it is done. At temperature 0 the local model is deterministic: the same input gives the same output, so a repair loop retries its way to the identical wrong answer. Self-repair added 4x latency and zero accuracy." }, { "kind": "dont", "text": "review every output. Universal review degraded accuracy in every configuration tested. The reviewer promotes correct low-confidence calls to confident wrong ones. Gate review on a deterministic trigger, or skip it." }, { "kind": "dont", "text": "credit escalation with work the local model already did. A later audit of the harness found nine invalid checks that recorded escalation events for tasks that were actually unsolved on broken tests. Corrected, the frontier tier cleared little the local tier had not, and the honest headline is how much the small model did on its own." } ] }, "objection": { "q": "But self-correction loops are the whole point of agents.", "a": [ "They are the part that mostly does not pay. The band where a repair loop adds value, the model understands the fix but makes an implementation error a round of feedback can correct, was about 5% of tasks. Difficulty is bimodal: 42% of tasks were trivially easy and passed on the first attempt, 54% were too hard for a single pass and needed a more capable model, and only the thin band between them is where iteration helps. That is too narrow to build a production strategy around.", "And for a deterministic model the loop pays nothing at all. On near-miss tasks where six of seven tests passed, all four repair attempts produced byte-identical output. The win is not the loop, and once the validator was corrected it was not escalation either. It is the deterministic evaluator, and it showed the small model was already clearing the solvable work on its own. That is a policy you own, not an autonomy you grant the model." ] }, "numbers": [ { "label": "Pass rate, local model only", "value": "25% (2/8)", "note": "Gemma 4 26B-A4B on the DGX Spark, one attempt" }, { "label": "Pass rate, local + repair loop", "value": "25% (no gain)", "note": "temperature 0: same input, same output, same failure" }, { "label": "Pass rate, tiered escalation (as first reported)", "value": "75% (6/8)", "note": "Overstated. Nine invalid validator checks booked escalations for unsolved cases; corrected, the escalation gain largely dissolves and the local tier carried the solvable work. See the methodology note." }, { "label": "Small model clears the work (Lab 9)", "value": "21 of 39 · 0 false passes", "note": "The same validator re-run at rigor: a 12B at Q4 and the 26B MoE cleared the solvable set with zero confirmed false passes across 585 attempts. Correctness lives in the gate." }, { "label": "Repair-loop sweet spot", "value": "~5% of tasks", "note": "bimodal difficulty: 42% trivial, 54% too hard for one pass" }, { "label": "Frontier second-attempt recovery (as first reported)", "value": "67% of near-misses", "note": "Same correction applies; the figure is superseded by the validator audit." }, { "label": "Total run cost", "value": "~$4.60 · 156 API calls", "note": "~$0.16/task at a 25% local pass rate; pay for capability only on rejection" }, { "label": "Local throughput, MoE FP8 vs dense", "value": "48 vs 1 tok/s", "note": "FP8 native on the GB10; makes the free tier practical" } ], "opened": [ { "q": "How much of the work can the local tier actually clear on its own?", "a": "This lab could not say cleanly, because the escalation numbers were contaminated by the invalid validators. Lab 9, Recovered capacity is real, answered it on the corrected instrument: a lower-class model cleared the solvable set behind the same deterministic gate, with zero confirmed false passes across 585 attempts. The small model was doing the work; this lab had mis-credited the tier above it." }, { "q": "Would a non-deterministic local model make the repair loop worth running?", "a": "At temperature 0 the local repair loop added nothing, and tiered escalation made higher temperatures unnecessary to test. Whether sampling breaks the convergence and reopens the local repair loop as a cheaper alternative to escalation is an unexplored axis." }, { "q": "Does this hold in domains without an executable evaluator?", "a": "The finding rests on a deterministic test harness. Coding has one. Triage, content generation, and planning do not: the evaluator signal becomes a weaker proxy, and the subjective phase of this experiment showed proxy-gated review is fragile. Whether determines-done authority can be made deterministic enough outside code is the open question, and it is where DCITL either generalizes or stops." } ], "not_proved": [ "The task set is small. The culminating escalation ran on 8 tasks (26 calibrated). Good for the architecture verdict, not a benchmark of pass rates.", "Two of the failures were not capability findings. One task had a bad test that passes on buggy code; another was an edge-case blind spot shared across every model family.", "The third tier did not earn its place. The most-capable model solved nothing the middle tier failed, in this set. It is an architectural proof point, not a demonstrated advantage.", "Scale economics were not measured. A better-calibrated local model with a 50 to 70% pass rate would change the cost curve; this set produced 25%.", "The evaluator only checks what the tests cover. A correct fix that fails a bad test, and a wrong fix that passes thin tests, are both invisible to it." ], "executive_summary": [ "The experiment reframes loop control away from who controls the loop and toward who determines done. The analogy is a support desk: an L1 analyst attempts the ticket, but a deterministic escalation rule, not the analyst’s self-assessment, decides whether it goes to L2. Here the L1 is a local model on a DGX Spark, the evaluator is a test harness, and the frontier model is the L2 it escalates to.", "The evaluator is not a model. It is a deterministic test harness that runs the same tests against every tier’s output and returns the same verdict regardless of which model wrote the code. It cannot be persuaded, anchored, or biased. That property is what makes the escalation chain reliable, and it is the finding the rest of the numbers support." ], "dapm": [ { "instrument": "4plus1", "layer": "layer0", "placement": "Retained", "note": "Local first-attempt tier on the DGX Spark. A mixture-of-experts model in FP8 (48 tok/s vs 1 for a dense model) makes a free local attempt practical." }, { "instrument": "4plus1", "layer": "layer2b", "placement": "Retained", "note": "The test harness and the escalation orchestrator are deterministic code you run, not a model." }, { "instrument": "4plus1", "layer": "layer2c", "placement": "Retained", "note": "The determines-done authority. It stays with the deterministic evaluator and the escalation policy. Frontier model capability is ceded and rented on demand, but the decision to escalate and the accept/reject verdict are never the model’s." } ], "seam_map": null, "writeup": [ "The experiment ran two phases on the same DGX Spark: a subjective RSS-triage phase that exposed failure patterns, and a coding phase with deterministic evaluation. The coding phase carries the findings, because the test harness gives an unfalsifiable pass/fail signal that a subjective gold key cannot. Bug-fix tasks were mined from real open-source repositories by git archaeology: roll the source back to before a fix, and use the fix’s own tests as the gate.", "Repair loops are model-dependent, and that is the finding that reorders the rest. For a deterministic local model at temperature 0, the same input produces the same output, so error feedback changes the prompt text but not the answer. On near-miss tasks where six of seven tests passed, all four repair attempts produced byte-identical output: four times the latency, zero improvement. For a non-deterministic reasoning model the story inverts. The evaluator’s test output steers the model toward a genuinely different attempt, and a second try can recover a near-miss the first pass failed. (The specific recovery figure first reported here is among the escalation statistics the validator audit supersedes; see the methodology note.) The variable that decides whether a repair loop is worth running is not the quality of the feedback. It is whether the model can produce a different output given the same feedback.", "Reviewing everything makes it worse. Every configuration that applied a review pass to all outputs degraded accuracy, whether the reviewer was the local model or a frontier governor. The failure was consistent: asked to review a decision, the model promotes correct low-confidence calls to confident wrong ones, because it asks what if this matters more often than what if it does not. A frontier reviewer shown the local model’s answer also anchored to it, confirming a call it would have made differently on a fresh look. Review earns its place only when a deterministic trigger says an output is worth challenging.", "The escalation number did not survive the audit, and that is the correction this lab is issuing. As first reported, tiered escalation looked like the architecture that paid: local-only cleared 25%, local plus a repair loop cleared the same 25%, and three tiers gated by the evaluator cleared 75%. Then the harness itself was checked. Nine of its validators were invalid, empty or self-contradictory gates that no output could pass, and they had booked escalation events for tasks that were never solved. Corrected, most of the apparent escalation gain dissolves. The tasks the small model could do, it did on its own; the rest were unsolvable on the instrument, not rescued by a bigger model. Lab 9 re-ran the same validator at rigor and put the number where it belongs: a lower-class model clears the solvable work, and the capable tier buys throughput, not correct answers.", "The cost model is the appeal. Each tier starts free on the local box and pays for a frontier call only when the evaluator rejects, so cost scales with the local model’s failure rate: about three cents a task at a 75% local pass rate, sixteen cents at the 25% this task set produced, and never more than frontier-only. The whole five-day run cost about $4.60 across 156 API calls. Two hardware notes carried the local tier: a mixture-of-experts model in FP8 ran at 48 tokens a second on the GB10 against 1 for a dense model, which is what makes a free local first attempt practical; and published token rates overstated the real frontier bill by four to five times, so paper cost projections are not the real economics.", "Two things set the ceiling, and neither is the loop. Model capability came first: a mixture-of-experts model passed tasks that a larger dense model and a coding-specialized model failed across every mode, so architecture amplifies capability but cannot substitute for it. The evaluator came second: the test harness is the system’s judgment, and a task with a bad test was unsolvable by every tier regardless of capability. The lever is not prompt engineering. It is the quality of the deterministic check.", "So the control point in an agentic loop is not the loop. It is the deterministic code around it: the evaluator that determines done and the policy that decides when to escalate. That is where the reasoning-plane authority sits, and it is a thing you own rather than an autonomy you grant the model. Deterministic Code In The Loop is not a constraint on the agent. It is the part of the system you can actually trust." ], "abstraction": [], "method": [ "Methodology Note: I identified a logic error in nine validator checks that incorrectly recorded escalation events. After correcting the validators, those cases are classified as persistent failures rather than escalation. This changes the reported escalation statistics but does not change the engineering conclusion of the lab: these cases remain unsolved under the tested loop-control strategy.", "Self-funded, no sponsor, run over five days on a single NVIDIA DGX Spark (GB10) with real open-source bugs and about $4.60 of frontier API spend. Local models ran on Ollama and vLLM; the frontier tiers were o3 and gpt-5.5.", "Tasks were mined from real repositories (more-itertools, httpx, PyJWT, h2, and others) by git archaeology: roll the source back to before a fix, and use the fix’s own test suite as an unfalsifiable pass/fail gate. Each tier ran from a clean workspace with no prior-tier context, and the deterministic harness was the sole authority on accept, reject, and escalation.", "The experiment code, the mode definitions, the per-run traces, and the failure-mode catalog are captured in the raw lab detail. The specific task set and the tuning particulars stay in the working notes." ], "assessments": [ { "vendor": "NVIDIA AI Platform", "layer": "Layer 0 · Compute", "grade": "NVIDIA Strength — Silicon Authority", "assessed_date": "May 22, 2026", "version": "v1.3 - Prose-Coherence (GA-Gate)", "source_url": "https://layer2c.com/assessment/nvidia", "snapshot_date": "2026-07-07" }, { "vendor": "NVIDIA AI Platform", "layer": "Layer 2C · Reasoning", "grade": "Runtime Governance Only — Not a Reasoning Plane", "assessed_date": "May 22, 2026", "version": "v1.3 - Prose-Coherence (GA-Gate)", "source_url": "https://layer2c.com/assessment/nvidia", "snapshot_date": "2026-07-07" } ], "publisher": "The CTO Advisor LLC", "program": "Layer2C Labs", "url": "https://labs.layer2c.com/labs/loop-control", "markdown_url": "https://labs.layer2c.com/labs/loop-control.md", "json_url": "https://labs.layer2c.com/labs/loop-control.json" }