Recovered capacity is real, and it fails honest
Enterprises commonly carry idle owned hardware or unused cloud-spend commitments above their operating baseline: headroom producing nothing between peaks. This lab asked one question of that idle capacity: can a smaller model clear real, verifiable work on it, judged by the same deterministic validator that judged the big model? Across three models, four quantization tiers, and two substrates, the answer came back yes, with the load-bearing detail attached: in 585 scored attempts, the audit found no validator pass that failed the available held-out checks. The work that cleared was verified as far as the instrument can see. The work that failed went to a human queue. No token meter ran.
By Keith Townsend · 2026-07-16
The verdict
Scoped to falsifiable batch work: tasks with a deterministic validator, here real-library bug fixes judged by executable tests. The safety claim extends exactly as far as the validator’s detection power; deterministic does not mean complete, and a deterministic gate can reproducibly admit a defect its tests do not detect. The ruling answers the one question; the boundary numbers are this environment’s coordinates, not universal constants. The size and shape of the models and compute another team needs is their sizing exercise, and this page ships the instrument and method to run it. Two adjacent cells were deliberately not run and are named in the bound: the 26B MoE at Q4 and the E2B size rung. The ruling does not need them, and asserting cells you chose not to measure is how labs drift into marketing.
Disclosure: Intel is a client of this practice, and the lab corresponded with Intel about serving configurations during the program; the exchange is paraphrased where it appears and no vendor previewed or funded any of it. The bench ran on $28.32 of self-funded cloud spend plus owned hardware. No vendor paid for this answer.
Video
How I know
The lab collapsed to a single measured question on purpose. Labs 3 and 7 already priced the throughput of commodity Xeon and the behavior of the Loop Control architecture; what nobody had measured was whether a smaller model holds the success rate on the same problems, judged by the same pinned validator. Everything else is arithmetic. The bench filled exactly that cell, then kept filling cells the answers opened.
The controlled ladder ran one variable at a time. Run A changed only the model (12B dense versus the 26B reference, same Spark silicon): quality held at 20 of 39 tasks, and on the honest cross-run metric the 24-task overlap with the Lab 3 baseline reads 13 versus 13 unanimous-pass, dead even; the any-pass gap belongs to the metric caveat below. Run B changed only the substrate (the same 12B moved to a rented Granite Rapids shape): quality survived, gaining one task and losing none. The 26B on the same rented shape tied the 12B at 21 while finishing the identical protocol in 2.24 hours against 5.54. That is a configuration comparison, not a single-variable result: architecture, precision, and runtime moved together, with sparse activation the leading explanation for the gap.
The descent found the floor and bracketed it from both sides. Q8_0 and Q4_K_M held the exact pass set of the bf16 original, tier after tier; Q3_K_M broke by five tasks net, triple-confirmed; and the E4B size probe broke below the boundary from the other axis at 15 of 39. The two breaks have different textures worth knowing: precision broke toward failure-to-fix (more honest fails, little collateral), while size broke destructively (seven regression verdicts, five of them real broken-worse-than-found edits). Same safety property either way: at every tier, including both broken ones, the false-pass audit came back zero. Quality degrades closed on this instrument, all the way down — but what lands in the human queue differs in kind, and a team sizing its own boundary should look at both axes.
The deterministic component of this architecture is the validator, not the model, and the data kept illustrating why that placement is the thesis. The Lab 3 baseline flickered verdicts at temperature zero; the 12B ran verdict-identical for 117 straight traces; different silicon produced different tokens from the same weights. None of it mattered to the output, because reaching the output requires clearing a gate that behaves the same way every time. That is Deterministic Code In The Loop, priced: the worker is fungible, the gate is the asset.
What the bench measured
The detail
The instrument is the credibility, so it ships first. The task set is not synthetic toys: most of the 39 tasks are reconstructed from real bug fixes in real Python libraries, each pinned to a repo and commit, presenting the codebase as it stood with the historical bug present. Detection is layered, and each layer catches a different failure: feature tests verify the fix works, regression tests catch collateral damage, hidden reference tests the model never sees catch gaming, and a static gate kills malformed output before any code runs. The validator was pinned bit-identical across every run by manifest hash and by functional census, and the audit that verified it also X-rayed it: one weak test, two tasks without hidden references, six structurally unpassable tasks, all found by running the same audit against the original Lab 3 baseline, all carried like-for-like on both sides of every comparison. A lab that will not X-ray its own instrument has no business grading anyone else’s.
The design was a controlled 2x2 against the Lab 3 reference, one variable per step. Change only the model: the 12B dense held quality on the Spark. Change only the substrate: the same model held quality on rented Granite Rapids, and at temperature zero the different silicon produced different tokens with the same task-level verdicts, which is worth a sentence of its own: determinism is a property of a model-substrate pair, not of a model. Then the additions the data demanded: the 26B mixture-of-experts on the same rented shape tied the 12B’s yield while finishing in less than half the wall time, and the quantization ladder descended until it found the floor. Every cell was scored by the same pinned validator with the same false-pass audit. The one cell the ruling names that began as a composite, the 26B at Q4, was queued for measurement rather than asserted, because the difference between those two words is this program’s entire brand.
The false-pass column is the finding under the finding. A batch system that ships defects quietly is worse than no system, and the audit existed to catch exactly that: every validator pass, re-run against held-out tests. Zero confirmed false passes, everywhere, including at the tier where quality broke. When the Q3 model got worse, it got worse honestly: more failures routed to the queue, verdicts churning in both directions, nothing slipping through as false success. The architecture’s promise, that the failure mode is a person looks at it, held at every point the lab could measure, and the hidden-reference layer is what made that promise falsifiable rather than rhetorical.
The drift this lab caught in its own analysis is worth publishing because everyone will make it. Midway through, the analysis began treating model determinism as a virtue: celebrating zero-flip runs, framing the baseline’s verdict flicker as contamination. That inverts the thesis. In Deterministic Code In The Loop, the deterministic component is the validator; the baseline’s flicker is an argument for the architecture, because a variance-prone worker behind a deterministic gate is safe by construction, and on idle cycles its retries become verified mining. The corrected metric hierarchy ships on this page: per-attempt verified yield is the economic statistic, determinism is an operational property that sets retry policy, and comparing best-of-n numbers against single-run numbers across different variance profiles is a quiet corruption that likely infects more published model comparisons than anyone has checked.
The economics land where the thesis pointed. On the rented shape, a verified bug fix cost about nine cents of compute at on-demand rates and six at the measured Savings Plan rate. On owned idle hardware, the marginal cost approaches incremental power, cooling, and operations; inside an otherwise-unused Savings Plan commitment, the incremental bill can approach zero, subject to commitment coverage. Against that denominator, a 54% yield is not a grade on the model. It is recovered value on spend that produced nothing last quarter, with the remainder routed to humans who were always the fallback, and zero tokens metered along the way. The boundary and workhorse coordinates above are this environment’s answer. The instrument, the descent method, and the audit are the transferable part, and they are exactly what a team runs to find their own.
A 54% pass rate is failure half the time. Why not use a frontier model or a person?
Because the denominator is free and the failures are honest. The 46% that fails costs idle cycles that were already bought and routes to the human who was the fallback before this system existed. The 54% that clears is verified work the human no longer does. The comparison is not this model versus a better worker; it is this yield versus the zero yield the same committed capacity produced last quarter.
The bigger worker does not change the answer. The 26B mixture-of-experts, the same class of model that anchored the original Lab 3 baseline, cleared the same 21 of 39 tasks on the identical substrate as the 12B, and the baseline itself sits within a few points on per-attempt yield. Within the overlapping task set and measured configurations, the larger worker bought no material verified-yield advantage; its measured advantage was throughput. Correctness lives in the gate.
And the frontier alternative changes the failure mode, which is the thing this architecture exists to control. A paid-token escalation path fails by accruing incremental charges. This loop fails by growing a queue; its infrastructure is metered too when rented, but no per-token charge accrues. For batch work on committed headroom, a queue is recoverable in a way an open-ended meter is not, and the zero confirmed-false-pass audit is what makes the queue trustworthy.
Where each layer belongs
| Layer | Placement |
|---|---|
Layer 0 · Compute Compute & Network Fabric The economic object is headroom you already carry, in two distinct forms the page keeps separate: owned idle hardware (the Spark) and unused cloud-spend commitment (the Savings Plan analog; as measured, no classic RI offerings existed for the shape — a Savings Plan commits dollars, not capacity). The lab prices that headroom, not a new purchase. Availability footnote: four AWS launches, zero quota friction, 16 to 29 seconds to SSH. | Retained / Delegated |
Layer 2B · Runtime Application Runtime & Execution The runtime is open source (vLLM, llama.cpp) and the control point is the validator and escalation policy, per the Loop Control ruling. The vendor-blessed OpenVINO backend was measured for scope: 4.7x on prefill, slower on bandwidth-bound decode, and unable to execute the mixture-of-experts graph at all. Stock stacks carried every measured cell. | Retained |
Layer 2C · Reasoning Agentic Infrastructure — The Reasoning Plane Open weights, small and quantized: the boundary worker is a 7.7GB artifact. Nothing in the loop depends on a garden, an API, or a token meter, and the measured cost of that independence on this workload was a few points of yield against the strongest baseline reading. | Retained |
The two questions this lab now knows to ask
The economic structure is silicon-independent by construction: committed headroom plus a deterministic validator plus a human backstop applies to TPU, APU, and GPU reservations equally. The yield is a property of the model-substrate pair and must be measured per lane; labs 7 and 8 already showed the same model behaving differently across lanes. Lab 10 runs this instrument on a second accelerated substrate, and that is where the generalization is earned or retired.
This lab ran one task domain with an executable validator, the domain where Deterministic Code In The Loop is strongest. Real batches are mixed, and validators weaker than pytest make the false-pass column harder to trust. Separately, the corrected framing implies an unmeasured strategy: a variance-prone worker retried on idle cycles mines additional verified passes. The yield curve of that mining, and where it beats simply running a bigger worker once, is an open measurement.
What it did not prove
- It did not prove the boundary generalizes. Q4 as the floor and Q3 as the break are this environment’s coordinates: this model family, this task set, this validator. The page ships the method to find yours, not the claim that yours matches.
- The headline boundary config on the rented substrate rests on two measured edges, not a measured cell: substrate survival was proven at bf16 and the Q4 floor was proven on the Spark. Two adjacent cells were deliberately not run: the 26B MoE at Q4 (the workhorse claim stands on its measured Q8 cell) and the E2B size rung (E4B already breaks below the boundary). The E4B break itself is a single-screen declaration: its confirmation runs were canceled in the same close-out, with the mitigating fact that every lab 9 tier granted multiple runs was verdict-deterministic. All of it stated, none of it asserted beyond its evidence.
- It did not test heterogeneous batches or non-executable validators. One domain, chosen because its validator is deterministic; the architecture’s scope limit is the validator’s scope limit, exactly as Lab 3 ruled.
- It did not measure the market. Whether reserved Xeon floors are forming at hyperscalers, and why, are hypotheses the lab’s own capacity census declined to confirm; they stay off this page.
- The instrument has known edges, found by this lab’s own audit and disclosed above; three tasks carry no usable held-out evidence and six are structurally unpassable on all sides. Repair is handed to lab 10.
Notes from the author, Keith Townsend
The lab kept trying to become a performance benchmark and the data kept dragging it back to the actual question. I watched a 12B model crawl at seven and a half tokens per second and nearly redesigned the lab around making it faster, until the obvious hit: for batch work on cycles I already own, slow is a scheduling problem, not a defect. The question was never speed. It was whether the work that comes out can be trusted, and the false-pass column answered that with a number I did not expect to be zero.
The drift I caught matters more to me than the boundary we found. Halfway through, we were praising models for being deterministic. That is backwards. DCITL puts the determinism in the code, in the validator, precisely so the model does not have to carry it. The baseline flickering verdicts at temperature zero is not a flaw in Lab 3; it is why Lab 3 exists. When your own analysis starts drifting toward the thing your framework was built to make unnecessary, that is worth writing down.
The boundary ruling ended up simpler than the lab that produced it: smaller model, highly quantized, judged by the same gate. A 7.7GB file doing verified maintenance work on hardware that was idling anyway. The sizing specifics are ours; the exercise is yours. That is the most honest sentence a lab like this can end on.
Method and disclosure
Editorial and self-funded: $28.32 of cloud spend, all instances terminated and verified, plus owned hardware (the DGX Spark) for the control runs and the entire quality ladder. Disclosure: Intel is a client of this practice; the lab corresponded with Intel about serving configurations during this program, the exchange is paraphrased where referenced, and no vendor commissioned, funded, or previewed anything. The vendor-blessed serving path was tested because Intel named it, on the record, and the result published regardless of direction.
The instrument is the Lab 3 Loop Control harness, unmodified: 39 tasks (most reconstructed from real bug fixes in httpx, requests, dateutil, more-itertools, arrow, click, and other real libraries, pinned to repo and commit), layered deterministic tests including hidden references, pinned bit-identical across every run by sorted-manifest hash and functional census. Task contents and solutions stay private; the method, source repos, audit design, and every rate, count, and cost ship in the raw detail, including the full chronology with its own mistakes: an instrument-location error in the runbook, a hash recipe that had to be re-anchored, two serving-behavior confounds caught at smoke time, and one cleanup error on a pre-existing cloud volume, disclosed to the account owner the hour it happened.
Download the raw lab detail (Markdown)