Lab 009 · Editorial lab

Recovered capacity is real, and it fails honest

Enterprises commonly carry idle owned hardware or unused cloud-spend commitments above their operating baseline: headroom producing nothing between peaks. This lab asked one question of that idle capacity: can a smaller model clear real, verifiable work on it, judged by the same deterministic validator that judged the big model? Across three models, four quantization tiers, and two substrates, the answer came back yes, with the load-bearing detail attached: in 585 scored attempts, the audit found no validator pass that failed the available held-out checks. The work that cleared was verified as far as the instrument can see. The work that failed went to a human queue. No token meter ran.

By Keith Townsend · 2026-07-16

The call

The verdict

Scoped to falsifiable batch work: tasks with a deterministic validator, here real-library bug fixes judged by executable tests. The safety claim extends exactly as far as the validator’s detection power; deterministic does not mean complete, and a deterministic gate can reproducibly admit a defect its tests do not detect. The ruling answers the one question; the boundary numbers are this environment’s coordinates, not universal constants. The size and shape of the models and compute another team needs is their sizing exercise, and this page ships the instrument and method to run it. Two adjacent cells were deliberately not run and are named in the bound: the 26B MoE at Q4 and the E2B size rung. The ruling does not need them, and asserting cells you chose not to measure is how labs drift into marketing.

DoPut committed idle capacity to work on falsifiable batches behind a deterministic validator. Measured yield: 51 to 54% of attempts cleared as verified passes among the configurations above the measured boundary (numerators on the card), with the Lab 3 baseline at 47.8% on its own 29-task coverage. On the rented Granite Rapids shape, the full 117-attempt protocol produced 63 verified bug fixes in 2.24 hours: about nine cents of on-demand compute per verified fix, six cents at the measured one-year Savings Plan rate, and an incremental bill that can approach zero inside an otherwise-unused spend commitment.
DoTrust the gate, not the model. Zero confirmed false passes in 585 scored attempts across every configuration measured. Failures routed to the human queue or died at the output gate; no confirmed false success cleared the measured gate, and that claim extends exactly as far as the validator and its held-out evidence detect. The failure mode of this architecture is a person looks at it, not a token meter running.
DoSize down with a controlled descent. In this environment the measured model boundary landed at a 7.7GB artifact: Gemma 4 12B at Q4_K_M held the identical pass set as its bf16 original across three quantization tiers, measured on the Spark control. Substrate survival was proven at bf16, so the boundary config on rented Xeon rests on two measured edges rather than a measured cell; that bound is stated, not hidden. One rung lower broke, cleanly and honestly. Your boundary will differ; the screen-then-confirm ladder that finds it takes an afternoon.
Don’tDo not require the model to be deterministic, and do not read variance as defect. The deterministic component of this architecture is the validator. A variance-prone worker behind a deterministic gate fails closed against the defects the gate is built to detect, and on idle cycles its retries become mining: every additional pass it flickers into is checked against the held-out evidence before it counts.

Disclosure: Intel is a client of this practice, and the lab corresponded with Intel about serving configurations during the program; the exchange is paraphrased where it appears and no vendor previewed or funded any of it. The bench ran on $28.32 of self-funded cloud spend plus owned hardware. No vendor paid for this answer.

The walkthrough

Video

Video slot — sponsored labs fill this with a series.
The bench

How I know

The lab collapsed to a single measured question on purpose. Labs 3 and 7 already priced the throughput of commodity Xeon and the behavior of the Loop Control architecture; what nobody had measured was whether a smaller model holds the success rate on the same problems, judged by the same pinned validator. Everything else is arithmetic. The bench filled exactly that cell, then kept filling cells the answers opened.

The controlled ladder ran one variable at a time. Run A changed only the model (12B dense versus the 26B reference, same Spark silicon): quality held at 20 of 39 tasks, and on the honest cross-run metric the 24-task overlap with the Lab 3 baseline reads 13 versus 13 unanimous-pass, dead even; the any-pass gap belongs to the metric caveat below. Run B changed only the substrate (the same 12B moved to a rented Granite Rapids shape): quality survived, gaining one task and losing none. The 26B on the same rented shape tied the 12B at 21 while finishing the identical protocol in 2.24 hours against 5.54. That is a configuration comparison, not a single-variable result: architecture, precision, and runtime moved together, with sparse activation the leading explanation for the gap.

The descent found the floor and bracketed it from both sides. Q8_0 and Q4_K_M held the exact pass set of the bf16 original, tier after tier; Q3_K_M broke by five tasks net, triple-confirmed; and the E4B size probe broke below the boundary from the other axis at 15 of 39. The two breaks have different textures worth knowing: precision broke toward failure-to-fix (more honest fails, little collateral), while size broke destructively (seven regression verdicts, five of them real broken-worse-than-found edits). Same safety property either way: at every tier, including both broken ones, the false-pass audit came back zero. Quality degrades closed on this instrument, all the way down — but what lands in the human queue differs in kind, and a team sizing its own boundary should look at both axes.

The deterministic component of this architecture is the validator, not the model, and the data kept illustrating why that placement is the thesis. The Lab 3 baseline flickered verdicts at temperature zero; the 12B ran verdict-identical for 117 straight traces; different silicon produced different tokens from the same weights. None of it mattered to the output, because reaching the output requires clearing a gate that behaves the same way every time. That is Deterministic Code In The Loop, priced: the worker is fungible, the gate is the asset.

What the bench measured

Confirmed false passes
Across 585 scored attempts: three models (12B dense, 26B MoE, the E4B size probe), four quant tiers, two substrates. Every validator pass re-checked against hidden reference tests the model never sees. Fail-closed held everywhere, including at the broken tier
0
Per-attempt verified yield
The primary economic statistic, numerators visible: 12B on Spark 60/117 (51.3%), 12B on Xeon 62/117 (53.0%), 26B-Q8 63/117 (53.8%); Lab 3 baseline 43/90 (47.8%) on its 29-task coverage (57.3% restricted to the 24-task overlap, where the 12B reads 39/72 = 54.2%). Attempts are the unit of cost, verified passes the unit of yield
51-54%
Cost per verified bug fix
63 verified fixes in 2.24h: nine cents at on-demand, six at the measured 1yr Savings Plan rate. On owned idle hardware, marginal cost approaches incremental power and operations; inside an otherwise-unused Savings Plan commitment, the incremental bill can approach zero. Zero paid tokens in the loop, by construction
$0.09 / $0.06
The boundary (this environment)
Identical pass set to bf16 across three quant tiers. Bracketed from both sides: Q3 breaks below it on precision (net -5, triple-confirmed) and E4B breaks below it on size (15/39). Both breaks fail-closed. The descent method ships; your boundary is an afternoon away
12B at Q4_K_M, 7.7GB
The workhorse
Same 21/39 yield as the 12B on identical silicon, 2.24h vs 5.54h for the identical protocol. A configuration result: architecture, precision, and runtime changed together; sparse activation is the leading explanation
26B MoE, 2.5x faster
Quality across the 2x2
Model size (Run A, Spark control): 20/39 vs baseline. Substrate (Run B, Granite Rapids): 21/39, gained one, lost none. One variable moved per run; the validator pinned bit-identical throughout
held, both axes
Baseline comparison, both metrics
The 26B-on-Xeon versus the Lab 3 baseline, 29-task overlap: any-pass reads 17 vs 12, a scary 5-task deficit; unanimous-pass reads 13 vs 12, one task. Any-pass rewards the baseline’s own verdict flicker (4 flip tasks) against a zero-flip run. Never compare best-of-n to single-run across different variance profiles
17 vs 12 becomes 13 vs 12
Vendor-blessed config
llama.cpp-OpenVINO, Intel’s named path: real prefill gains on dense, 16% slower on bandwidth-bound decode, and a repeatable assert running the MoE. Stock llama.cpp remains the only stack that runs the MoE and engages AMX (hardware counters)
4.7x prefill, MoE: crash
Reserved economics footnote
As measured 2026-07-15 (EC2 API, us-east-1), zero classic Reserved Instance offerings for m8i.12xlarge; the reserved analog is a Savings Plan ($1.68/hr 1yr no-upfront vs $2.54 on-demand). A Savings Plan commits dollars, not capacity — the math prices unused spend commitment and says so
Savings Plans, not RIs
Lab spend
All cloud legs, instances terminated and verified. The Spark legs and the entire quality ladder ran on owned hardware at zero marginal cost, which is the thesis performing itself
$28.32

The detail

The instrument is the credibility, so it ships first. The task set is not synthetic toys: most of the 39 tasks are reconstructed from real bug fixes in real Python libraries, each pinned to a repo and commit, presenting the codebase as it stood with the historical bug present. Detection is layered, and each layer catches a different failure: feature tests verify the fix works, regression tests catch collateral damage, hidden reference tests the model never sees catch gaming, and a static gate kills malformed output before any code runs. The validator was pinned bit-identical across every run by manifest hash and by functional census, and the audit that verified it also X-rayed it: one weak test, two tasks without hidden references, six structurally unpassable tasks, all found by running the same audit against the original Lab 3 baseline, all carried like-for-like on both sides of every comparison. A lab that will not X-ray its own instrument has no business grading anyone else’s.

The design was a controlled 2x2 against the Lab 3 reference, one variable per step. Change only the model: the 12B dense held quality on the Spark. Change only the substrate: the same model held quality on rented Granite Rapids, and at temperature zero the different silicon produced different tokens with the same task-level verdicts, which is worth a sentence of its own: determinism is a property of a model-substrate pair, not of a model. Then the additions the data demanded: the 26B mixture-of-experts on the same rented shape tied the 12B’s yield while finishing in less than half the wall time, and the quantization ladder descended until it found the floor. Every cell was scored by the same pinned validator with the same false-pass audit. The one cell the ruling names that began as a composite, the 26B at Q4, was queued for measurement rather than asserted, because the difference between those two words is this program’s entire brand.

The false-pass column is the finding under the finding. A batch system that ships defects quietly is worse than no system, and the audit existed to catch exactly that: every validator pass, re-run against held-out tests. Zero confirmed false passes, everywhere, including at the tier where quality broke. When the Q3 model got worse, it got worse honestly: more failures routed to the queue, verdicts churning in both directions, nothing slipping through as false success. The architecture’s promise, that the failure mode is a person looks at it, held at every point the lab could measure, and the hidden-reference layer is what made that promise falsifiable rather than rhetorical.

The drift this lab caught in its own analysis is worth publishing because everyone will make it. Midway through, the analysis began treating model determinism as a virtue: celebrating zero-flip runs, framing the baseline’s verdict flicker as contamination. That inverts the thesis. In Deterministic Code In The Loop, the deterministic component is the validator; the baseline’s flicker is an argument for the architecture, because a variance-prone worker behind a deterministic gate is safe by construction, and on idle cycles its retries become verified mining. The corrected metric hierarchy ships on this page: per-attempt verified yield is the economic statistic, determinism is an operational property that sets retry policy, and comparing best-of-n numbers against single-run numbers across different variance profiles is a quiet corruption that likely infects more published model comparisons than anyone has checked.

The economics land where the thesis pointed. On the rented shape, a verified bug fix cost about nine cents of compute at on-demand rates and six at the measured Savings Plan rate. On owned idle hardware, the marginal cost approaches incremental power, cooling, and operations; inside an otherwise-unused Savings Plan commitment, the incremental bill can approach zero, subject to commitment coverage. Against that denominator, a 54% yield is not a grade on the model. It is recovered value on spend that produced nothing last quarter, with the remainder routed to humans who were always the fallback, and zero tokens metered along the way. The boundary and workhorse coordinates above are this environment’s answer. The instrument, the descent method, and the audit are the transferable part, and they are exactly what a team runs to find their own.

The obvious objection

A 54% pass rate is failure half the time. Why not use a frontier model or a person?

Because the denominator is free and the failures are honest. The 46% that fails costs idle cycles that were already bought and routes to the human who was the fallback before this system existed. The 54% that clears is verified work the human no longer does. The comparison is not this model versus a better worker; it is this yield versus the zero yield the same committed capacity produced last quarter.

The bigger worker does not change the answer. The 26B mixture-of-experts, the same class of model that anchored the original Lab 3 baseline, cleared the same 21 of 39 tasks on the identical substrate as the 12B, and the baseline itself sits within a few points on per-attempt yield. Within the overlapping task set and measured configurations, the larger worker bought no material verified-yield advantage; its measured advantage was throughput. Correctness lives in the gate.

And the frontier alternative changes the failure mode, which is the thing this architecture exists to control. A paid-token escalation path fails by accruing incremental charges. This loop fails by growing a queue; its infrastructure is metered too when rented, but no per-token charge accrues. For batch work on committed headroom, a queue is recoverable in a way an open-ended meter is not, and the zero confirmed-false-pass audit is what makes the queue trustworthy.

Where it belongs

Where each layer belongs

LayerPlacement
Layer 0 · Compute
Compute & Network Fabric
The economic object is headroom you already carry, in two distinct forms the page keeps separate: owned idle hardware (the Spark) and unused cloud-spend commitment (the Savings Plan analog; as measured, no classic RI offerings existed for the shape — a Savings Plan commits dollars, not capacity). The lab prices that headroom, not a new purchase. Availability footnote: four AWS launches, zero quota friction, 16 to 29 seconds to SSH.
Retained / Delegated
Layer 2B · Runtime
Application Runtime & Execution
The runtime is open source (vLLM, llama.cpp) and the control point is the validator and escalation policy, per the Loop Control ruling. The vendor-blessed OpenVINO backend was measured for scope: 4.7x on prefill, slower on bandwidth-bound decode, and unable to execute the mixture-of-experts graph at all. Stock stacks carried every measured cell.
Retained
Layer 2C · Reasoning
Agentic Infrastructure — The Reasoning Plane
Open weights, small and quantized: the boundary worker is a 7.7GB artifact. Nothing in the loop depends on a garden, an API, or a token meter, and the measured cost of that independence on this workload was a few points of yield against the strongest baseline reading.
Retained
What it opened

The two questions this lab now knows to ask

Does the yield survive a second substrate class?

The economic structure is silicon-independent by construction: committed headroom plus a deterministic validator plus a human backstop applies to TPU, APU, and GPU reservations equally. The yield is a property of the model-substrate pair and must be measured per lane; labs 7 and 8 already showed the same model behaving differently across lanes. Lab 10 runs this instrument on a second accelerated substrate, and that is where the generalization is earned or retired.

Does the profile hold on heterogeneous work, and what does variance mining actually yield?

This lab ran one task domain with an executable validator, the domain where Deterministic Code In The Loop is strongest. Real batches are mixed, and validators weaker than pytest make the false-pass column harder to trust. Separately, the corrected framing implies an unmeasured strategy: a variance-prone worker retried on idle cycles mines additional verified passes. The yield curve of that mining, and where it beats simply running a bigger worker once, is an open measurement.

The bound

What it did not prove

  • It did not prove the boundary generalizes. Q4 as the floor and Q3 as the break are this environment’s coordinates: this model family, this task set, this validator. The page ships the method to find yours, not the claim that yours matches.
  • The headline boundary config on the rented substrate rests on two measured edges, not a measured cell: substrate survival was proven at bf16 and the Q4 floor was proven on the Spark. Two adjacent cells were deliberately not run: the 26B MoE at Q4 (the workhorse claim stands on its measured Q8 cell) and the E2B size rung (E4B already breaks below the boundary). The E4B break itself is a single-screen declaration: its confirmation runs were canceled in the same close-out, with the mitigating fact that every lab 9 tier granted multiple runs was verdict-deterministic. All of it stated, none of it asserted beyond its evidence.
  • It did not test heterogeneous batches or non-executable validators. One domain, chosen because its validator is deterministic; the architecture’s scope limit is the validator’s scope limit, exactly as Lab 3 ruled.
  • It did not measure the market. Whether reserved Xeon floors are forming at hyperscalers, and why, are hypotheses the lab’s own capacity census declined to confirm; they stay off this page.
  • The instrument has known edges, found by this lab’s own audit and disclosed above; three tasks carry no usable held-out evidence and six are structurally unpassable on all sides. Repair is handed to lab 10.
In the author’s words

Notes from the author, Keith Townsend

The lab kept trying to become a performance benchmark and the data kept dragging it back to the actual question. I watched a 12B model crawl at seven and a half tokens per second and nearly redesigned the lab around making it faster, until the obvious hit: for batch work on cycles I already own, slow is a scheduling problem, not a defect. The question was never speed. It was whether the work that comes out can be trusted, and the false-pass column answered that with a number I did not expect to be zero.

The drift I caught matters more to me than the boundary we found. Halfway through, we were praising models for being deterministic. That is backwards. DCITL puts the determinism in the code, in the validator, precisely so the model does not have to carry it. The baseline flickering verdicts at temperature zero is not a flaw in Lab 3; it is why Lab 3 exists. When your own analysis starts drifting toward the thing your framework was built to make unnecessary, that is worth writing down.

The boundary ruling ended up simpler than the lab that produced it: smaller model, highly quantized, judged by the same gate. A 7.7GB file doing verified maintenance work on hardware that was idling anyway. The sizing specifics are ours; the exercise is yours. That is the most honest sentence a lab like this can end on.

How it was built

Method and disclosure

Editorial and self-funded: $28.32 of cloud spend, all instances terminated and verified, plus owned hardware (the DGX Spark) for the control runs and the entire quality ladder. Disclosure: Intel is a client of this practice; the lab corresponded with Intel about serving configurations during this program, the exchange is paraphrased where referenced, and no vendor commissioned, funded, or previewed anything. The vendor-blessed serving path was tested because Intel named it, on the record, and the result published regardless of direction.

The instrument is the Lab 3 Loop Control harness, unmodified: 39 tasks (most reconstructed from real bug fixes in httpx, requests, dateutil, more-itertools, arrow, click, and other real libraries, pinned to repo and commit), layered deterministic tests including hidden references, pinned bit-identical across every run by sorted-manifest hash and functional census. Task contents and solutions stay private; the method, source repos, audit design, and every rate, count, and cost ship in the raw detail, including the full chronology with its own mistakes: an instrument-location error in the runbook, a hash recipe that had to be re-anchored, two serving-behavior confounds caught at smoke time, and one cleanup error on a pre-existing cloud volume, disclosed to the account owner the hour it happened.

Download the raw lab detail (Markdown)