The floor is trained, not sized

Measured on localized repair with an executable test, one 22-task pool, one box, one week of open-weight checkpoints. The rulings are about which self-hosted model clears an escalation bar, and what the apparatus costs in wall clock rather than dollars. The open edges are repository-scale debugging without localization, domains without an executable evaluator, and any model too large for a single 128GB unified-memory box.

Self-funded editorial. No vendor paid for this answer, reviewed it, or saw it before publication. The models under test are Alibaba's Qwen, Mistral's Devstral, Zhipu's GLM, and Google's Gemma, all run on owned NVIDIA hardware. Disclosure: Google Cloud is a client of The CTO Advisor LLC. Alibaba, Mistral, Zhipu, NVIDIA, and OpenAI are not. The ruling is the author's alone.

Lab 012 ended on a bracket instead of an answer. It showed Gemma 4 31B failing the instrument-literacy bar and a hosted mini clearing it, which proves the bar exists without saying where it sits. For anyone who already owns hardware, that gap is the entire decision. So I ran the descent: the same 22 certified bug-fix repairs, the same deterministic test gate, five open-weight models served on one NVIDIA DGX Spark through vLLM, every arm inside the same headless Claude Code harness. Which of them clears the bar Gemma missed? And what does self-hosting actually cost once the dollars read zero?

The floor is a coder model, and it sits lower than the bracket implied. Qwen3-Coder-Next, a 30B-class checkpoint in NVFP4, cleared 18 of 22 gate-verified repairs on that single box. Devstral Small 2 at 24B and a Qwen general build cleared 15 each. Gemma 4 26B cleared 14, GLM-4.7-Flash cleared 13. Against the 11 of 22 that Gemma 4 31B posted in the same harness, four of the five beat the bar. The model that beat it hardest wasn't the biggest one that fits. It was the one trained for code and tools. That's the shopping rule this bench supports: pick for training, not for parameter count.

Name the apparatus before you rank anything

Then the result that should change how anyone reads a model table. Gemma 4 31B scores 17 of 22 in the constrained edit loop, an apparatus where the model gets one shot to reproduce an anchor from memory and never touches a tool. GLM-4.7-Flash scores 4 in that same loop. Swap both into the agentic harness and the ordering reverses: Gemma falls to 11, Flash rises to 13. Neither model is better. Gemma has byte-exact recall and can't drive tools. Flash drives tools and can't reproduce anchors. Any leaderboard that ranks one above the other is right in one apparatus and wrong in the other.

The obvious objection is that this is one box and one task pool, so the numbers won't transfer. Partly true, and the scope says so. What transfers isn't the count. It's the inversion, because that result doesn't depend on the box at all. Capability in an agentic stack is a pair, not a number: a model carries a separate literacy in each apparatus you might run it in, and those literacies are separably trained. An architect picking a self-hosted model off a benchmark table is exposed to that inversion whatever hardware they own.

The bill comes in hours

So what does self-hosting cost? Not money. Every local arm ran at zero metered spend, which is the whole appeal. The bill lands in the column nobody prices. The best local arm spent 11.4 hours of GPU time to finish 18 repairs, at a 22-minute median and a 108-minute worst case. The hosted mini finished all 22 in 40 minutes for $3.14. That's roughly 18x the wall clock to land four repairs short. The spread holds across the whole field too: local medians ran 22 to 62 minutes per task where every hosted arm in this campaign ran 1 to 2 on identical work, a 30x gap measured across six models and two vendors.

Idle hardware makes the dollars free. It doesn't make the hours free. If the work is unattended and runs overnight, hours are cheap and self-hosting wins outright. If somebody is waiting on the result, $3.14 buys back most of a working day. The whole campaign cost roughly 107 hours of owned GPU time and a power bill nobody metered. And I haven't decomposed where the gap comes from: serving throughput, tool-call round trips, and retry behavior are all candidates with different fixes. The gap is real. Its cause is unattributed.

The disqualification that wasn't

One model looked disqualified and wasn't, and it's the most instructive row on the bench. GLM-4.5-Air, 106B parameters in four-bit Activation-aware Weight Quantization (AWQ), was staged and appeared not to fit. An agentic harness needs 131K tokens of serving context, the key-value cache for that context lives in the same unified memory as the weights, and at the utilization first tried there was no room left for sessions. That reading survived until the serving config was corrected. At a utilization setting of 0.80 the same weights on the same box yield 36.56 GiB of cache, 1.59 concurrent full-context sessions, and 12 of 22 solved. The box was never the constraint. A default max-num-seqs setting was, the same default that cost another model three times its cache. I wrote the disqualification into the published lab before testing it against a corrected config, and had to fix the entry hours after it shipped. Venue qualification is still the right discipline. It's only as trustworthy as the serving config it's computed against, and the cache figure should be read off the server, not derived. Three other derived figures died in this campaign the same way: a cache table the server's own reported number contradicted, a token total that missed the cache traffic, and a quantization recommendation borrowed from a benchmark whose own authors warned it wouldn't generalize. Every one was caught by reading an instrument instead of trusting arithmetic, which is the same argument the gate makes about models.

Air also lands where the headline predicts. At 106B it's three times the size of the model that beats it, and it finishes sixth of seven arms on this pool. What it owns is speed: a 9.2-minute median against Qwen3-Coder's 22, and 4.5 hours for the full pool against 11.4, because 12B active parameters decode fast. Fast, and mediocre. Size bought throughput and no capability.

Coin flips at the sharp end

The last finding is about the bench, and it cost me a claim I'd already published. The bar was written partly as clearing the residual set, the five hardest repairs in the pool. I reported that GLM-4.7-Flash owned that sharp end at 4 of 5, better than Qwen's 3, and built a distinction on top of it. Two more runs of Flash returned 1 and 1. The 4 was its best draw, and I'd read a draw as a property. Across five observations of the residual set the counts came back 3, 3, 4, 1 and 1. One task solved every time. One solved once. The middle three sat near a coin flip, and Qwen scored 3 of 5 twice on almost disjoint task sets. Aggregate counts replicate. Individual task verdicts don't, so any claim shaped like "this model solved that task" is reporting a coin flip somebody happened to watch. Running each of those tasks enough times to estimate a solve probability would give the bar a denominator. The tooling exists in the harness. It has never had the replicates to feed it.

Scope this before it travels. One box, one pool, one week of checkpoints, localized repair with an executable test. Every task arrived with its files named, so Qwen3-Coder's lead may be a code-reading advantage that repository-scale search would widen, or an artifact of already knowing where to look. The bar itself has a disclosed wobble: Gemma 4 31B's 11 of 22 was measured on a rented L40S venue, not the Spark, and that venue later measured slower than the owned box. The bar may sit slightly low, and the direction of that error flatters my headline, which is why I'm stating it instead of quietly adjusting it.

No self-hosted model reached 22 of 22. The hosted tier saturates this pool, where the mini and two Opus generations all clear it clean, which is why a planned frontier probe never ran: paying to watch another model clear a saturated pool teaches nothing. The four repairs between the best local arm and that sweep are uncharacterized, and the residual finding says this pool currently can't tell hard from unlucky. So the verdict stands where the evidence puts it. The self-hosted floor is real, it's trained rather than sized, and it's priced in hours. Whether those four missing repairs are worth $3.14 to you depends on whether your work can wait. Whether they're hard at all is the question this lab leaves standing.

The numbers

Where each layer belongs

Assessments at the time of the lab

Method and disclosure

Self-funded editorial. The harness is loopcontrolbench, published open under the MIT license at github.com/kltownsend/loopcontrolbench: the reproduce-or-drop gate, the sandboxed runner, and the arm definitions. This lab added a second agentic harness behind an environment switch, so the driver process can change while the prompt, workspace, gate and result schema stay identical, plus a trajectory reader that reports what a session actually did rather than only whether it passed. The vendor assessments and the layer placements this evidence feeds stay proprietary.

Every local arm ran on one NVIDIA DGX Spark with 128GB of unified memory, served through vLLM. Serving configuration is recorded with each result: the checkpoint path, the utilization setting, and the key-value cache the server reported at startup. Earlier arms in this campaign recorded only a proxy alias, which is why one arm in the record cannot name the weights that produced it. Closing that gap is what the provenance capture was added for.

The essay on this page was rendered from the lab’s frozen structured record by a model writing under this practice’s voice specification, with a deterministic gate checking every figure against the record before publication. The record is the canonical surface: where prose and record disagree, the record rules.