{ "slug": "self-host-floor", "lab_number": 13, "title": "The floor is trained, not sized", "type": "editorial", "status": "published", "sponsor": null, "author": "Keith Townsend", "date": "August 5, 2026", "date_iso": "2026-08-05", "layers": [ "layer0", "layer2b", "layer2c" ], "axes": [ { "instrument": "4plus1", "layers": [ "layer0", "layer2b", "layer2c" ], "has_dapm_table": true } ], "supersedes": [], "superseded_by": null, "corrects": [], "corrected_by": [], "vendors": [ { "key": "nvidia", "role": "hw" }, { "key": "alibaba", "role": "model" }, { "key": "mistral", "role": "model" }, { "key": "zhipu", "role": "model" }, { "key": "google", "role": "model" }, { "key": "openai", "role": "ref" } ], "themes": [ "small-model-viability", "model-ownership", "ai-factory-economics", "agentic-repair", "validator-authority", "recovered-capacity" ], "finding": "A 30B coder model on one box clears 18 of 22 gate-verified repairs. A 31B general model cleared 11. Seven parameters apart, same hardware, same gate, and what separates them is training for tools rather than size. The self-hosted floor is real and lower than expected. The bill just isn't in dollars: 18 of 22 took 11.4 hours of owned hardware against 40 minutes and $3.14 for all 22 on a hosted mini.", "question": "Lab 012 bracketed the instrument-literacy bar without locating it. Gemma 4 31B failed it, a hosted mini cleared it, and everything in between was guesswork. If you already own the hardware, that gap is the whole decision. So I ran the descent: the same 22 certified bug-fix repairs, the same deterministic test gate, five open-weight models on one NVIDIA DGX Spark, each inside the same headless Claude Code harness. Which of them clears the bar Gemma missed, and what does self-hosting actually cost once you stop counting dollars?", "load": "The 22 repairs Lab 011's free local loop first missed, each a real bug-fix pull request shipping its own test suite as the pass/fail gate. Six open-weight models served on one 128GB Spark through vLLM: Qwen3-Coder-Next in NVFP4, Devstral Small 2 24B, a Qwen general build, Gemma 4 26B in NVFP4, GLM-4.7-Flash in BF16, and GLM-4.5-Air in AWQ 4-bit. Eighteen arms across two apparatuses, three of them replicated, plus a five-task residual set run twice more to test whether the sharp end was measuring anything at all. Zero metered spend on the local arms. The cost is roughly 107 hours of owned GPU time and a power bill nobody metered.", "verdict": { "scope": "Measured on localized repair with an executable test, one 22-task pool, one box, one week of open-weight checkpoints. The rulings are about which self-hosted model clears an escalation bar, and what the apparatus costs in wall clock rather than dollars. The open edges are repository-scale debugging without localization, domains without an executable evaluator, and any model too large for a single 128GB unified-memory box.", "independence": "Self-funded editorial. No vendor paid for this answer, reviewed it, or saw it before publication. The models under test are Alibaba's Qwen, Mistral's Devstral, Zhipu's GLM, and Google's Gemma, all run on owned NVIDIA hardware. Disclosure: Google Cloud is a client of The CTO Advisor LLC. Alibaba, Mistral, Zhipu, NVIDIA, and OpenAI are not. The ruling is the author's alone.", "calls": [ { "kind": "do", "text": "pick the model trained for tool use, not the biggest one that fits. A 30B coder build cleared 18 of 22 where a 31B general build cleared 11, on the same box and the same gate." }, { "kind": "do", "text": "budget wall clock as the real cost of self-hosting. The best local arm ran 11.4 hours to finish 18 repairs. A hosted mini finished all 22 in 40 minutes for $3.14. That gap is what you're buying when you pay." }, { "kind": "do", "text": "compute venue qualification against a correct serving config, not a default one. GLM-4.5-Air looked disqualified on this box until the serving config was fixed, then it ran and cleared 12 of 22. A wrong utilization setting will retire a model that works." }, { "kind": "dont", "text": "read a leaderboard rank as instrument literacy. Gemma 4 31B scored 17 of 22 in the constrained loop and 11 in the harness. GLM-4.7-Flash scored 4 and 13. The ordering inverts with the apparatus, so any single ranking of those two is wrong in one of the two settings." }, { "kind": "dont", "text": "trust one run on the hard tasks. Across five observations of the residual set, one task solved every time, one solved once, and the middle three sat near a coin flip. The same model scored 3 of 5 twice on almost disjoint tasks." } ] }, "objection": { "q": "One box, one task pool. Isn't \"18 of 22\" just this Spark's answer to this bench, with no reason to think it transfers?", "a": [ "Partly, and scope says so. What transfers isn't the number, it's the inversion. Gemma and Flash swap places when the apparatus changes, 17 against 4 in the loop and 11 against 13 in the harness, and that result doesn't depend on the box at all. It says capability here isn't a single quantity, so \"which local model is better\" has no answer until you name the apparatus. Any architect picking a self-hosted model off a benchmark table is exposed to that inversion no matter what hardware they own.", "The wall-clock finding transfers too, and it's the one that moves budgets. Every local arm ran a median between 22 and 62 minutes per task. Every hosted arm in this campaign ran between 1 and 2 on the same work. That's a 30x spread measured across six models and two vendors, wide enough that the specific box stops mattering." ] }, "numbers": [ { "label": "Best self-hosted result, one Spark", "value": "18 of 22", "note": "Qwen3-Coder-Next NVFP4, 22-minute median per task, 11.4 hours total, zero metered spend" }, { "label": "The bar it cleared", "value": "11 of 22", "note": "Gemma 4 31B in the same harness, the model Lab 012 used to bracket the bar from below" }, { "label": "Apparatus inversion, same two models", "value": "17/4 then 11/13", "note": "Gemma and Flash in the constrained loop, then in the harness; the ordering reverses" }, { "label": "Wall clock, best local against hosted mini", "value": "11.4h vs 0.6h", "note": "for 18 of 22 against 22 of 22; the hosted arm cost $3.14 and the local arm cost hours" }, { "label": "Local medians per task, seven arms", "value": "9 to 62 min", "note": "every hosted arm in this campaign ran 1 to 2 minutes on the same tasks" }, { "label": "A 106B model on one box", "value": "12 of 22", "note": "GLM-4.5-Air AWQ 4-bit, sixth of seven, six repairs behind a 30B coder build a third its size" }, { "label": "Residual-set replicates", "value": "3, 3, 4, 1, 1", "note": "five observations of the same five tasks; counts move wider than the seed band and task identity barely overlaps" } ], "opened": [ { "q": "Does the coder-model advantage survive losing localization?", "a": "Every task here arrived with its files named. Qwen3-Coder's lead may be a code-reading advantage that repository-scale search would widen, or an artifact of already knowing where to look. Withholding localization separates those two, and it's the condition an enterprise codebase actually resembles." }, { "q": "Where does the wall-clock gap come from?", "a": "Local medians run roughly 30x the hosted ones on identical work, and this lab measured that without decomposing it. Serving throughput, tool-call round trips, and retry behavior are all candidates with different fixes. A per-turn breakdown would say whether self-hosting is slow because the model is slow or because the loop around it is." }, { "q": "What does a replicated difficulty rating do to the bar?", "a": "The residual set is measuring noise per task, which means this bench's hardest tier is currently unrated rather than hard. Running each task enough times to estimate a solve probability per capability class would give the bar a denominator. The tooling already exists in the harness and has never had the replicates to feed it." } ], "not_proved": [ "Per-task residual claims are not supported by this data and should not be quoted. Five observations produced counts of 3, 3, 4, 1 and 1, and which tasks solved barely overlapped between runs of the same model. Aggregate solve counts replicate; individual task verdicts do not.", "The Gemma 4 31B row of 11 of 22 was measured on a rented L40S venue, not on the Spark that produced every other number in the comparison. Later measurement showed that venue runs slower than the owned box, so the bar this lab clears may be set slightly low. The direction of that error flatters this lab's headline, which is why it is stated here rather than adjusted quietly.", "No self-hosted model reached 22 of 22. The bar Lab 012 defined is cleared, the hosted tier's clean sweep is not matched, and the four-repair gap is uncharacterized. Whether those four are hard or simply unlucky is exactly the question the residual finding says this pool cannot currently answer.", "Serving configuration moves local numbers by about two repairs. GLM-4.7-Flash scored 13 and then 11 across a change that tripled its key-value cache. That sits inside the seed band, so it is not a capability claim in either direction, but it means no local number here should be read without its serving config.", "The wall-clock comparison is measured but not decomposed. Local arms ran 22 to 62 minute medians against 1 to 2 minutes hosted, and this lab did not separate serving throughput from harness overhead from retry behavior. The gap is real. Its cause is unattributed." ], "executive_summary": [ "The question was where the instrument-literacy bar sits between a local model that fails it and a hosted one that clears it. It sits lower than the bracket suggested, and it isn't a size. Qwen3-Coder-Next, a 30B-class model quantized to NVFP4 on a single Spark, cleared 18 of 22 gate-verified repairs. Gemma 4 31B, a larger general model, cleared 11.", "What separates them is training for tool use, and the evidence is an inversion. In the constrained edit loop Gemma scores 17 and GLM-4.7-Flash scores 4. Swap both into the agentic harness and the order reverses: Gemma drops to 11, Flash climbs to 13. Neither model is better. They're literate in different instruments, and a table that ranks them once is wrong half the time.", "Self-hosting cleared the bar and still lost the economics, in a currency the rate card doesn't print. The best local arm spent 11.4 hours of owned GPU time to finish 18 repairs. A hosted mini finished all 22 in 40 minutes for $3.14. That's roughly 18x the wall clock to land four repairs short. Idle hardware makes the dollars free. It does not make the hours free.", "One methodological result outranks any model ranking here. The five hardest tasks, the ones the bar was written against, are measuring noise per task. Across five observations one solved every time, one solved once, and the middle three landed near half. The same model took 3 of 5 twice on almost entirely different tasks. Aggregate counts replicate. Individual task verdicts don't, and a bar written as \"solved task X\" is reading a coin flip." ], "dapm": [ { "instrument": "4plus1", "layer": "layer0", "placement": "Retained", "note": "One 128GB unified-memory box holds the entire local escalation rung at zero marginal cost, including a 106B model. The procurement finding is a qualification rule rather than a purchase: weights plus full-context key-value cache plus host headroom must fit before capability is worth measuring. The rule is only as good as the serving config it is computed against, and a default one retired a model here that in fact runs." }, { "instrument": "4plus1", "layer": "layer2b", "placement": "Retained", "note": "The gate and the harness are code you run. Serving configuration turned out to belong here too, not to the model. The same checkpoint scored 13 and then 11 across a configuration change, inside the seed band but wide enough that no local number should travel without its serving config attached." }, { "instrument": "4plus1", "layer": "layer2c", "placement": "Delegated", "note": "Repair reasoning moves to whichever open-weight model is literate in the apparatus you're running, and that seat is genuinely shoppable now. The shopping criterion is training rather than parameter count. The determines-done authority stays in the gate, which is what makes the seat safe to shop at all." } ], "seam_map": [], "writeup": [ "Lab 012 left a gap nobody could spend against. It showed a free local model failing the instrument-literacy bar and a hosted mini clearing it, which brackets the bar without locating it. For anyone who already owns hardware, that gap is the entire decision. So this lab ran the descent: five open-weight models, one Spark, the same 22 repairs and the same deterministic gate, each inside the same headless Claude Code harness.", "The floor is a coder model, and it sits lower than the bracket implied. Qwen3-Coder-Next in NVFP4 cleared 18 of 22. Devstral Small 2 and a Qwen general build cleared 15 each. Gemma 4 26B cleared 14, GLM-4.7-Flash cleared 13. Against Gemma 4 31B's 11, four of five models beat the bar, and the one that beat it hardest was trained specifically for code and tools.", "Then the result that should change how anyone reads a model table. Gemma 4 31B scores 17 of 22 in the constrained edit loop, where the model gets one shot to reproduce an anchor from memory and no tools at all. GLM-4.7-Flash scores 4 in that same loop. Swap both into the agentic harness and it reverses: Gemma falls to 11, Flash rises to 13. Neither is the better model. Gemma has byte-exact recall and can't drive tools. Flash drives tools and can't reproduce anchors. Any ranking that puts one above the other is right in one apparatus and wrong in the other.", "So what does self-hosting cost? Not money. Every local arm here ran at zero metered spend, which is the whole appeal. The cost lands in the column nobody prices. The best local arm spent 11.4 hours of GPU time to finish 18 repairs, at a 22-minute median and a 108-minute worst case. The hosted mini finished all 22 in 40 minutes for $3.14. That's about 18x the wall clock to end up four repairs short, and the spread holds across the whole field: local medians ran 22 to 62 minutes where every hosted arm ran 1 to 2.", "That gap is the product you buy when you pay, and it deserves naming because \"the hardware is already paid for\" hides it. Idle hardware makes the dollars free. It doesn't make the hours free. If the work is unattended and runs overnight, hours are cheap and self-hosting wins outright. If somebody is waiting on the result, $3.14 buys back most of a working day.", "One model looked disqualified and wasn't, which is the most instructive row here. GLM-4.5-Air, 106B parameters in AWQ 4-bit, was staged and appeared not to fit: an agentic harness needs 131K tokens of serving context, the key-value cache for that lives in the same unified memory as the weights, and at the utilization first tried there was no room left for sessions. That reading survived until the serving config was corrected. At util 0.80 the same weights on the same box yield 36.56 GiB of cache, 1.59 concurrent full-context sessions, and 12 of 22 solved. The box was never the constraint. A default `--max-num-seqs` was, the same setting that cost another model three times its cache. Venue qualification is still the right discipline, and it is only as trustworthy as the config it is computed against.", "Air also lands where the headline predicts. At 106B it is three times the size of the model that beats it and finishes sixth of seven, one repair above a 31B and six below a 30B coder build. What it does own is speed: a 9.2-minute median against Qwen3-Coder's 22, and 4.5 hours for the full pool against 11.4, because 12B active parameters decode far faster than a dense 24B. Fast, and mediocre. Size bought throughput and no capability.", "The last finding is about the bench rather than the models, and it invalidates a claim this campaign already published. The bar was written partly as \"clears the residual tasks,\" the five hardest repairs in the pool. Replicating those five showed per-task outcomes close to random. One task solved in all five observations. One solved in one. The middle three sat near half. Qwen scored 3 of 5 twice on almost disjoint task sets, overlapping only on the task everything solves. The aggregate count carries signal. The task identity does not, so any claim of the form \"this model solved that task\" is reporting a coin flip somebody happened to watch.", "Scope it. One box, one pool, one week of checkpoints, localized repair with an executable test. What carries off this bench is narrower than the numbers suggest: pick for training rather than size, name the apparatus before ranking anything, and price self-hosting in hours, because the dollars will tell you it's free." ], "abstraction": [ "Capability in an agentic stack is a pair, not a number. A model carries some literacy in each apparatus you might run it in, and those literacies are separably trained. Ranking models without naming the apparatus produces an ordering that inverts the moment somebody changes the runtime, which an architect can't see from a leaderboard and will feel in production.", "Venue qualification precedes capability measurement, and it's computable in advance. For a one-box agentic deployment the arithmetic is weights, plus concurrent sessions times key-value cache per session, plus host headroom, against the box. The trap is that the same arithmetic run against a default serving config retires models that work: it disqualified a 106B model here that later cleared 12 of 22 on the identical hardware. Compute it against a tuned config, and read the cache figure off the server rather than deriving it.", "Price self-hosting in the currency it actually costs. Dollars are zero on idle hardware and hours are not, and a 30x wall-clock multiple against a hosted tier decides whether free is worth it. That's AI Factory Economics with a different denominator: the same cost-per-unit-of-output discipline, run against time instead of spend, because on owned capacity time is the meter still running." ], "method": [ "Self-funded editorial. The harness is loopcontrolbench, published open under the MIT license at github.com/kltownsend/loopcontrolbench: the reproduce-or-drop gate, the sandboxed runner, and the arm definitions. This lab added a second agentic harness behind an environment switch, so the driver process can change while the prompt, workspace, gate and result schema stay identical, plus a trajectory reader that reports what a session actually did rather than only whether it passed. The vendor assessments and the layer placements this evidence feeds stay proprietary.", "Every local arm ran on one NVIDIA DGX Spark with 128GB of unified memory, served through vLLM. Serving configuration is recorded with each result: the checkpoint path, the utilization setting, and the key-value cache the server reported at startup. Earlier arms in this campaign recorded only a proxy alias, which is why one arm in the record cannot name the weights that produced it. Closing that gap is what the provenance capture was added for." ], "assessments": [ { "instrument": "4plus1", "vendor": "NVIDIA AI Platform", "layer": "Layer 0 · Compute", "grade": "NVIDIA Strength — Silicon Authority", "assessed_date": "July 31, 2026", "version": "v1.5 - Idle-Silicon Economics Reconciliation", "source_url": "https://layer2c.com/assessment/nvidia", "snapshot_date": "2026-08-11" } ], "publisher": "The Advisor Bench LLC", "program": "Layer2C Labs", "url": "https://labs.layer2c.com/labs/self-host-floor", "markdown_url": "https://labs.layer2c.com/labs/self-host-floor.md", "json_url": "https://labs.layer2c.com/labs/self-host-floor.json" }