They can all write it. Not all of them take direction.
By Keith Townsend · August 11, 2026
The pitch is that picking the right model tier is the decision that determines whether delegated coding work succeeds. This lab walked a model ladder down from an 80B coder to a 2B, on one from-scratch task judged by a deterministic conformance gate, expecting to find the size where capability breaks. It did not find that. Above roughly 12B every rung wrote a structurally complete, plausibly organized server, and the same 26B model on the same task produced three different approaches in three runs, scoring 7 of 7, then 6 of 7, then 7 of 7, in 12, 79 and 21 minutes. The approach predicted the outcome. The size mostly did not.
Scoped to one from-scratch API contract judged by one conformance gate, on a single owned box. The ladder is not evenly sampled: the 26B ran three times and gpt-5.4-mini twice, while the 80B, 12B, E4B and E2B rungs are single arms and cannot carry a distribution claim. The difficulty of this contract is concentrated in one operation, so the seven-unit score offers less resolution than its denominator suggests. Nothing here measures whether a passing implementation survives a downstream consumer, which is the number the migration control plane actually needs and which this lab does not touch.
Don’t Do not buy a model tier expecting it to buy you consistency. The 26B-A4B scored 7 of 7, then 6 of 7, then 7 of 7 on an identical venue, taking 12, 79 and 21 minutes. The 12B scored 6 of 7. On this contract the tiers overlap, and picking up a tier bought a narrower spread rather than a better ceiling.
Do Constrain the toolset instead of lengthening the playbook. Every arm that attempted large exact-match edits lost an operation; every arm that used small edits or none at all scored 7 of 7. The failure lived in editing strategy, a subject no API playbook would think to cover. Removing the operation removes the failure class, where another page of instructions would not have touched it.
Do Ratchet the artifact and cap the iteration. A repair loop closed five of six failures in a single turn once its feedback carried the gate's remediation text, then destroyed its own best result twice when left running: 1, 6, 3, 1. Keep the best artifact, refuse a regressing turn, escalate rather than iterate. The loop earns its keep exactly once.
Do Read variance as the deliverable, not as noise. Per-attempt verified yield was 2 of 2 for gpt-5.4-mini at about $2.08 and six minutes per verified unit, and 2 of 3 for the 26B at zero marginal dollars and roughly 56 expected minutes. The yield, not the ceiling, is what prices a retry policy.
Disclosure: Google Cloud is a client of this practice, and the model family at the centre of this lab is Google Gemma 4. No vendor commissioned, funded, or previewed any of this, and no result was held or softened because of a relationship. The local rungs ran on owned hardware at zero marginal cost; the only money spent was a few dollars of API tokens, itemised on the card.
The pitch behind every model-tier decision says size is the variable that matters: pick the right tier and delegated coding work succeeds, pick too small and it breaks. This lab walked a ladder down from an 80B coder to a 2B, on one from-scratch task judged by a deterministic conformance gate, expecting to find the size where capability breaks. It didn't find one. Above roughly 12B every rung wrote a structurally complete, plausibly organized server, and what separated the arms was whether the session noticed its own approach failing. The size mostly didn't predict the score. The approach did, every time.
The instrument is a from-scratch build, chosen because translation tasks hide the interesting failure. The workspace gets a six-operation OpenAPI contract, a task document, and a 438-chunk corpus fixture drawn from 258 documents, and nothing else. The reference implementation exists and is never mounted. Schemathesis judges the result with every check enabled, examples, coverage, fuzzing, and a stateful sequence suite, scored per operation across seven units. The workers were gpt-5.4-mini as the metered reference, Qwen3-Coder-Next 80B-A3B, DeepSeek V4 Flash, and the Gemma 4 family at 26B-A4B, 12B QAT (quantization-aware trained), E4B and E2B. Every local rung ran on an owned NVIDIA DGX Spark, driven through the same agentic harness, Anthropic's Claude Code.
The cliff that never came
gpt-5.4-mini, Qwen3-Coder-Next and DeepSeek V4 Flash cleared the contract. Gemma 4 26B-A4B cleared it twice in three attempts. The 12B missed by one route in a 405 handler while passing 333 of 333 stateful scenarios. Even the E4B wrote a complete server, 9,410 bytes, longer than either arm that scored 7 of 7, with sensible structure, error handling, and an in-memory index build. It scored 1 of 7 on three small independent defects. It split the fixture on the literal two-character sequence backslash-n instead of on a newline, which shreds all 438 records at their escaped line breaks; it omitted the Allow header on unsupported methods, the same miss the 12B made; and it violated the response schema on two operations. Patching the escaping error alone, a verified one-line diff, cleared the 404s and left the score at 1 of 7, because the other two defects were already failing the same endpoints. The 12B wrote that line correctly. Only the E2B failed to produce a program at all. It drove the harness to a clean exit in three minutes and left behind a 108-byte lockfile and nothing else. The floor is real, and it sits below 4B.
That E4B row is why "small models can't code" is the wrong reading. If small models produced incoherent output, the fix is a bigger model. They produced plausible output carrying a handful of small precision errors, and the fix for that is a gate, because a reviewer reading a correct-looking corpus loader doesn't catch the escaping and won't count the missing header either. The conformance run caught all of it in 1.4 seconds. Worth being precise about, since I got it wrong the first time: the E4B is not one edit from working. Fixing its escaping error changes nothing about the score. The honest limit is that the E4B arm is a single run, and its rerun failed differently: it loaded the corpus correctly and registered no routes at all, returning a well-formed 404 for every endpoint in the contract. Both failures are plausible-looking code. Neither is incoherence.
Same weights, three strategies
The result I didn't expect came from running the 26B three times. Same weights, same box, same task document. One session chose surgical edits, one chose whole-file replacement through an exact-match edit tool, one used shell heredocs. The scores were 7 of 7, then 6 of 7, then 7 of 7, in 12, 79 and 21 minutes. Across all eight arms measured, every one that attempted large exact-match edits lost an operation, and every one that used small edits or none at all scored 7 of 7. There is no counterexample in the set.
The mechanism is countable from a session log without judging a line of code. The failing run answered a 6,397-character match failure by trying 19,023 characters, then 19,252. It responded to failure by tripling the size of the string it couldn't reproduce. It then found the working approach at 53 characters, abandoned it, went back to roughly 6,100 and repeated that seven more times. The passing run failed once at 1,648 characters and its next edit was 282. One session noticed its strategy was failing. The other didn't. That's judgment, measured, and it's the intervention I used to make by hand in an earlier generation of coding assistant. I stopped having to make it somewhere along the way, without noticing when.
I've argued for years that handing three engineers the same migration playbook gets you three approaches. Here one model produced three in an afternoon, and the difference between them was invisible in the playbook's own vocabulary, which talks about contracts and status codes while the failure lived in how a file got edited. Another page of instructions doesn't reach that axis. Constraining the toolset does: remove the exact-match edit tool and the failure class goes with it. The acceptance criteria are the one thing that can be held fixed across executors, which is the argument for a deterministic gate arriving from the delegation side rather than the determinism side. It holds for human executors too, with one break. An engineer notices when an approach stops working. This session tripled a failed edit twice, then discarded the one approach that worked.
Price the variance, not the ceiling
The tiers overlap on this contract, so what a metered tier sells is the distribution. Per-attempt verified yield was 2 of 2 for gpt-5.4-mini at about $2.08 and 5.6 minutes mean per attempt, and 2 of 3 for the 26B at zero marginal dollars and 37.5 minutes mean, roughly 56 expected minutes to bank one verified unit. Paying bought a narrower spread, not a better ceiling. That's a different purchase than the tier pitch describes, and the yield, not the ceiling, is what prices a retry policy.
The harness moved cost more than the model did. The same 12B reached the same 6 of 7 in 63 minutes through the agentic harness and in 4.1 minutes through a constrained loop that asks for whole files: 1,040,460 input tokens against 29,614 total. Fifteen times the wall clock, roughly 35 times the tokens, identical score. Agentic scaffolding is priced for tasks whose inputs must be discovered, and this task hands its inputs over at the start.
The repair loop earns its keep exactly once. With feedback carrying only a failure count, four turns produced 1, 1, 1, 1. With feedback carrying the gate's own remediation text, the same model produced 1, 6, 3, 1: five operations closed in a single turn, then an overcorrection that flipped a check to its mirror image, then a rewrite that inverted a working handler. Iteration count did nothing. Feedback content did everything, once. So ratchet the artifact, refuse a regressing turn, and escalate instead of iterating.
What this lab can't say yet
Bounds first. Four of the seven rungs are single arms, so no distribution claim survives there. The contract concentrates its difficulty in one operation, the endpoint carrying typed query parameters with date-format validation, so seven units resolve less than the denominator suggests. Nothing here measures whether a passing implementation survives a downstream consumer, which is the number the migration control plane actually needs. And the harness produced twelve defects during this program, every one found, fixed, and disclosed, and every one pointed at the model and away from my own tooling. Run unattended, this descent would have reported a capability floor near 80B with every rung below it wrong, and every wrong rung would have looked clean.
The decisive experiment is one arm, and it's unrun. Take the 26B, remove the exact-match edit tool from its toolset, and run it again. If per-attempt yield goes to three of three, the variance was a harness property, and the local rung is materially more viable than this page reports, with expected time per verified unit falling from about 56 minutes to about 21. If the yield holds at two of three, the strategy correlation is coincidence and my judgment section is wrong. Either answer is worth more than another rung on the ladder. That's the arm the next session runs first.
The numbers
Where each layer belongs
| Layer | Placement |
|---|---|
Layer 2B · Runtime Application Runtime & Execution The control point is the conformance gate and the escalation policy, not the model. The gate is an OpenAPI contract plus Schemathesis with all checks on, and it caught every defect on this page including a one-character escaping error inside otherwise-correct code. The harness itself is an authority surface the lab did not expect: an exact-match edit tool is a constraint on who can do the work, and it silently penalises models with lower reproduction precision. | Retained |
Layer 2C · Reasoning Agentic Infrastructure — The Reasoning Plane Open weights on owned hardware cleared the contract, so the reasoning plane can be retained. What cannot be retained by model choice alone is consistency: the same local rung delivered a verified unit on two attempts in three, with a 6.6x wall-clock spread. Delegating to a metered tier bought a narrower distribution rather than a higher ceiling, which is a different purchase than the one usually described. | Retained / Delegated |
Method and disclosure
Editorial and self-funded. The local rungs ran on an owned NVIDIA DGX Spark at zero marginal cost; the metered arms cost a few dollars of API tokens, itemised on the card. Disclosure: Google Cloud is a client of this practice and the Gemma 4 family is the centre of this lab, chosen because it is the only family cached locally at five sizes spanning dense and sparse, which holds tokenizer and training recipe roughly constant while parameter count moves. Anthropic's Claude Code is the agentic harness for every arm and is itself implicated in the findings, particularly the exact-match edit tool. No vendor commissioned, funded, or previewed any of this, and no result was held because of a relationship.
The instrument: a six-operation OpenAPI contract over a 438-chunk, 258-document fixture, judged by Schemathesis 4.24.3 with all checks on, scored per operation. Local rungs served on vLLM 0.25.1 at each model's native context with a per-model tool-call parser, gated before every arm by a tool-call smoke test. The contract, the fixture manifest, the scoring script and the full defect chronology ship in the raw detail; the corpus content stays private.
Twelve harness defects were found and fixed during the program and all of them are disclosed, because a descent is a venue audit that happens to produce capability data. In order: a 32,000-token output reservation that exceeded a small model's context before it saw the task; a context window served below native; an output budget consumed entirely by a reasoning model's thinking; an arm timeout sized against a fast rung; an output contract that required naming a JSON file with a comment, which npm cannot parse; a feedback window that truncated away the validator's remediation text; a driver reading a scoreboard instead of the detailed gate output; a 2,000-character context truncation that hid the worker's own working implementation from it; a hardcoded tool-call parser that fails silently with clean HTTP 200s; the same output reservation, correct for the small rungs, failing a large one mid-write; a controller returning an empty string silently; and a loop that scored a stale artifact three times because it never checked whether a turn wrote a file. Every one produced a confident wrong reading and every one pointed at the model.
The essay on this page was rendered from the lab’s frozen structured record by a model writing under this practice’s voice specification, with a deterministic gate checking every figure against the record before publication. The record is the canonical surface: where prose and record disagree, the record rules.
Download the raw lab detail (Markdown)