{ "slug": "tier-boundary", "lab_number": 16, "title": "They can all write it. Not all of them take direction.", "type": "editorial", "status": "published", "sponsor": null, "author": "Keith Townsend", "date": "August 11, 2026", "date_iso": "2026-08-11", "layers": [ "layer2b", "layer2c" ], "axes": [ { "instrument": "4plus1", "layers": [ "layer2b", "layer2c" ], "has_dapm_table": true } ], "supersedes": [], "superseded_by": null, "corrects": [], "corrected_by": [], "vendors": [ { "key": "google", "role": "model" }, { "key": "openai", "role": "model" }, { "key": "alibaba", "role": "model" }, { "key": "deepseek", "role": "model" }, { "key": "nvidia", "role": "hw" }, { "key": "anthropic", "role": "harness" } ], "themes": [ "judgment-transfer", "loop-control", "small-model-viability", "validator-authority" ], "finding": "Every model above a 2B rung wrote a complete, plausible API implementation; the same model on the same task took three different approaches across three runs, and the approach predicted the score in every arm measured.", "question": "The pitch is that picking the right model tier is the decision that determines whether delegated coding work succeeds. This lab walked a model ladder down from an 80B coder to a 2B, on one from-scratch task judged by a deterministic conformance gate, expecting to find the size where capability breaks. It did not find that. Above roughly 12B every rung wrote a structurally complete, plausibly organized server, and the same 26B model on the same task produced three different approaches in three runs, scoring 7 of 7, then 6 of 7, then 7 of 7, in 12, 79 and 21 minutes. The approach predicted the outcome. The size mostly did not.", "load": "One from-scratch implementation task: a six-operation OpenAPI contract over a 438-chunk, 258-document corpus fixture, with no reference implementation in the workspace, judged by Schemathesis with all checks on (examples, coverage, fuzzing, and a stateful sequence suite). Scoring is per operation, seven units. Workers: gpt-5.4-mini, Qwen3-Coder-Next 80B-A3B, DeepSeek V4 Flash, and the Gemma 4 family at 26B-A4B, 12B QAT, E4B and E2B, all local rungs served on an owned NVIDIA DGX Spark and driven through the same agentic harness.", "verdict": { "scope": "Scoped to one from-scratch API contract judged by one conformance gate, on a single owned box. The ladder is not evenly sampled: the 26B ran three times and gpt-5.4-mini twice, while the 80B, 12B, E4B and E2B rungs are single arms and cannot carry a distribution claim. The difficulty of this contract is concentrated in one operation, so the seven-unit score offers less resolution than its denominator suggests. Nothing here measures whether a passing implementation survives a downstream consumer, which is the number the migration control plane actually needs and which this lab does not touch.", "independence": "Disclosure: Google Cloud is a client of this practice, and the model family at the centre of this lab is Google Gemma 4. No vendor commissioned, funded, or previewed any of this, and no result was held or softened because of a relationship. The local rungs ran on owned hardware at zero marginal cost; the only money spent was a few dollars of API tokens, itemised on the card.", "calls": [ { "kind": "dont", "text": "Do not buy a model tier expecting it to buy you consistency. The 26B-A4B scored 7 of 7, then 6 of 7, then 7 of 7 on an identical venue, taking 12, 79 and 21 minutes. The 12B scored 6 of 7. On this contract the tiers overlap, and picking up a tier bought a narrower spread rather than a better ceiling." }, { "kind": "do", "text": "Constrain the toolset instead of lengthening the playbook. Every arm that attempted large exact-match edits lost an operation; every arm that used small edits or none at all scored 7 of 7. The failure lived in editing strategy, a subject no API playbook would think to cover. Removing the operation removes the failure class, where another page of instructions would not have touched it." }, { "kind": "do", "text": "Ratchet the artifact and cap the iteration. A repair loop closed five of six failures in a single turn once its feedback carried the gate's remediation text, then destroyed its own best result twice when left running: 1, 6, 3, 1. Keep the best artifact, refuse a regressing turn, escalate rather than iterate. The loop earns its keep exactly once." }, { "kind": "do", "text": "Read variance as the deliverable, not as noise. Per-attempt verified yield was 2 of 2 for gpt-5.4-mini at about $2.08 and six minutes per verified unit, and 2 of 3 for the 26B at zero marginal dollars and roughly 56 expected minutes. The yield, not the ceiling, is what prices a retry policy." } ] }, "objection": { "q": "A 4B model wrote a complete server and scored 1 of 7. Is this not just a demonstration that small models cannot code?", "a": [ "No, and the failure is specific enough to say why. Gemma 4 E4B wrote a 9,410-byte server, longer than either arm that scored 7 of 7, correctly structured, with error handling and an in-memory index build. It carried three independent defects. It split the fixture on the two-character sequence backslash-n rather than on a newline, which shreds every JSON record at its escaped line breaks; it omitted the `Allow` header on unsupported methods, the same defect the 12B hit; and it violated the response schema on two operations. Patching only the escaping error, verified as a one-line diff, cleared the data-loading failures and left the score at 1 of 7, because the other two defects were already failing the same endpoints. Those are precision defects inside competent structure, not an inability to write the program.", "The distinction matters because it changes what you buy. If small models produced incoherent output, the fix is a bigger model. If they produce plausible output with a fatal typo, the fix is a gate, because that failure mode is exactly the one human review handles worst. A reviewer reading a correct-looking loadCorpus function does not catch the escaping. The conformance run caught it in 1.4 seconds.", "The honest limit is that the E4B arm is a single run, and its rerun failed differently. On the second attempt the same model wrote a server that loaded the corpus correctly and registered no routes at all, returning a well-formed 404 for every endpoint in the contract. Both failures are plausible-looking code. Neither is incoherence." ] }, "numbers": [ { "label": "Same model, same task, three approaches", "value": "7/7, 6/7, 7/7", "note": "Gemma 4 26B-A4B NVFP4, identical venue, in 12, 79 and 21 minutes. Median exact-match edit target of 254 characters, then 6,112, then none at all. The approach tracked the score" }, { "label": "Approach predicted the score", "value": "8 of 8 arms", "note": "Every arm with a small or absent exact-match target scored 7 of 7; both arms with a median target above 6,000 characters scored 6 of 7. The one arm outside the pattern, E4B at 1 of 7, failed for a separately diagnosed reason" }, { "label": "Judgment, measured", "value": "6,397 to 19,023 to 19,252", "note": "The failing run's response to three consecutive match failures was to enlarge the target. The passing run failed once at 1,648 characters and dropped to 282. Countable from a session log without judging any code" }, { "label": "Harness cost at identical score", "value": "63 min vs 4.1 min", "note": "Gemma 4 12B QAT reached 6 of 7 through the agentic harness and through a constrained loop. 1,040,460 input tokens against 29,614 total across all four loop turns" }, { "label": "The repair loop, two feedback conditions", "value": "1,1,1,1 vs 1,6,3,1", "note": "Identical model, task and turn count. The only change was whether the feedback window carried the validator's remediation text or only its failure counts" }, { "label": "Three defects, not one", "value": "E4B, 1 of 7", "note": "A 9,410-byte server, longer than either 7-of-7 arm, carrying an escaping error that shredded 438 records, a missing Allow header on five operations, and two schema violations. Patching the escaping error alone is a one-line diff that clears the 404s and leaves the score at 1 of 7. Corrected after publication: the entry first attributed all six lost operations to the one character" }, { "label": "One header, five operations", "value": "12B, 1 of 7 to 6 of 7", "note": "The clean case of doing the work and failing the gate. The 12B read the task, wrote its own 405 handler, and omitted the Allow header the task does not name. Patching only that header into its own handler took the same implementation to 6 of 7, and the model closed the gap itself one turn later once the feedback carried the remediation" }, { "label": "Per-attempt verified yield", "value": "2/2 vs 2/3", "note": "gpt-5.4-mini at about $2.08 and 5.6 minutes mean per attempt; Gemma 4 26B-A4B at zero marginal dollars and 37.5 minutes mean, giving roughly 56 expected minutes to bank one verified unit" }, { "label": "The bottom of the ladder", "value": "E2B, no build", "note": "Drove the agentic harness to a clean exit in three minutes, passed the tool-call gate, and produced a 108-byte lockfile and no implementation. The floor is real and it is below 4B" }, { "label": "Controller footprint", "value": "418 in / 270 out tokens", "note": "A frontier controller reading the gate and directing the next attempt costs roughly nine tenths of a cent per turn. It never writes code. Measured, not run to a conclusion: see the bound" }, { "label": "Venue defects found and fixed", "value": "12", "note": "Every one produced a confident wrong reading, and every one pointed the same direction: at the model and away from the harness. Run unattended this descent would have reported a capability floor near 80B with every rung below it wrong" } ], "opened": [ { "q": "Does removing the exact-match edit tool close the variance?", "a": "The correlation is eight arms with no counterexample, and correlation is where observation saturates. The decisive experiment is one arm: run the 26B with the edit tool removed from its toolset. If per-attempt yield goes to three of three, the variance was a harness property and the local rung is materially more viable than this page reports, with expected time per verified unit falling from about 56 minutes to about 21. If the yield holds at two of three, the strategy correlation is coincidence and the judgment section on this page is wrong. That arm is unrun, and it is the first thing the next session should do." }, { "q": "Where is the floor for loaned judgment, as opposed to the floor for doing the work?", "a": "The loop-control arm put a frontier controller behind a 4B worker with the edit tool removed, and it did not answer the question. The controller performed correctly, producing accurate code-free diagnoses in about a thousand characters a turn. The worker stopped calling tools after its first turn, reasoned at length, and printed its answer into chat; the recovered chat implementation loaded the corpus correctly and registered no routes at all. So the observed failure is that receiving direction knocked a 4B out of tool-use mode, and that the second attempt was worse than the first. Whether a 4B can be steered at all, given a delivery mechanism that does not depend on it calling a write tool, is open. The harness now detects a no-write turn, which it did not when this arm ran." } ], "not_proved": [ "It did not establish a tier floor. A 26B looked like one after a single arm scoring 7 of 7, and the claim did not survive the second arm. The correct statement is that this contract is reachable at 26B and not reliable there, and the sequence of asserting a floor on one run and losing it on the next is itself part of the finding.", "Four of the seven rungs are single arms. The 80B, 12B, E4B and E2B results cannot carry a distribution claim, and per-attempt yield is precisely the statistic that most needs a larger sample. Only the 26B (three arms) and gpt-5.4-mini (two) support any variance statement on this page.", "The contract concentrates its difficulty in one operation. Nearly every failure landed on the single endpoint carrying typed query parameters with date-format validation; the other six test whether a model can serve JSON from a fixture, which everything above the 2B rung can do. Partial credit out of seven is real but its resolution is largely illusory, and a pool that separates tiers needs its difficulty distributed deliberately.", "It did not measure downstream survival. The gate rules on conformance to a contract. Whether a passing implementation breaks a consumer that depends on it is the number the migration control plane needs, and it stays unmeasured here.", "The loop-control result is not a capability verdict. The 4B arm failed on delivery, not on judgment: the worker stopped writing files after turn 1 and the harness scored a stale artifact three times before that was caught. The finding that survives is narrow, that a follow-up prompt containing review feedback knocked a 4B out of tool-use mode, and it rests on one arm.", "The harness was the source of twelve defects during this program, all found and fixed, all disclosed in the method note. Their existence is a caution about every unattended descent, including the parts of this one that produced clean-looking numbers before the defect was found." ], "executive_summary": [ "The lab was staged to find the size where capability breaks and it did not find one. Above the 2B rung every model wrote a structurally complete, plausibly organised server: sensible naming, error handling, comments, an in-memory index built from the fixture. Whether that index was built correctly is where the rungs separate, and at the bottom it was not. The bottom is real and it is low. Gemma 4 E2B drove the agentic harness to a clean exit in three minutes and produced a 108-byte lockfile and nothing else. Everything above it produced a program.", "What separated the arms was the approach each session happened to take. The same 26B model on the same task, same weights, same box, same contract, chose surgical edits on one run, whole-file replacement through an exact-match tool on another, and shell heredocs on a third. It scored 7 of 7, 6 of 7, and 7 of 7 in 12, 79, and 21 minutes. Across every arm measured, each one that attempted large exact-match edits lost an operation and each one that did not scored 7 of 7. There is no counterexample in the set.", "The mechanism is legible in the trace. The failing run answered a 6,397-character match failure by trying 19,023 characters, then 19,252: it responded to failure by tripling the size of the string it could not reproduce. It then found the working approach at 53 characters, abandoned it, returned to roughly 6,100 and repeated that seven more times. The passing run failed once at 1,648 characters and its next edit was 282. One noticed the strategy was failing. The other did not.", "The harness mattered more than the model. The same 12B reached the same 6 of 7 in 63 minutes through the agentic harness and in 4.1 minutes through a constrained loop that asks for whole files and does its own file handling. Fifteen times the wall clock, and roughly 35 times the tokens, for an identical score. Agentic scaffolding is priced for tasks whose inputs must be discovered, and this task hands its inputs over at the start.", "The repair loop is worth exactly one turn. With feedback carrying only a failure count, four turns produced 1, 1, 1, 1. With feedback carrying the gate's own remediation text, the same model on the same task produced 1, 6, 3, 1: five operations closed in a single turn, then an overcorrection that flipped a check to its mirror image, then a rewrite that inverted a working handler's semantics. Iteration count did nothing. Feedback content did everything, once." ], "dapm": [ { "instrument": "4plus1", "layer": "layer2b", "placement": "Retained", "note": "The control point is the conformance gate and the escalation policy, not the model. The gate is an OpenAPI contract plus Schemathesis with all checks on, and it caught every defect on this page including a one-character escaping error inside otherwise-correct code. The harness itself is an authority surface the lab did not expect: an exact-match edit tool is a constraint on who can do the work, and it silently penalises models with lower reproduction precision." }, { "instrument": "4plus1", "layer": "layer2c", "placement": "Retained / Delegated", "note": "Open weights on owned hardware cleared the contract, so the reasoning plane can be retained. What cannot be retained by model choice alone is consistency: the same local rung delivered a verified unit on two attempts in three, with a 6.6x wall-clock spread. Delegating to a metered tier bought a narrower distribution rather than a higher ceiling, which is a different purchase than the one usually described." } ], "seam_map": null, "writeup": [ "The instrument is a from-scratch task, chosen because translation tasks hide the interesting failure. The workspace receives an OpenAPI contract, a task document and a 438-chunk corpus fixture, and nothing else; the reference implementation exists but is never mounted. Scoring is per operation across six endpoints plus a stateful sequence suite, judged by Schemathesis with all checks enabled. The task document states explicitly that every check the runner performs is part of the bar, including checks the contract does not spell out, and names HTTP method handling as an example. That sentence is load-bearing: it is what makes an unstated requirement a fair test rather than a trap.", "The descent found no capability cliff, which was the expected result and did not arrive. gpt-5.4-mini, Qwen3-Coder-Next 80B-A3B and DeepSeek V4 Flash cleared the contract. Gemma 4 26B-A4B cleared it twice in three attempts. The 12B missed by one route in a 405 handler while passing 333 of 333 stateful scenarios. The E4B wrote the longest implementation of any arm and scored 1 of 7 on three small independent defects. Only the E2B failed to produce a program at all. From 2B to 80B, the thing that changed was not whether the model could write the program.", "What changed was the density of small precision errors and, separately, whether the model noticed one. Those are two different axes and the lab was measuring neither at the outset. The precision axis is visible in the E4B's escaping error and in the 12B's missing route. The judgment axis is visible only in the trace of a failing run: an exact-match edit tool requires reproducing a target string character-perfect from memory, and one 26B session answered three consecutive failures by enlarging the target from 6,397 to 19,023 to 19,252 characters. It later found the working approach at 53 characters and abandoned it. The passing session, given the same tool and the same task, shrank its target after its single failure.", "That is the migration playbook thesis under laboratory conditions. Give three engineers the same playbook and you get three approaches; here one model given one task document produced three. The approaches were not equally good, and the difference was invisible in the playbook's own vocabulary, which talks about contracts and status codes while the failure lived in how a file got edited. No amount of additional playbook detail reaches that axis. Which leaves the acceptance criteria as the only thing that can be held fixed across executors, and that is the argument for a deterministic gate arriving from the delegation side rather than the determinism side. It holds for human executors too.", "The human analogy breaks in one place, and the break is the case for the gate. Three engineers taking three approaches can each say why, and each notices when theirs stops working. None of them answers a failed edit by tripling its size, twice, then discards the approach that worked. With human executors a validator catches the occasional bad approach. With delegated model work it is the only thing standing between a thrash spiral and a shipped 6 of 7 that passes 333 of 333 stateful scenarios and reads as finished.", "Five control mechanisms came out of this, each forced by an observed failure rather than designed in advance. The gate catches a wrong result. An action-space constraint, removing the exact-match edit tool, makes thrash structurally impossible. A ratchet stops a regressing turn from overwriting a better artifact. An iteration cap stops unbounded self-directed retry. A thrash detector, N consecutive same-mode failures with no reduction in target size, is the correct trigger for escalation rather than iteration. The constraint has a cost the bench measured: removing the edit tool forces whole-file regeneration, and regeneration is what destroyed the repair loop's best result. The mechanisms are complements, not alternatives." ], "abstraction": [], "method": [ "Editorial and self-funded. The local rungs ran on an owned NVIDIA DGX Spark at zero marginal cost; the metered arms cost a few dollars of API tokens, itemised on the card. Disclosure: Google Cloud is a client of this practice and the Gemma 4 family is the centre of this lab, chosen because it is the only family cached locally at five sizes spanning dense and sparse, which holds tokenizer and training recipe roughly constant while parameter count moves. Anthropic's Claude Code is the agentic harness for every arm and is itself implicated in the findings, particularly the exact-match edit tool. No vendor commissioned, funded, or previewed any of this, and no result was held because of a relationship.", "The instrument: a six-operation OpenAPI contract over a 438-chunk, 258-document fixture, judged by Schemathesis 4.24.3 with all checks on, scored per operation. Local rungs served on vLLM 0.25.1 at each model's native context with a per-model tool-call parser, gated before every arm by a tool-call smoke test. The contract, the fixture manifest, the scoring script and the full defect chronology ship in the raw detail; the corpus content stays private.", "Twelve harness defects were found and fixed during the program and all of them are disclosed, because a descent is a venue audit that happens to produce capability data. In order: a 32,000-token output reservation that exceeded a small model's context before it saw the task; a context window served below native; an output budget consumed entirely by a reasoning model's thinking; an arm timeout sized against a fast rung; an output contract that required naming a JSON file with a comment, which npm cannot parse; a feedback window that truncated away the validator's remediation text; a driver reading a scoreboard instead of the detailed gate output; a 2,000-character context truncation that hid the worker's own working implementation from it; a hardcoded tool-call parser that fails silently with clean HTTP 200s; the same output reservation, correct for the small rungs, failing a large one mid-write; a controller returning an empty string silently; and a loop that scored a stale artifact three times because it never checked whether a turn wrote a file. Every one produced a confident wrong reading and every one pointed at the model." ], "assessments": [], "publisher": "The Advisor Bench LLC", "program": "Layer2C Labs", "url": "https://labs.layer2c.com/labs/tier-boundary", "markdown_url": "https://labs.layer2c.com/labs/tier-boundary.md", "json_url": "https://labs.layer2c.com/labs/tier-boundary.json" }