# Lab 016 raw lab detail: They can all write it. Not all of them take direction.

Working notes behind the published entry. Every substrate detail ships here: the contract, the
serving configs, the per-arm numbers, the session traces, and the twelve harness defects that were
caught and corrected on the way. The corpus content behind the fixture, the assessment methodology
it encodes, and the judgment that turns a score into a placement stay proprietary. Returns, not
algorithms.

## The one question

Lab 012 found gpt-5.4-mini and Opus 4.8 tied at 22 of 22 on a repair pool: a fourfold price gap
that bought nothing measurable. That is a statement about a saturated pool, not about the models.
This lab went looking for the difficulty line above which a higher tier starts buying solves, and
approached it from the cheap end: hold one task and one gate fixed, walk a model ladder down from
an 80B coder to a 2B, and find where it breaks.

It did not break where expected. Above roughly 12B every rung wrote a structurally complete,
plausibly organised server. The separation that did appear was not between models.

## The instrument

One from-scratch implementation task. The workspace receives an OpenAPI 3.1 contract with six
operations, a task document, and a corpus fixture. It does not receive a reference implementation:
that file exists, at 6,573 bytes, and is never mounted in any workspace.

The fixture is 438 chunks across 258 documents, sha256 `fbc8c9d8...42b1`, stratified so that no
single field dominates. An earlier build sorted by date and produced 91.5% undated records; it was
rebuilt. The corpus content is proprietary and does not ship.

Scoring is per operation: six endpoints plus a stateful sequence suite, seven units. The judge is
Schemathesis 4.24.3 with `--checks all`, which runs examples, coverage, fuzzing and stateful
phases and includes checks the contract does not spell out. The task document states that
explicitly and names HTTP method handling as an example, which is what makes an unstated
requirement a fair test rather than a trap.

Local rungs served on one NVIDIA DGX Spark, 128GB unified memory, GB10, through vLLM 0.25.1, each
at its own native context with a per-model tool-call parser. Every arm was gated by a tool-call
smoke test before it ran. The agentic harness is a headless Claude Code session routed through a
LiteLLM proxy; the constrained loop is a separate driver that asks for whole files and does its own
file handling.

## The measured spine

Agentic harness, one arm per row unless noted:

| rung | precision | contract | wall clock | turns | in / out tokens |
| --- | --- | --- | --- | --- | --- |
| gpt-5.4-mini (run 1) | hosted | 7/7 | 6.8 min | 42 | 292,978 / 53,465 |
| gpt-5.4-mini (run 2) | hosted | 7/7 | 4.4 min | 21 | 108,236 / 31,999 |
| Qwen3-Coder-Next 80B-A3B | local | 7/7 | not recorded | n/a | n/a |
| DeepSeek V4 Flash | FP8, 156 GiB | 7/7 | not recorded | n/a | n/a |
| Gemma 4 26B-A4B (run 1) | NVFP4 | 7/7 | 12 min | 28 | 460,609 / 18,045 |
| Gemma 4 26B-A4B (run 2) | NVFP4 | **6/7** | **79 min** | 39 | 2,277,297 / 79,159 |
| Gemma 4 26B-A4B (run 3) | NVFP4 | 7/7 | 21.5 min | n/a | n/a |
| Gemma 4 12B QAT | w4a16 | 6/7 | 63 min | 19 | 1,040,460 / 76,510 |
| Gemma 4 E4B | bf16 | 1/7 | 8 min | 5 | 25,471 / 8,514 |
| Gemma 4 E2B | bf16 | DID_NOT_BUILD | 3 min | 6 | 18,581 / 6,301 |

Every 7/7 was verified against the port precondition and re-scored where it mattered. The 26B run
1 was independently re-scored on a verified-free port and returned 7/7 a second time; its
implementation is 6,191 bytes against the reference's 6,573, different hash, different filename.
That verification exists because an earlier arm in this program produced a false 7/7: a previous
model's server still held the scoring port and was scored twice. The precondition added after that
fired again during the 26B run and is recorded in its venue file.

Decode rate, single stream, same box:

| model | precision | active params | decode |
| --- | --- | --- | --- |
| Qwen3-8B | bf16 dense | 8B | 10-11 tok/s |
| Gemma 4 12B QAT | w4a16 dense | 12B | 22.0 tok/s |
| Gemma 4 26B-A4B | NVFP4 sparse | ~4B | 30.0 tok/s |
| DeepSeek V4 Flash | FP8 sparse | sparse | 48.8 tok/s |

The 26B is faster than the 12B from its own family while carrying twice the weights. Decode tracks
active parameters and bytes read per token, not total parameters.

## What actually separated the arms

Edit strategy, measured from the session logs. `old_string` is the target the model must reproduce
character-perfect for an exact-match edit to apply:

| arm | Edit calls | median `old_string` | max | match failures | score |
| --- | --- | --- | --- | --- | --- |
| gpt-5.4-mini run 1 | 1 | 81 | 81 | 0 | 7/7 |
| gpt-5.4-mini run 2 | 0 | n/a | n/a | 0 | 7/7 |
| DeepSeek V4 Flash | 0 (170 Bash) | n/a | n/a | 0 | 7/7 |
| Gemma 26B run 1 | 11 | 254 | 2,002 | 1 | 7/7 |
| Gemma 26B run 2 | 13 | **6,112** | **19,252** | **8** | 6/7 |
| Gemma 26B run 3 | 0 (26 Bash) | n/a | n/a | 0 | 7/7 |
| Gemma 12B QAT | 2 | **7,315** | 7,315 | 1 | 6/7 |
| Gemma E4B | 2 | 0 | n/a | 0 | 1/7 |

Every arm with a small or absent exact-match target scored 7/7. Both arms with a median target
above 6,000 characters scored 6/7. The E4B is outside the pattern for a separately diagnosed
reason, below.

## The judgment trace

Edit sizes in chronological order, `!` marking a match failure. Same weights, same box, same task
document, four hours apart:

```
run 1 (7/7, 12 min):
  372  2002  203  239  240  254  148  1648!  282  298  194

run 2 (6/7, 79 min):
  6397!  19023!  19252!  0  53  6148  6046!  6168  6112!  6112!  6112  6093!  6093!
```

Run 1 failed once at 1,648 characters and its next edit was 282. It shrank the target.

Run 2 failed at 6,397 and answered by trying 19,023, then 19,252. It responded to failure by
tripling the size of the string it could not reproduce, twice. It then found the working approach
at 0 and 53 characters, abandoned it, climbed back to roughly 6,100 and repeated that size seven
more times.

Neither run lacked the knowledge to write the program; run 2 reached 6 of 7. What separated them
was whether the model noticed a strategy was failing and abandoned it. The metric is the response
to failure: does the next attempt shrink, or escalate and repeat. It is countable from any session
log and does not require judging the code.

## Harness cost at identical score

The same 12B, the same 6 of 7, two apparatuses:

| apparatus | wall clock | tokens |
| --- | --- | --- |
| agentic harness | 63 min | 1,040,460 in / 76,510 out |
| constrained loop, 6/7 at turn 2 | 4.1 min | 29,614 total across all four turns |

Loop turn timestamps: arm started 00:31:11, turn 1 scored 00:33:11, turn 2 at 00:35:16, turn 3 at
00:36:57, turn 4 at 00:38:35.

The agentic harness spends its budget exploring: tool loops, file reads, re-reading context it
already holds. This task hands its complete input over at the start, so that exploration buys
nothing. The constrained loop also has no exact-match edit operation, which is why the same
whole-file strategy that thrashes under the agentic harness succeeds immediately in the loop.

## The repair loop, two feedback conditions

Identical model, task, quantization and turn count. The only variable is what the feedback window
carried:

| feedback carried | trajectory |
| --- | --- |
| failure counts only | 1, 1, 1, 1 |
| the validator's remediation text | 1, **6**, 3, 1 |

The first condition was a harness defect: the driver fed `out[-3000:]`, the tail of the conformance
run, and Schemathesis puts each failure's remediation inline while the trailing SUMMARY carries
counts only. So the worker was told it failed and never told how, three times, and did not move.
Underneath that the driver was reading the wrong file entirely: the scoring script prints a
PASS/FAIL scoreboard and the detail lives in `schemathesis.stdout` inside the run's evidence
directory. The remediation was produced correctly and discarded twice before reaching the model.

Fixed by selecting failure sections, deduplicating repeats and keeping the summary. The window went
from a 3,000-character tail carrying no remediation to 1,672 characters carrying both distinct
failures.

Turn 2 then closed five of six failures in one turn. Turn 3 overcorrected, flipping `API accepted
schema-violating request` into `API rejected schema-compliant request`. Turn 4 rewrote the working
405 handler as:

```js
const allowedMethods = ['GET','POST','PUT','DELETE','PATCH','OPTIONS','HEAD'];
if (!allowedMethods.includes(req.method)) { /* 405 with Allow */ }
next();
```

That is a list of methods that exist in HTTP, not methods this path serves. Probed directly:
`GET /health` 200, `TRACE /health` 405, `POST /health` 404 where the task requires 405. The code
sets the right header on the right status for the wrong set.

A second harness defect contributed: the driver appended only `text[:2000]` of each worker turn to
the conversation. Turn 2 emitted 8,375 characters and the word `Allow` appears zero times in the
first 2,000, so from turn 3 the worker could not see its own working implementation. It regenerated
an 8,000-character file from partial memory every turn. Fixed by feeding back the actual on-disk
artifact, which is what the gate tested.

## One header was worth five of seven operations

The 12B's first constrained-loop attempt scored 1 of 7. Five of its six failures were the same
defect: `TRACE returned 405 without required Allow header`. The model had read the task, which
states the 405 requirement, and written its own handler commented `// Handle 405 Method Not Allowed
for existing paths`. It returned 405. It omitted the `Allow` header, which the task does not name,
and Schemathesis checks that header on every operation.

Patching only that header into the model's own handler, changing nothing else, took the same
implementation from 1 of 7 to 6 of 7. The counterfactual is not a score and does not enter the
ladder; it answers one question, whether the gap was capability or a single missed line. Here it
was a single missed line, and the model closed it itself one turn later once the feedback carried
the remediation.

Consequence for reporting: per-operation scoring is not linear in correctness. A cross-cutting
requirement multiplies rather than partitions, and two implementations at 1 of 7 can be a stub and
a near-complete server.

## Three defects, not one (corrected after publication)

Gemma 4 E4B, served at its native 131,072 with the correct parser, drove the harness to a clean
exit in 8 minutes and scored 1 of 7. It wrote a 9,410-byte `server.js`, longer than either arm that
scored 7 of 7, correctly structured, with a `loadCorpus()` that reads the fixture and builds
in-memory indexes.

It carried three independent defects. The first is one character:

```js
const lines = data.trim().split('\\n');
```

In JavaScript `'\\n'` is the two-character string backslash-n, not a newline. JSON encodes newlines
inside strings as exactly that sequence, so the split shredded 438 clean records at every escaped
line break and fed markdown fragments to `JSON.parse`. Verified at byte level with `cat -A`,
against a fixture confirmed as 438 parseable lines and zero unparseable. The 12B wrote `split('\n')`
and read the same file correctly.

The other two defects are a missing `Allow` header on unsupported methods, the same miss the 12B
made, and two response-schema violations. Patching only the escaping error, verified as a one-line
diff with `node --check` clean, cleared the "3 operations repeatedly returned 404" warning and left
the score at **1 of 7**, because the other two defects were already failing the same endpoints.

This entry originally attributed all six lost operations to the single character. That was inferred
from one diagnosed defect without running the counterfactual, which is the same test this lab did
run for the 12B. Corrected after publication, and the corrected version is less flattering to the
model: the E4B is not one edit from working.

The clean case of doing the work and failing the gate is the 12B, not the E4B. One missing header,
1 of 7, and 6 of 7 the moment that header is patched into the model's own handler.

None of these models produced gibberish. What descends the ladder is the density of small precision
errors, and the consequence of each is wildly disproportionate to its size.

## The loop-control arm, and why it did not answer its question

Configuration: E4B worker in the agentic harness with `Edit` removed from its toolset, gpt-5.4-mini
as controller reading the gate output and the worker's source, forbidden from writing code or using
code blocks, ratchet on, four turns.

The controller performed correctly. Roughly 1,000 characters of guidance per turn, zero code
fences, and accurate diagnoses: response shape wrong (`items` versus `data`), `/health` missing
`chunks`, a handler referencing a map that was out of scope. Its footprint is 418 tokens in and
270 out per turn, about nine tenths of a cent.

The worker stopped calling tools after turn 1.

| turn | tools | chat text | wrote a file |
| --- | --- | --- | --- |
| 1 | Bash x5 | 6,841 chars | yes, 7,910 B, scored 1/7 |
| 2 | none | 14,460 chars | no |
| 3 | Bash x1, Read x1 | 17,078 chars | no |
| 4 | n/a | n/a | no |

`server.js` kept its turn-1 mtime throughout, so the gate re-scored the same artifact three times
and returned an identical failure profile each time. That reads as "cannot act on guidance" and is
not what happened.

Recovering the code from turn 2's chat and standing it up separately: a 10,896-byte server that
parses, starts, and loads the corpus correctly at 438 chunks and 258 documents. It registers no
routes at all. Every endpoint in the contract returns a well-formed JSON 404.

So the finding that survives is narrow. A follow-up prompt containing review feedback knocked a 4B
out of tool-use mode, and its second attempt was worse than its first. Whether a 4B can be steered
given a delivery mechanism that does not depend on it calling a write tool is open. The driver now
fingerprints the source files before and after each turn and records a `NO_WRITE` turn distinctly,
which it did not when this arm ran.

## Economics

Per-attempt verified yield at the 7 of 7 bar:

| rung | yield | mean cost per attempt | mean wall clock |
| --- | --- | --- | --- |
| gpt-5.4-mini | 2/2 | $2.08 | 5.6 min |
| Gemma 4 26B-A4B | 2/3 | zero marginal | 37.5 min |
| Gemma 4 12B QAT | 0/1 | zero marginal | 63 min |

Cost per verified unit: about $2.08 and six minutes for the hosted tier, zero marginal dollars and
roughly 56 expected minutes for the local 26B, because at 67% yield three attempts bank two units.

Variance shows up as consumption, not only as score. The 26B's failing run burned 2,277,297 input
tokens against its passing run's 460,609: five times the work for a worse result. A variance-prone
worker does not fail fast, it thrashes, and a capacity model that assumes a failed attempt costs
what a successful one costs is wrong in the expensive direction.

The gate is what makes a retry strategy purchasable. The 26B's 6 of 7 passed 333 of 333 stateful
scenarios and reads as a working API. Without the validator nobody retries it.

The cost that appears in no accounting is the engineering. Twelve venue defects in one program, each
producing a confident wrong reading. At $2.08 per verified unit the hosted tier runs about a
thousand times before its spend matches one engineer-day of making the local venue trustworthy.

## The twelve harness defects

Every one produced a confident wrong reading, and every one pointed the same direction: at the
model and away from the harness. Run unattended, this descent would have reported a capability
floor near 80B with every rung below it wrong, and each wrong rung would have looked like a clean
result rather than an error.

1. **Output reservation, 32,000 default.** Exceeded a small model's context before it saw the task.
   Presents as DID_NOT_BUILD. Pinned to 8192.
2. **Context served below native.** Qwen3-4B served at 32,768 against a native 262,144. A small
   window presented as a model property when it was a harness choice.
3. **Loop output budget, 8,192.** A reasoning model spent every turn inside its thinking and emitted
   no code. Reads as "cannot follow the output contract".
4. **Arm timeout, 1,800s against an 11 tok/s rung.** An arm needing 45 minutes killed at 30. Reads
   as a hang.
5. **Output contract required naming JSON files with a comment.** The system prompt gave
   `// package.json` as its example. npm rejected every turn with `EJSONPARSE`. The model followed
   the instruction exactly. On a descent, an instruction the model obeys precisely is more dangerous
   than one it ignores.
6. **Feedback window took a tail.** `out[-3000:]` delivered the counts-only summary and dropped the
   remediation text.
7. **Driver read the scoreboard, not the gate.** The detailed output lives in a separate file the
   driver never opened.
8. **Context truncated to 2,000 characters.** Removed the worker's own working implementation from
   its view.
9. **Tool-call parser hardcoded to `hermes`.** The only defect on this list that fails silently:
   clean HTTP 200s, no tool calls, no error. Cost a full 0/12 arm on Qwen3-Coder. Now an argument,
   gated by a smoke test before every arm. Gemma 4 needs `gemma4`.
10. **The fix for defect 1 broke the top of the ladder.** `CLAUDE_CODE_MAX_OUTPUT_TOKENS=8192`,
    correct for small rungs, failed a large one mid-write with `response exceeded the 8192 output
    token maximum`. The rule that a rung's parameters derive from the model applies upward as well
    as downward.
11. **Controller returned an empty string silently.** Would have fed the worker a blank instruction
    and read as the architecture failing. Now raises.
12. **Loop scored a stale artifact.** No check that a turn wrote a file, so three identical scores
    read as a capability plateau.

Two adjacent notes from the same program, kept because they cost real time. A scoring script named
`bisect.py` shadowed the standard library module that `urllib.request` imports. And a `pkill -f`
pattern matched the remote shell's own command string and killed the session running it, twice.

## What is not here

The corpus content and the assessment methodology it encodes. The reference implementation. The
task document's full text. The method, the contract shape, the scoring design, every number above
and the entire defect chronology ship; the material that makes the corpus worth querying does not.