# Lab 013 raw lab detail: The floor is trained, not sized Working notes behind the published entry. Every substrate detail ships here: serving configs, the numbers, the gotchas, and the measurement errors that were caught and corrected on the way. The assessment methodology, the vendor grades this evidence feeds, and the judgment that turns a score into a placement stay proprietary. Returns, not algorithms. ## The one question Lab 012 bracketed the instrument-literacy bar and never located it: Gemma 4 31B failed it, a hosted mini cleared it, and everything between was guesswork. For anyone who already owns hardware, that gap is the whole decision. This lab holds the harness and the gate fixed and walks six open-weight models down onto one box. ## The instrument The 22 repairs Lab 011's free local loop first missed, each certified by a reproduce-or-drop gate that runs inside the scoring container. Eighteen arms across two apparatuses: the constrained edit loop and a headless Claude Code session driven through a LiteLLM translation proxy. Three arms replicated, plus a five-task residual set run twice more. All local arms on one NVIDIA DGX Spark, 128GB unified memory, GB10, served through vLLM 0.25.1. Serving config is recorded per result: checkpoint path, utilization, and the key-value cache the server reported at startup. ## The measured spine Local harness arms, one box, zero metered spend: | model | solved | residual 5 | median | max | total | | --- | --- | --- | --- | --- | --- | | Qwen3-Coder-Next NVFP4 | 18/22 | 3 | 22.0m | 108.5m | 11.4h | | Devstral Small 2 24B | 15/22 | 2 | 29.7m | 120.0m | 20.3h | | Qwen general | 15/22 | 2 | 33.5m | 120.0m | 15.6h | | Gemma 4 26B NVFP4 | 14/22 | 2 | 61.8m | 120.0m | 24.6h | | GLM-4.7-Flash BF16 | 13/22 | 4 | 39.0m | 91.5m | 16.7h | | GLM-4.5-Air AWQ-4bit (106B) | 12/22 | 1 | 9.2m | 59.8m | 4.5h | | GLM-4.7-Flash, replicate | 11/22 | 1 | 31.1m | 103.7m | 13.3h | | Gemma 4 31B NVFP4 (rented L40S) | 11/22 | 1 | | | | Constrained loop, same tasks and gate: Gemma 4 31B 17/22, a Qwen base build 14/22, Qwen3-Coder 10/22, GLM-4.7-Flash 4/22. Hosted reference from Lab 012, for the wall-clock comparison: gpt-5.4-mini 22/22 in 38.4 minutes total at $3.14, median 1.67m per task. **The inversion.** Gemma 17 in the loop and 11 in the harness; Flash 4 and 13. Same tasks, same gate, opposite ordering. The loop demands byte-exact anchor reproduction from a once-read prompt; the harness demands instrument operation. Separable, trained-in capabilities. ## Serving configuration, and the venue-qualification rule An agentic harness needs >= 131K serving context and the key-value cache to hold it. On unified memory `--gpu-memory-utilization` budgets model plus KV, not total footprint, so lowering it starves KV without reducing overhead. Measured on the same model and box: | | util 0.60, default max-num-seqs | util 0.85, max-num-seqs 4 | | --- | --- | --- | | KV cache | 13.15 GiB | 44.41 GiB | | Max concurrency @131K | ~1 | 6.72x | | Host free after load | 3G (wedged) | 8G | `--max-num-seqs` unset lets vLLM capture CUDA graphs for batch sizes up to 512 on hardware the vendor documents as small-batch. That cost ~31 GiB of KV for nothing. Set it to 1-4 on GB10. **GLM-4.5-Air was NOT disqualified. That claim was published and is wrong.** The original reading: 60GB of weights fit the box, at util 0.65 there was no room left for sessions, and at 32K context it served perfectly while every harness session returned a context-window error on its opening request. All of that happened. The conclusion drawn from it did not follow. Re-measured at the corrected serving config (util 0.80, `--max-num-seqs 4`), same weights, same box: ``` Available KV cache memory: 36.56 GiB GPU KV cache size: 208,368 tokens (184 KB/token) Maximum concurrency @131K: 1.59x host free with Air resident: 13G ``` Air serves, passes a 54,017-token smoke, and clears **12 of 22** at a 9.2-minute median. At util 0.70 it misses by 0.55 GiB and vLLM says so precisely ("23.0 GiB needed, 22.45 available, estimated maximum model length 127952"), which is a tuning gap, not a disqualification. Qualification arithmetic is still the right discipline. It is only as good as the config it is computed against, and a default `--max-num-seqs` retired a working 106B model here. Quantization on GB10 is a wash for dense models. Gemma 4 31B measured 6.9 / 7.1 / 6.9 tok/s in NVFP4 and 6.8 / 6.9 / 6.8 in FP8. FP8 runs near the 273 GB/s memory-bandwidth roofline; NVFP4 moves half the bytes and gives the advantage back to a kernel SM121 cannot run natively. Dense 31B on this box is ~7 tok/s either way. ## Findings that died in measurement, kept deliberately - **A KV-per-token table computed from a formula was wrong by 19x.** Gemma 4 31B measures 50.8 KB/token (51.99 GiB / 1,073,273 tokens, 8.19 concurrent full-context sessions). The derived figure said 960. Read `Available KV cache memory` from the server; never compute it. - **A token total was wrong by 122x.** An arm reported 78,879 tokens; the real figure is 9,658,345. The `input_tokens` field excludes `cache_read_input_tokens` and `cache_creation_input_tokens`, and on that path 99% of traffic is cache. - **A quantization recommendation came from the wrong model class.** The benchmark behind it ran a mixture-of-experts model with 3B active parameters, and its authors warned it would not generalize to dense transformers. Dense decode is bandwidth-bound; MoE decode is not. - **A "best at the hard end" distinction did not survive replication.** One arm took 4 of the 5 residual tasks on its first run. Two further runs returned 1 and 1. ## The residual set is measuring noise per task Five observations of the same five tasks, across two models: | task | run counts | | --- | --- | | task A | 5 of 5 | | task B | 2 of 5 | | task C | 2 of 5 | | task D | 1 of 5 | | task E | 2 of 5 | Per-arm totals: 3, 3, 4, 1, 1. One model scored 3 of 5 twice on almost disjoint task sets, overlapping only on the task every arm solves. Aggregate counts carry signal. Task identity does not. A bar written as "solved task X" is reading a coin flip. ## Operational record, mistakes included - A KV-starvation arm was run on the same box with KV as the only variable, 52.29 GiB against 8.52 GiB. Solve rate went 5 of 11 to 0 of 11 (Fisher's exact p ~= 0.035). The mechanism is wall clock, not tool failure: aggregate generation throughput fell from 12.2 tok/s to 0.2 at matched load, and sessions ran out of a two-hour ceiling. That work moved to its own lab. - A second agentic harness was wired behind an environment switch to test whether the collapse generalized. Four wiring faults surfaced first, one of which produced a clean-looking false result: the harness resolves its own project root, ignored the subprocess working directory, ran against the wrong tree, and scored a plausible failure verdict. The fix is an explicit directory flag, recorded in the driver with the date and symptom. - A 140KB prompt exceeded the Linux single-argument limit when passed positionally. Small smoke prompts passed and real tasks died before the session started. - Arms run before a provenance fix record only a proxy alias, so one arm in the record cannot name the weights that produced it. Results now capture the served checkpoint path and the serving config verbatim. - Two serving-config guards exist because their absence produced bad data: assert the server's utilization matches the phase before running it, and abort a phase that writes zero results instead of proceeding. ## Spend Zero metered spend on every local arm. The cost is roughly 102 hours of owned GPU time and a power bill nobody metered. The hosted figures quoted for comparison were measured in Lab 012. ## What this hands the next labs - **The localization question.** Every task here arrived with its files named. Whether the leading local model's advantage is code-reading skill or an artifact of knowing where to look is the next lab's first arm, and it reuses this certification with no new mining. - **The KV envelope.** Starved KV is a wall-clock multiplier, not a capability cliff, and it kills whatever was already near its timeout. Generalizing that needs a dose-response sweep, a concurrency cross, and a prediction made before a run rather than after. - **The tier above one box.** A model too large for 128GB is outside this lab's scope by its own scope line. That needs a second box and an interconnect measured at spec.