# Lab 9 — Raw lab detail: Recovered capacity is real, and it fails honest

Layer2C Labs, editorial, self-funded. Published 2026-07-16. Substrate engineering behind
the ruling at labs.layer2c.com/labs/xeon-rematch.

**Publish boundary.** The instrument's method, source repos, audit design, and every
rate, count, latency, and cost ship. Task fixture contents, reference solutions, and
hidden test contents stay private (the tasks derive from public library history; the
curated harness is the practice's instrument). **Disclosure:** Intel is a client of this
practice; the lab corresponded with Intel about serving configurations during this
program (paraphrased where referenced); no vendor commissioned, funded, or previewed
anything.

## The one question

Labs 3 and 7 already owned the throughput and loop-behavior numbers. The unmeasured cell:
on the exact Lab 3 problem set, through the bit-identical Lab 3 validator and escalation
policy, does a smaller model hold the success rate — and does it fail honest? Everything
else is arithmetic on owned numbers. 585 scored attempts later: yes, and yes.

## The instrument

The Lab 3 Loop Control harness, unmodified: 39 coding tasks (task-c001..c008,
task-r001..r031), most reconstructed from real bug fixes in real Python libraries —
httpx, requests, dateutil, more-itertools, arrow, click, h2, pyjwt, prettytable — each
pinned to repo and commit, presented with the historical bug in place. Layered
detection: feature tests (did you fix it), regression tests (did you break something
else), hidden reference tests the model never sees (did you game the visible tests),
and a JSON/static gate (malformed output dies before code runs — fail-closed by
construction).

**Validator pin:** LC_ALL=C-sorted sha256sum manifest over validator source, the task
prompt, and all task data (caches excluded), hashed once more:
`f22a7a2ff0a7475c5def045ed5477ac880e9f3d51a10861101a46c858f03c828` — verified identical
across the original instrument, the staged copy, and both cloud instances, plus a
functional census (buggy-fixture scan) reproduced exactly on every host. Lesson learned
the hard way: an earlier pin value was recorded without its exact recipe and could not
be reproduced (12 reconstructions failed); the chain was re-anchored directly on the
original instrument, which is stronger. Record your hash recipes as scripts.

**Instrument census findings (found by auditing our own instrument, carried
like-for-like in every comparison):** one weak feature test that passes on buggy code
(r003 — never trusted on validator evidence alone); six structurally unpassable tasks
with empty test files (r016/r018/r019/r024/r026/r030); two tasks with no hidden
references (r028/r029); three tasks whose hidden references disagree with the validator
symmetrically for every model including the Lab 3 baseline (r022/r025/r031 — an
instrument property, proven by running the audit against the baseline's own pass
workspaces). Repair handed to lab 10.

## The runs

| Run | Model | Substrate | Protocol | Task passes | False passes | Notes |
|---|---|---|---|---|---|---|
| Lab 3 ref | 26B-A4B MoE (ollama) | Spark GB10 | 90 stored runs, 29 tasks | any-pass 17/29, unanimous 13/29 | — | 4 flip tasks at temp 0 |
| Run A | 12B dense bf16 (vLLM) | Spark GB10 | 39×3 = 117 | 20/39 (0 flips) | 0 | 5h35m, $0 |
| Leg 1 | 12B dense bf16 (vLLM CPU nightly) | m8i.12xlarge GNR | 39×3 = 117 | 21/39 (1 flip: c007, audit-clean) | 0 | 5.54h; quality SURVIVES substrate |
| Leg 2 | 26B-A4B MoE Q8_0 (llama.cpp) | m8i.12xlarge GNR | 39×3 = 117 | 21/39 (0 flips) | 0 | 2.24h — same yield, 2.5x throughput |
| Ladder Q8_0 | 12B (llama.cpp) | Spark | 39 screen | 21/39 | 0 | HOLDS |
| Ladder Q4_K_M | 12B | Spark | 39 screen | 21/39, pass set bit-identical to Q8 | 0 | HOLDS — the boundary |
| Ladder Q3_K_M | 12B | Spark | 39×3 = 117 | 16/39 ×3, bit-identical | 0 | BREAK, triple-confirmed, fail-closed |
| Ladder E4B | E4B bf16 (vLLM, Run A's stack) | Spark | 39 screen | 15/39 | 0 | BREAK below the boundary, single screen |
| Canceled | 26B-A4B Q4_K_M; E2B | — | staged, unrun | — | — | deliberately not run; stated, not asserted |

**585 scored attempts, zero confirmed false passes.** Every validator pass re-run
against hidden references on the stored workspace. The closed-to-open flip was never
observed at any tier, any model, any substrate.

## The boundaries (this environment's coordinates, not universal constants)

- **Quant floor: Q4_K_M.** Identical 21-task pass set as Q8_0 and bf16-equivalent
  yield; decode rose 15.3 → 21.5 tok/s median (Spark, llama.cpp). On this instrument
  there is no reason to serve the 12B above Q4_K_M.
- **Quant break: Q3_K_M.** Net -5 tasks, triple-confirmed with bit-identical pass sets
  and status mixes (16/19/2/2 ×3). Texture: failure-to-fix (fails and errors, little
  collateral).
- **Size floor: 12B. Size break: E4B** (15/39 on Run A's exact serving stack — no
  confound). Texture: destructive — 7 regression verdicts, 5 of them real
  broken-worse-than-found edits. The validator caught every one.
- Degradation moves verdicts BOTH directions at temp 0: broken tiers gained tasks
  (r004/r005, +r008 at Q3, c007 everywhere below bf16-Spark) while losing more. The net
  is what breaks; per-task flips are noise you must expect.

## The metric ruling (generalizes beyond this lab)

Per-attempt verified yield is the economic statistic (attempt = cost, verified pass =
yield), numerators visible and bases stated: full-set — 12B Spark 60/117 (51.3%), 12B
Xeon 62/117 (53.0%), 26B-Q8 63/117 (53.8%); Lab 3 baseline 43/90 (47.8%) on its 29-task
coverage. Overlap-restricted (24 shared tasks): baseline 43/75 (57.3%) vs 12B 39/72
(54.2%) — overlap and full-set figures are different bases and must not be mixed in one
sentence. Model determinism is an operational
property that sets retry policy, not a quality axis — in DCITL the deterministic
component is the validator, and the Lab 3 baseline's verdict flicker at temp 0 is an
argument FOR the architecture (a variance-prone worker behind a deterministic gate is
safe, and its idle-cycle retries mine verified passes). Statistical hygiene: never
compare best-of-n against single-run across different variance profiles — the
26B-vs-baseline "deficit" reads 17 vs 12 on any-pass and 13 vs 12 on unanimous-pass.
Substrate note: at temp 0, the same weights emitted different tokens on different
silicon (Spark JSON-error tasks became parseable honest fails on x86); task-level
verdicts are what transfer, determinism is a property of a model-substrate pair.

## The economics

- Rented anchor (date-stamped 2026-07-15, us-east-1): m8i.12xlarge on-demand
  $2.54016/hr. **As measured (EC2 API, us-east-1, that date), zero classic Reserved
  Instance offerings existed for m8i.12xlarge** — the reserved analog is an EC2
  Instance Savings Plan (1yr no-upfront $1.68032/hr; 3yr $1.15222). Category note: a
  Savings Plan commits DOLLARS, not capacity; the recovered-value math prices unused
  spend commitment, which is a different object from reserved or owned hardware.
- Leg 2: 117 attempts, 63 verified fixes, 2.24h ≈ **$0.09/verified fix on-demand,
  ~$0.06 at the SP rate.** On owned idle hardware, marginal cost approaches incremental
  power, cooling, and operations; inside an otherwise-unused Savings Plan commitment,
  the incremental bill can approach zero (subject to commitment coverage). Zero paid
  tokens in the loop by construction (frontier keys stripped; escalation routes to a
  human queue).
- Throughput anchors (24 GNR cores, SMT off): 12B bf16 — 6.25 tok/s c1, prefill 864
  tok/s (vLLM), c16 aggregate 82.9. 26B-A4B Q8_0 — 24.7 tok/s c1, c16 aggregate 113.3
  (llama.cpp). Decode is memory-bandwidth-bound; sparse activation, not parameter
  count, is the fit variable for decode-dominated batch.
- AMX engagement proven by hardware counters on AWS (exe.amx_busy: 19.9B cycles/10s
  under c16 12B load; 428M cycles/3.2s on the 26B decode bench).

## Vendor-blessed config (measured for scope)

Intel pointed to the OpenVINO backend for llama.cpp (correspondence, paraphrased).
Measured, 38 minutes of a 2-hour time box: dense 12B bf16 pp1024 477.6 tok/s (4.7x the
native build's 100.9) and tg128 5.78 (16% BELOW native's 6.87 — decode is
bandwidth-bound and no backend manufactures DRAM bandwidth). The 26B MoE does not run:
repeatable GGML_ASSERT crash in CPU-fallback expert routing after graph partitioning.
Native llama.cpp remains the only stack measured that both runs the MoE and engages AMX.
Build friction, our words: a CPU-only build still required OpenCL dev packages; one
failed-configure round trip, ~20 minutes.

## Operational record (mistakes included)

- Runbook said the instrument lived at ~/bench; it lived at ~/loopcontrol. Verify
  locations, not documentation.
- llama-server's chat template opens a thinking channel that vLLM's guided decoding
  suppresses; caught at smoke on two separate hosts, disabled (--reasoning-budget 0),
  two contaminated traces purged before any protocol ran.
- AMI staging shortcut: snapshot-born EBS lazy-hydrates at S3 rates (~5-9MB/s); re-pull
  large artifacts to fresh blocks (27GB GGUF: 2 min) instead of hydrating (50+ min).
- ModifyInstanceCpuOptions now exists: SMT-off survives an EC2 stage-small-then-resize.
- Zero quota or capacity friction across four AWS launches (16-29s to SSH), including
  two 48-vCPU instances in one AZ in one hour. The GCP-style quota theater (lab 7) did
  not occur. Watchdogs armed from first boot on every instance; zero triggers.
- **Cleanup error, disclosed:** the final verify-empty script deleted one pre-existing
  unattached volume (10GB, untagged, created 2017, no snapshots) that was not a lab
  resource — it conflated verify-empty with make-empty, and the volume is
  unrecoverable. Rule adopted: lab resources are tagged at creation, teardown filters
  on the tag exclusively, verification is read-only.

## Spend

Cloud: **$28.32** (staging $0.07, instance 1 $16.75, instance 2 $10.93, EBS/snapshot/
egress ~$0.57), instances terminated and verified. Spark legs and the entire ladder:
**$0**, 234 traces. Rates date-stamped in-line above.

## What this hands lab 10

Second-substrate generalization (TPU/APU/GPU reservation, same instrument), heterogeneous
batches and non-executable validators, retry-mining yield curves on variance-prone
workers, and instrument repair (the weak test, the hidden-reference gaps, the empty-test
sextet, the r022/r025/r031 disagreement trio).