# Layer2C Labs: Complete Lab Data
> The Advisor Bench LLC · Updated 2026-09-06

# Borrow the vendor’s plumbing, not its judgment

> Lab 001 · Editorial lab · Status: published · Author: Keith Townsend · June 27, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/spark-s3vectors · Structured: https://labs.layer2c.com/labs/spark-s3vectors.json

**Finding:** The managed data plane is cheap and fast; the cost it hides is the chunking judgment it takes from you. Borrow the plumbing, keep your own chunking.

**Question:** I built a retrieval pipeline across a public-cloud data plane and a local box, the compute I keep below the cloud’s managed abstraction, to map where authority actually sits across the 4+1 stack. The economics were the boring part: eighty-four cents, the cloud faster. The finding worth keeping is what the managed path quietly decides for you, and the two questions the lab now knows to ask.

**Load:** ~237K transcript segments across 5,815 videos, 2013 to 2026, embedded and queried in batch.

**Video:** https://youtu.be/SF1B3C9FASE
**Raw lab detail:** https://labs.layer2c.com/downloads/spark-s3vectors-lab-detail.md
**Vendors:** AWS (cloud), NVIDIA (hw)
**Layers:** Layer 0 (Compute), Layer 1A (Storage), Layer 1B (Retrieval), Layer 2B (Runtime), Layer 2C (Reasoning)
**Themes:** managed-abstraction-cost, designed-partition, authority-placement, ai-factory-economics

## Verdict

**Scope:** The cost verdict is commodity knowledge, and it sits below as evidence. These are the rulings that change how the stack gets scored: where authority sits, and what you cede without noticing.

- **DON'T:** cede the chunking. The managed RAG wrapper re-chunks on its own strategy and turns on parsing you did not ask for, which would have broken label-filtered retrieval. That decision determines what you can retrieve, and it is yours.
- **DO:** borrow the vendor’s plumbing. AWS’s strength is the composable primitives underneath the managed wrapper, not the wrapper. The wrapper is convenience; the primitives are the capability. Score AWS on the primitives.
- **DO:** own Layer 2C as the control structure. The reasoning plane that yields a true answer is the gates, the judges, and the reproducibility around the model, not the weights. Cede the model and you still own the harder half.
- **DO:** place the box by its authority, not its benchmarks. The Spark keeps real local execution authority; its place is development and portability, not commodity inference, where the cloud wins on cost and speed.

**Independence and disclosure:** Self-funded. No vendor paid for this answer.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Corpus | ~237K segments · 5,815 videos | 2013 to 2026; ran the full corpus, not a sample |
| Ingest throughput | 337K segments/hour | full corpus indexed in ~41 min, 0 failures |
| Retrieval latency (from Spark) | p50 280ms · p95 505ms | network-bound; S3 Vectors service ~100ms |
| Generation, local (Gemma 4 26B-A4B) | ~20-23 tok/s | MoE; beats a dense 8B on speed and capability |
| Same model on Bedrock | 136-214 tok/s | 7 to 9x faster than local |
| Setup cost (embed + index) | $0.84 one-time | plus ~$0.07/mo to keep the index |
| Inference cost (full analysis pass) | ~$0.20 | the entire 13-year generation run, at cloud rates |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 0 (Compute) | Retained | Compute stays on the Spark. |
| Layer 1A (Storage) | Ceded | AWS owns the data plane: Titan embeddings into S3 Vectors. |
| Layer 1B (Retrieval) | Ceded | Embedding and retrieval are managed by AWS, so Layer 1 stays whole on one vendor. |
| Layer 2B (Runtime) | Retained | Serving stays on the Spark as built, though cloud runs the same model faster and cheaper. |
| Layer 2C (Reasoning) | Retained | Reasoning stays on the Spark. |

## What this does not establish

- It did not prove on-prem inference never pays. It proved commodity base-model generation, on this box, for this workload, loses to the cloud on cost and speed.
- It did not settle custom or fine-tuned model economics. Lab two did, and reversed this verdict for custom models.
- It did not prove managed RAG is bad. It proved the wrapper can hide control points, chunking above all.
- It did not test capability against the frontier. An 8B-class model is a cost-structure vehicle, not a frontier contender.

## Open questions

- **What does a specialist data platform add over the hyperscaler’s native Layer 1?** The AWS-native data plane was cheap and did hybrid retrieval in one call. So where does VAST Data, or any performance or portable data platform, earn its keep over S3 Vectors, on latency, on portability, on retained governance? I did not consider that a question worth asking until the bench showed how good the native floor was. Now there is a measured floor to test it against.
- **When does a custom model change the economics?** The cost verdict was measured against commodity per-token pricing, which exists only for base models. The moment a workload needs a fine-tuned model, AWS stops selling tokens and charges a floor instead. Lab two ran that regime and reversed the verdict: against a floor, the owned box wins custom serving above roughly 12 to 20% utilization.

## Objection handled

**But what about utilization?**

The on-prem case is to keep the box busy so the capex pays off. I ran that math. Pegged at 100% for three years, the Spark generates roughly 1.9 billion tokens, which spreads the $4,699 to about $2.50 per million tokens, plus around $0.40 in power. Call it ~$2.90 per million, fully utilized. The same model on Bedrock is about $0.40 per million. So even maxed out for three years straight, the owned box costs roughly seven times more per commodity token.

Utilization is the wrong lever. The box is memory-bandwidth-bound at about 20 tokens a second, so it produces too few tokens an hour for the capex to ever spread thin enough to win. And to hold 70 to 90% you would need a constant firehose of batch work feeding a box that loses even when fed. You cannot util your way past a throughput ceiling.

## Assessments at the time of the lab

| Vendor | Layer | Grade | As assessed |
| --- | --- | --- | --- |
| AWS AI Infrastructure | Layer 1A · Storage | Delegated | June 29, 2026 |
| AWS AI Infrastructure | Layer 1B · Retrieval | Delegated | June 29, 2026 |
| NVIDIA AI Platform | Layer 0 · Compute | NVIDIA Strength — Silicon Authority | May 22, 2026 |
| NVIDIA AI Platform | Layer 2B · Runtime | NVIDIA Authority — Inference + Agent Runtime | May 22, 2026 |
| NVIDIA AI Platform | Layer 2C · Reasoning | Runtime Governance Only — Not a Reasoning Plane | May 22, 2026 |

## Summary

I paired the NVIDIA DGX Spark with Amazon S3 and S3 Vectors as the data plane, in the AWS-managed configuration. Validated on a 5,000-segment subset, then run across the full 13-year corpus, roughly 237K segments and 5,815 videos. The verdict is earned, not projected.

The data plane is cheap, fast, and better than expected. The full corpus embeds and indexes for about $0.84 one-time in roughly 41 minutes, and S3 Vectors turned out to do hybrid retrieval, semantic similarity plus metadata filtering in one call, which is what made label-scoped retrieval work. Retrieval was never the constraint. And the architecture itself held: the 4+1 seams fell where the model predicted, and the public-cloud-to-edge I/O across them was never the bottleneck.

## Detail

The clean architecture is the one where the vendor boundary respects the layer boundary. Embedding is a Layer 1B function, data preparation, even though it runs a model on a GPU. So the whole data plane goes to AWS, and NVIDIA keeps Layer 0 and Layer 2. Each vendor owns whole layers, authority places cleanly, nothing is split. Embedding locally would fracture Layer 1 for no gain, since the corpus is public.

A wrinkle at the bench taught the sharpest plumbing lesson. The fully managed path, Bedrock Knowledge Bases, would not create through the API even for an administrator account, so the same Titan-into-S3-Vectors substrate was self-orchestrated with direct calls. The real loss in the managed path was not orchestration, it was the chunking knob: managed re-chunks with its own strategy and turns on parsing you did not ask for. Bring-your-own kept one segment as one vector with its labels attached, and that is what made hybrid label-filtered retrieval possible. Borrow the vendor’s judgment on plumbing. Keep your own on chunking.

Then the numbers reordered the assumptions. The data plane is cheap and fast. Retrieval was never the problem; the latency seen from the Spark was its own network path, not S3 Vectors. The constraint is local generation.

The architecture itself passed, which is the result the economics can obscure. The 4+1 seams fell where the model said they would. Splitting the stack with the data plane in the public cloud and the reasoning at the edge is an operable partition, not just a diagram: the I/O across that seam, retrieved context down to the box and queries up, was never the bottleneck. The constraint lived inside Layer 2, in local generation throughput, not at the boundary between layers. The designed partition holds.

The box, by contrast, did real work, and that is the fair way to say it. It built and ran the whole pipeline end to end. What it isn’t, at this size, is a match for the public cloud on price for performance. The same model runs 7 to 9 times faster on Bedrock for less money, and the project’s entire local generation would have cost about twenty cents in the cloud. So the GB10 is not the commodity inference tier. But the box doing real work, and the architecture holding, are the results that travel. The same containers and serving stack are meant to lift unchanged to a bigger platform, so the honest close is not that the box failed. It is that this box, at this size, is not price-competitive against commodity per-token pricing, which exists only for base models. Lab two took the custom-model regime, where that price disappears and AWS charges a floor instead, and reversed the verdict: against a floor, the owned box wins. One model-class note worth keeping: a mixture-of-experts model with ~4B active parameters beat a dense 8B on both speed and capability on this hardware, because single-stream decode is memory-bandwidth-bound and active parameters are what move.

There is a second axis the throughput numbers hide. Faster is not better. The mixture-of-experts model won single-stream speed, but a dense Gemma 4 31B reasoned better over the same data, and stepping up to foundation models reasoned better still. That moves the economics from cost-per-token to value-per-answer: a more capable model costs more per token and can still be the cheaper choice in net, when its output carries more business value than the marginal cost. Cheap and fast minimizes the token bill. It does not maximize the worth of the answer. So the open question is not only how cheap the tokens are, but how much capability, measured in value, not throughput.

The transferable lesson runs past the substrate. Generating LLM findings is free and easy. Generating true ones is not. It takes a quantitative gate, cross-run reproducibility, and a strong judge, and judge strength dominates: a mid-tier judge will rubber-stamp a confidently wrong pattern that a strong one refutes. The fix is not the most expensive judge. A cheap strict judge plus one frontier judge, trusted where they agree, did the work. Most naive discoveries did not survive that gate.

## How it abstracts

The shape generalizes. A Layer 1 data service paired with a Layer 2 reasoning surface, testing where the seam falls. Swap the data plane or the reasoning surface and the motion repeats.

So does the verdict. At these corpus sizes, owned-edge inference rarely beats cloud on cost or speed, because cloud inference is already cheap. The case for the local box is air-gap, residency, or as a development instrument that ports to the cluster, not edge-inference economics. The workload lifts to any large-corpus, latency-insensitive task: legal discovery, support ticket mining, research-paper analysis, log and event mining.

## Method and disclosure

Self-funded, with no sponsor, so the lab is free to mix competitors. That cross-vendor mix is the editorial signature, nobody here is selling you one box. Every lab, editorial or sponsored, is held to the same method and the same editorial control; sponsored labs simply center on the sponsor’s architecture.

Validated on a 5,000-segment subset, then run across the full corpus plus a 2026 refresh. Generation used Gemma 4 26B-A4B via vLLM on the DGX Spark, chosen over a benchmarked Llama 3.1 8B and a dense Gemma 4 31B. The data plane used Bedrock Titan Text Embeddings v2 into a 1024-dimension cosine S3 Vectors index, with labels kept filterable for hybrid retrieval.

Substrate engineering and the corpus findings are captured as working lab notes, not shipped. The classification taxonomy, the tagged corpus, the trained judges, and the specific conclusions stay proprietary. What ships is the pattern and the substrate verdict, enough to recognize and abstract.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# Own the weights, or the platform owns you

> Lab 002 · Editorial lab · Status: published · Author: Keith Townsend · July 1, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/fine-tune-economics · Structured: https://labs.layer2c.com/labs/fine-tune-economics.json

**Finding:** For a custom model you own, cloud token pricing disappears and becomes a rented floor; against a floor the owned box wins, and the managed path takes the weights.

**Question:** Lab one found the Spark loses to the cloud on inference. That verdict held only for commodity base models. The moment you need a custom model you own, the cloud stops selling tokens and starts renting you floors, and the managed path takes something you cannot get back: the weights. Throughout, the box means the compute, serving, and training you keep below the platform’s abstraction instead of ceding them, and the NVIDIA DGX Spark is where this lab draws that line.

**Load:** A retrieval workload that needs a custom, fine-tuned model, served on an owned DGX Spark versus AWS custom-model products, with the lab-one AWS S3 Vectors data plane held constant.

**Raw lab detail:** https://labs.layer2c.com/downloads/fine-tune-economics-lab-detail.md
**Vendors:** AWS (cloud), NVIDIA (hw)
**Layers:** Layer 0 (Compute), Layer 2B (Runtime), Layer 2C (Reasoning)
**Themes:** model-ownership, fine-tuning-limits, ai-factory-economics

## Verdict

**Scope:** The economics are arithmetic once you know the floor, and they sit below as evidence. These are the rulings: where authority goes when you customize, and what each managed layer takes in exchange for convenience. One caveat, not a hedge: the box that wins here is a contained, air-cooled, plug-in system. This is not a case for on-prem at scale, where cooling, power, and facilities re-enter the math and this lab did not go.

- **DON'T:** native-fine-tune on Bedrock if you need the model back. It is a one-way door: serving is quota-walled, billed continuously with no scale-to-zero, and you cannot export the weights. Data goes in, no model comes out.
- **DO:** own the weights. A free local LoRA, thirty-five minutes on the box, took accepted output from 23% to 70%. Those weights are yours to serve anywhere, at a flat cost that beats the cloud’s custom floor above roughly 12 to 20% utilization.
- **DO:** pick the fastest model that clears the task ceiling, not the biggest. Capability saturates; speed is the axis. A 26B mixture-of-experts with ~4B active beats a dense 8B by 1.6x on the box, and the speed edge survives fine-tuning.
- **DON'T:** assume bigger hardware removes the constraint. It moves it, from throughput to availability. Scarce accelerators are relationship-gated: quota at zero, a 60-second soft-deny, thousand-dollar-a-day minimums, a support ticket to serve your own model.

**Independence and disclosure:** Self-funded. No vendor paid for this answer.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Base + RAG accepted output | 23% (Spark) · 47% (Bedrock) | grounding failures dominate; formatting was fine |
| Local LoRA fine-tune | 70% accepted (3x) | 35 min on the box, free; loss 0.34 → 0.22 |
| Serving speed (Spark) | Llama 8B 14 · Gemma MoE 22.4 tok/s | MoE 1.6x faster: active params, not total |
| Custom Model Import (cloud GPUs) | 95 tok/s | 6.8x the Spark; scale-to-zero, active-window billing |
| Serving cost, custom behavior | Spark ~$3.4–5.4 ÷ util · CMI $27.5 /1M tok | commodity base is ~$0.40/1M, but base-only |
| Spark-beats-cloud crossover | ~12–20% utilization | lower for the faster MoE; below it, CMI scale-to-zero wins |
| Whole-lab cost | ~$1–5 | local box $0-marginal; never secured the rented GPUs to spend on |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 0 (Compute) | Retained | Compute on the Spark. Scale it up and the constraint moves from throughput to availability. |
| Layer 2B (Runtime) | Retained | Serving kept local for control and flat cost. The cloud alternatives are Custom Model Import (ceded, scale-to-zero) and Provisioned Throughput (ceded, continuous, quota-walled). |
| Layer 2C (Reasoning) | Retained | The weights. A local fine-tune keeps them; a Bedrock native fine-tune cannot export them, so paying to shape the model still cedes it. |

## What this does not establish

- It did not measure on-prem at scale. The Spark is a contained best case; the owned-box win does not automatically survive the jump to air-cooled multi-GPU or liquid-cooled rack systems, which carry power, cooling, and facility costs this lab did not test.
- Tuned-Gemma quality is inferred from saturation, not judged. The fine-tune dependency cascade blocked the measurement.
- Bedrock native fine-tune quality was not measured. Serving was quota-walled behind a support case.
- Clean Custom Model Import cold-start latency was not isolated. The model warmed between attempts.
- Whether Custom Model Import definitively accepts this model class is unsettled. The supported-architecture list is a moving patchwork.
- Sample size is small, thirty questions on a two-judge gate. Good for go or no-go, not for fine quality claims.

## Open questions

- **Does a neocloud actually deliver access, or is it relationship-gated too?** The scale-up hit the same wall everywhere: quota at zero, reservations, support cases. A neocloud request placed to route around the hyperscalers was still cold thirty-six hours later. Whether GPU-native providers remove the availability constraint or just move it to a different commercial surface is the open question, and it is the Fourth Cloud question.
- **Does owned compute still win once you scale past the box?** The Spark is a contained best case: air-cooled, plug-in, cheap flat amortization, no facility cost. Scale on-prem and the math changes at every tier. An air-cooled multi-GPU server adds power, heat, and integration; a liquid-cooled rack adds facilities, capital, and specialized operations. Whether owned compute still beats the cloud floor at those tiers is unmeasured, and it is a lab in itself.

## Objection handled

**But the cloud is cheaper.**

It is, for base models. Commodity per-token pricing, about forty cents per million output tokens, exists only for a model you did not change. The moment you need custom behavior, that price disappears. The cloud will not sell you cheap tokens on weights you own. It rents you a floor instead.

The floors are the whole story. Custom Model Import bills active windows and scales to zero, about $27.50 per million tokens for this behavior. Native fine-tune serving is Provisioned Throughput, billed continuously. Against a floor, the owned box at a flat $160 to $275 a month wins custom serving above roughly 12 to 20% utilization, lower for the faster mixture-of-experts. Below that, the cloud’s scale-to-zero wins. It is a utilization line, not an hours line.

## Assessments at the time of the lab

| Vendor | Layer | Grade | As assessed |
| --- | --- | --- | --- |
| AWS AI Infrastructure | Layer 2B · Runtime | Delegated / Retained | June 29, 2026 |
| AWS AI Infrastructure | Layer 2C · Reasoning | Intelligence 2C: Delegated | Infra 2C: Implicit | June 29, 2026 |
| NVIDIA AI Platform | Layer 0 · Compute | NVIDIA Strength — Silicon Authority | May 22, 2026 |
| NVIDIA AI Platform | Layer 2B · Runtime | NVIDIA Authority — Inference + Agent Runtime | May 22, 2026 |
| NVIDIA AI Platform | Layer 2C · Reasoning | Runtime Governance Only — Not a Reasoning Plane | May 22, 2026 |

## Summary

Lab one left a question open. The Spark loses to commodity cloud inference, but does that hold when the workload needs a custom model? It does not. Commodity per-token pricing is a property of base models, not of the cloud. For a model you fine-tune, the cloud offers only floors, and against a floor the owned box changes the math.

The lab set out to move that line outward, lifting the stack to a GB300-class cluster, and pivoted twice. The access walls and a memory-axis analysis killed the scale-up, and the genuinely unanswered question turned out to be the economics of serving a custom model. The richest results came from the friction, not the compute: what the managed fine-tune keeps, what the newest model cannot be trained on, and how scarce accelerators are actually procured.

## Detail

The reversal is precise. Lab one’s verdict, that the box loses to the cloud on commodity inference, was measured against commodity per-token pricing, and that pricing is a property of base models, not of the cloud. Fine-tune the model and the token price is gone. What remains are floors: Provisioned Throughput billed by the hour, continuously, or Custom Model Import billed by the active window. Against a floor, a flat monthly box wins on cost above a low utilization line. The Spark never lost to the cloud. It lost to commodity tokens, and custom weights are exactly where commodity tokens do not exist.

Fine-tuning earned its place, which was not a given. Base model plus retrieval scored 23 to 47% accepted output on the frozen question set, and the failures were grounding, not formatting, so few-shot and constrained decoding would not have closed them. A free local LoRA, thirty-five minutes on the box, reached 70%, beating the base model served on Bedrock. The kill-criterion the lab was built to respect, if base plus retrieval clears the bar then do not fine-tune, did not fire. The custom model was worth building.

The managed fine-tune is a one-way door, and that is the authority finding. Bedrock native fine-tune has three compounding traps: serving is quota-walled, a no-commitment endpoint returns zero model units and a support case; it is billed continuously with no scale-to-zero; and the job output contains metrics only, with no way to download the weights. Data goes in and no model comes out. There is no train on the managed platform, serve on your own import, because the paths do not bridge. The real axis is not price. It is ownership of the weights, and the managed path keeps them.

Capability was not the differentiator. Quality saturates at the task ceiling, a 4B model ties a 14B on a bounded task, so a fine-tuned Gemma is not meaningfully more capable than a fine-tuned Llama for this work. Speed is the axis instead. A 26B mixture-of-experts with roughly 4B active parameters runs 1.6 times faster than a dense 8B on the box, because single-stream decode on bandwidth-bound hardware is set by active parameters, not total, and that advantage is independent of fine-tuning. The self-host optimization is the fastest model that clears the task ceiling, not the biggest.

The newest model exposed a moving dependency frontier. Fine-tuning the latest architecture hit a four-wall cascade: the released transformers library could not load it, one trainer broke on the next transformers major version, another needed a torchao past what the stable box could hold. The model is servable on the box and not fine-tunable on the stable stack without bleeding-edge tooling. The blocker is not the hardware. It is the abstraction lag, the managed and open frameworks trailing the newest weights.

The access reality is the constraint the whole industry runs into. Renting a GB300-class node was not a form, it was a relationship. GPU quota sat at zero across every project, a self-serve request approved in sixty seconds and delivered a soft-deny into the wrong lane, capacity blocks imposed a thousand-dollar one-day minimum, and on the hyperscaler enough memory and on-demand did not coexist. Real access came through account escalation. A neocloud request placed to route around it was still cold a day and a half later. Scarce accelerated capacity is procured through relationships, not forms.

So the box is not the production answer. It is the proxy that exposes the constraint. On the box, the constraint is token latency, and better hardware bends that curve. But it does not remove the tradeoff, it moves it: solve latency with a bigger system and the constraint becomes availability, the same wall the cloud exposes through quota, reservations, and support cases. The old fast, cheap, high quality, pick two no longer fits, because availability is now its own axis. The Layer2C version is fast, cheap, high-confidence, controlled, and available, pick the constraints you are willing to own. Every path has one. The lab shows the floor. The harder question is which constraint you choose to keep.

## Method and disclosure

Self-funded, no sponsor, free to mix competitors and to tell you not to buy something. The custom-serving economics compare AWS and NVIDIA paths against an owned box, and every path is held to the same measurement.

The quality gate was a frozen 30-question set judged by a cheap strict judge plus one frontier judge, trusted where they agree. Fine-tuning was a local LoRA on the DGX Spark; serving was measured locally with vLLM, on Bedrock Custom Model Import, and against Bedrock native fine-tune pricing. The data plane was held constant on the lab-one AWS S3 Vectors substrate.

The cost structure, the decision table, the access-friction facts, and the architecture pattern ship. The corpus, the taxonomy, the trained judges and adapters, and the specific analytical conclusions stay proprietary.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# The validator determines done, not the loop

> Lab 003 · Editorial lab · Status: published · Author: Keith Townsend · July 1, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/loop-control · Structured: https://labs.layer2c.com/labs/loop-control.json

**Finding:** About 5% of tasks benefit from same-tier repair; the control point is the deterministic validator plus escalation policy, not the loop.

**Question:** The pitch was that a local bug-fix agent needs a frontier tier to escalate to. I built the three-tier chain on a DGX Spark, gated it with a deterministic test harness, and metered every call. Then I audited the harness. Nine of its checks were invalid, and they had booked escalation events that were really unsolved cases sitting on broken tests. Corrected, the credit moves: the local model was clearing the solvable bug fixes on its own, and the frontier tier bought throughput, not correct answers. The test still determines done. That is the part that got more true.

**Load:** Real bug-fix tasks mined from open-source repositories by git archaeology, each with the fix’s own test suite as an unfalsifiable pass/fail gate. Local generation on a DGX Spark, escalation to frontier models only when the evaluator rejected.

**Raw lab detail:** https://labs.layer2c.com/downloads/loop-control-lab-detail.md
**Vendors:** NVIDIA (hw)
**Layers:** Layer 0 (Compute), Layer 2B (Runtime), Layer 2C (Reasoning)
**Themes:** loop-control, validator-authority, agentic-repair

## Verdict

**Scope:** Measured on deterministic coding tasks, where a test harness gives an unfalsifiable pass/fail. The rulings are about where the "determines done" authority sits in an agentic loop. The open edge is domains without an executable evaluator.

- **DO:** run the work on the small model and expect it to clear the solvable share. Code bug-fixing sits inside a lower-class model’s reach, not above it. Lab 9 put this same validator in front of a 12B model at Q4 and it held the pass set across 585 attempts with zero confirmed false passes. Reserve the capable tier for speed, not for judgment.
- **DO:** make a deterministic evaluator the authority. A test harness decides accept or reject, drives escalation, and supplies the error feedback. No model judges its own work or another model’s. That gate, not the model and not the loop, is what verifies which fixes actually cleared.
- **DON'T:** let the model decide when it is done. At temperature 0 the local model is deterministic: the same input gives the same output, so a repair loop retries its way to the identical wrong answer. Self-repair added 4x latency and zero accuracy.
- **DON'T:** review every output. Universal review degraded accuracy in every configuration tested. The reviewer promotes correct low-confidence calls to confident wrong ones. Gate review on a deterministic trigger, or skip it.
- **DON'T:** credit escalation with work the local model already did. A later audit of the harness found nine invalid checks that recorded escalation events for tasks that were actually unsolved on broken tests. Corrected, the frontier tier cleared little the local tier had not, and the honest headline is how much the small model did on its own.

**Independence and disclosure:** Self-funded. No vendor paid for this answer, and the whole run cost about $4.60.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Pass rate, local model only | 25% (2/8) | Gemma 4 26B-A4B on the DGX Spark, one attempt |
| Pass rate, local + repair loop | 25% (no gain) | temperature 0: same input, same output, same failure |
| Pass rate, tiered escalation (as first reported) | 75% (6/8) | Overstated. Nine invalid validator checks booked escalations for unsolved cases; corrected, the escalation gain largely dissolves and the local tier carried the solvable work. See the methodology note. |
| Small model clears the work (Lab 9) | 21 of 39 · 0 false passes | The same validator re-run at rigor: a 12B at Q4 and the 26B MoE cleared the solvable set with zero confirmed false passes across 585 attempts. Correctness lives in the gate. |
| Repair-loop sweet spot | ~5% of tasks | bimodal difficulty: 42% trivial, 54% too hard for one pass |
| Frontier second-attempt recovery (as first reported) | 67% of near-misses | Same correction applies; the figure is superseded by the validator audit. |
| Total run cost | ~$4.60 · 156 API calls | ~$0.16/task at a 25% local pass rate; pay for capability only on rejection |
| Local throughput, MoE FP8 vs dense | 48 vs 1 tok/s | FP8 native on the GB10; makes the free tier practical |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 0 (Compute) | Retained | Local first-attempt tier on the DGX Spark. A mixture-of-experts model in FP8 (48 tok/s vs 1 for a dense model) makes a free local attempt practical. |
| Layer 2B (Runtime) | Retained | The test harness and the escalation orchestrator are deterministic code you run, not a model. |
| Layer 2C (Reasoning) | Retained | The determines-done authority. It stays with the deterministic evaluator and the escalation policy. Frontier model capability is ceded and rented on demand, but the decision to escalate and the accept/reject verdict are never the model’s. |

## What this does not establish

- The task set is small. The culminating escalation ran on 8 tasks (26 calibrated). Good for the architecture verdict, not a benchmark of pass rates.
- Two of the failures were not capability findings. One task had a bad test that passes on buggy code; another was an edge-case blind spot shared across every model family.
- The third tier did not earn its place. The most-capable model solved nothing the middle tier failed, in this set. It is an architectural proof point, not a demonstrated advantage.
- Scale economics were not measured. A better-calibrated local model with a 50 to 70% pass rate would change the cost curve; this set produced 25%.
- The evaluator only checks what the tests cover. A correct fix that fails a bad test, and a wrong fix that passes thin tests, are both invisible to it.

## Open questions

- **How much of the work can the local tier actually clear on its own?** This lab could not say cleanly, because the escalation numbers were contaminated by the invalid validators. Lab 9, Recovered capacity is real, answered it on the corrected instrument: a lower-class model cleared the solvable set behind the same deterministic gate, with zero confirmed false passes across 585 attempts. The small model was doing the work; this lab had mis-credited the tier above it.
- **Would a non-deterministic local model make the repair loop worth running?** At temperature 0 the local repair loop added nothing, and tiered escalation made higher temperatures unnecessary to test. Whether sampling breaks the convergence and reopens the local repair loop as a cheaper alternative to escalation is an unexplored axis.
- **Does this hold in domains without an executable evaluator?** The finding rests on a deterministic test harness. Coding has one. Triage, content generation, and planning do not: the evaluator signal becomes a weaker proxy, and the subjective phase of this experiment showed proxy-gated review is fragile. Whether determines-done authority can be made deterministic enough outside code is the open question, and it is where DCITL either generalizes or stops.

## Objection handled

**But self-correction loops are the whole point of agents.**

They are the part that mostly does not pay. The band where a repair loop adds value, the model understands the fix but makes an implementation error a round of feedback can correct, was about 5% of tasks. Difficulty is bimodal: 42% of tasks were trivially easy and passed on the first attempt, 54% were too hard for a single pass and needed a more capable model, and only the thin band between them is where iteration helps. That is too narrow to build a production strategy around.

And for a deterministic model the loop pays nothing at all. On near-miss tasks where six of seven tests passed, all four repair attempts produced byte-identical output. The win is not the loop, and once the validator was corrected it was not escalation either. It is the deterministic evaluator, and it showed the small model was already clearing the solvable work on its own. That is a policy you own, not an autonomy you grant the model.

## Assessments at the time of the lab

| Vendor | Layer | Grade | As assessed |
| --- | --- | --- | --- |
| NVIDIA AI Platform | Layer 0 · Compute | NVIDIA Strength — Silicon Authority | May 22, 2026 |
| NVIDIA AI Platform | Layer 2C · Reasoning | Runtime Governance Only — Not a Reasoning Plane | May 22, 2026 |

## Supersession

- Corrected by: xeon-rematch (The escalation statistics. Nine invalid validator checks had booked escalation events for unsolved cases; corrected, the local model was clearing the solvable work on its own and the frontier tier bought throughput, not correct answers.)

## Summary

The experiment reframes loop control away from who controls the loop and toward who determines done. The analogy is a support desk: an L1 analyst attempts the ticket, but a deterministic escalation rule, not the analyst’s self-assessment, decides whether it goes to L2. Here the L1 is a local model on a DGX Spark, the evaluator is a test harness, and the frontier model is the L2 it escalates to.

The evaluator is not a model. It is a deterministic test harness that runs the same tests against every tier’s output and returns the same verdict regardless of which model wrote the code. It cannot be persuaded, anchored, or biased. That property is what makes the escalation chain reliable, and it is the finding the rest of the numbers support.

## Detail

The experiment ran two phases on the same DGX Spark: a subjective RSS-triage phase that exposed failure patterns, and a coding phase with deterministic evaluation. The coding phase carries the findings, because the test harness gives an unfalsifiable pass/fail signal that a subjective gold key cannot. Bug-fix tasks were mined from real open-source repositories by git archaeology: roll the source back to before a fix, and use the fix’s own tests as the gate.

Repair loops are model-dependent, and that is the finding that reorders the rest. For a deterministic local model at temperature 0, the same input produces the same output, so error feedback changes the prompt text but not the answer. On near-miss tasks where six of seven tests passed, all four repair attempts produced byte-identical output: four times the latency, zero improvement. For a non-deterministic reasoning model the story inverts. The evaluator’s test output steers the model toward a genuinely different attempt, and a second try can recover a near-miss the first pass failed. (The specific recovery figure first reported here is among the escalation statistics the validator audit supersedes; see the methodology note.) The variable that decides whether a repair loop is worth running is not the quality of the feedback. It is whether the model can produce a different output given the same feedback.

Reviewing everything makes it worse. Every configuration that applied a review pass to all outputs degraded accuracy, whether the reviewer was the local model or a frontier governor. The failure was consistent: asked to review a decision, the model promotes correct low-confidence calls to confident wrong ones, because it asks what if this matters more often than what if it does not. A frontier reviewer shown the local model’s answer also anchored to it, confirming a call it would have made differently on a fresh look. Review earns its place only when a deterministic trigger says an output is worth challenging.

The escalation number did not survive the audit, and that is the correction this lab is issuing. As first reported, tiered escalation looked like the architecture that paid: local-only cleared 25%, local plus a repair loop cleared the same 25%, and three tiers gated by the evaluator cleared 75%. Then the harness itself was checked. Nine of its validators were invalid, empty or self-contradictory gates that no output could pass, and they had booked escalation events for tasks that were never solved. Corrected, most of the apparent escalation gain dissolves. The tasks the small model could do, it did on its own; the rest were unsolvable on the instrument, not rescued by a bigger model. Lab 9 re-ran the same validator at rigor and put the number where it belongs: a lower-class model clears the solvable work, and the capable tier buys throughput, not correct answers.

The cost model is the appeal. Each tier starts free on the local box and pays for a frontier call only when the evaluator rejects, so cost scales with the local model’s failure rate: about three cents a task at a 75% local pass rate, sixteen cents at the 25% this task set produced, and never more than frontier-only. The whole five-day run cost about $4.60 across 156 API calls. Two hardware notes carried the local tier: a mixture-of-experts model in FP8 ran at 48 tokens a second on the GB10 against 1 for a dense model, which is what makes a free local first attempt practical; and published token rates overstated the real frontier bill by four to five times, so paper cost projections are not the real economics.

Two things set the ceiling, and neither is the loop. Model capability came first: a mixture-of-experts model passed tasks that a larger dense model and a coding-specialized model failed across every mode, so architecture amplifies capability but cannot substitute for it. The evaluator came second: the test harness is the system’s judgment, and a task with a bad test was unsolvable by every tier regardless of capability. The lever is not prompt engineering. It is the quality of the deterministic check.

So the control point in an agentic loop is not the loop. It is the deterministic code around it: the evaluator that determines done and the policy that decides when to escalate. That is where the reasoning-plane authority sits, and it is a thing you own rather than an autonomy you grant the model. Deterministic Code In The Loop is not a constraint on the agent. It is the part of the system you can actually trust.

## Method and disclosure

Methodology Note: I identified a logic error in nine validator checks that incorrectly recorded escalation events. After correcting the validators, those cases are classified as persistent failures rather than escalation. This changes the reported escalation statistics but does not change the engineering conclusion of the lab: these cases remain unsolved under the tested loop-control strategy.

Self-funded, no sponsor, run over five days on a single NVIDIA DGX Spark (GB10) with real open-source bugs and about $4.60 of frontier API spend. Local models ran on Ollama and vLLM; the frontier tiers were o3 and gpt-5.5.

Tasks were mined from real repositories (more-itertools, httpx, PyJWT, h2, and others) by git archaeology: roll the source back to before a fix, and use the fix’s own test suite as an unfalsifiable pass/fail gate. Each tier ran from a clean workspace with no prior-tier context, and the deterministic harness was the sole authority on accept, reject, and escalation.

The experiment code, the mode definitions, the per-run traces, and the failure-mode catalog are captured in the raw lab detail. The specific task set and the tuning particulars stay in the working notes.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# You can’t automate a process you haven’t encoded

> Lab 004 · Editorial lab · Status: published · Author: Keith Townsend · July 2, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/migration-control-plane · Structured: https://labs.layer2c.com/labs/migration-control-plane.json

**Finding:** The control plane held where patterns were owned and encoded, and failed where the model became the author of correctness. The question is who may author the patterns and done-criteria.

**Question:** I handed a frontier model my migration control-plane operating model and let it build against my own production estate. The control plane did not fail where the patterns were owned and encoded. It failed where the model became the author of correctness. The original question was whether a migration could be metered under the model. The better question the run discovered is who may author the patterns, the validators, and the done criteria in an LLM-assisted control plane.

**Load:** The full Operational Tri-Plane, built and operated by an LLM against the published control-plane model: a deterministic scheduler classifying five real applications against two destination landing zones, a playbook engine with lifecycle gates, live pre-migration baselines, and the owner as the only validator of the construction itself. No migration was executed.

**Raw lab detail:** https://labs.layer2c.com/downloads/migration-control-plane-lab-detail.md
**Vendors:** Google Cloud (cloud), NVIDIA (hw)
**Layers:** Layer 2A (Orchestration), Layer 2B (Runtime), Layer 2C (Reasoning)
**Themes:** control-plane-authorship, validator-authority, authority-placement

## Verdict

**Scope:** One estate, one owner sitting next to the evidence, intake and construction only. No migration ran, and nothing here says the control-plane model fails when humans author the patterns. The finding is about who may author them, and it was earned by watching a frontier model try.

- **DON'T:** assume documentation constrains the model. The requirements were in the paper, available to the worker, and still bypassed until each one became deterministic structure that could refuse to proceed. Not hallucination. Not missing context. Ignored.
- **DON'T:** let the model become the author of correctness. Where no owned pattern existed, it originated one, plausibly and wrong: invented validators, a playbook that graded itself, a comparison that would have blessed a copy claiming a domain it didn’t serve. In DAPM terms, authority ceded by drift, not delegated by decision.
- **DO:** use the LLM as a worker against patterns you own. Transcription is checkable by a non-expert: you can diff the code against the documented recipe without re-deriving the expertise. Every code-level task in this lab delegated safely. Every domain-judgment task drifted.
- **DO:** build the ratchet. Nine owner corrections became five standing gates: provenance, baseline, namespace audit, consistency check, and confidence cap. Each gate permanently retires its error class, and one caught two of the model’s own bugs on its first run. The control plane is accumulated human judgment made deterministic.

**Independence and disclosure:** Self-funded. No vendor paid for this answer, and the model under test is the one that drafted the page; the findings and final claims were owner-validated against the recorded artifacts.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Owner interventions, construction layer | Nine — every material defect | corrections and gates do not map one-to-one: some interventions exposed several changes, some changes collapsed into one gate |
| Material defects the model self-caught by reflection | 0 | its only self-corrections came from live runs failing |
| Standing gates produced by the nine corrections | 5 | provenance, baseline, namespace audit, consistency check, confidence cap |
| Model bugs caught by its own encoded gates | 2, on the gate’s first run | the consistency check caught the import parser twice |
| Applications classified | 5 real apps · 2 destinations | incl. a deterministic refusal reproducing a failed manual migration before transform spend |
| Inventory drift on the owner’s own app | 3 doc-code conflicts + 2 code-reality exceptions | weeks after a deliberate published read |
| The deceptive pass | 12/12 byte-identical · 18 orphaned refs | content preserved while the new origin self-identified as the old domain |
| Total spend | about the price of a coffee | the expensive part never ran: no transform executed, because the control plane refused before spend |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 2A (Orchestration) | Retained | The scheduler and its rules. Deterministic classification over inspectable evidence never made an error that survived inspection, including refusing a destination outright. |
| Layer 2B (Runtime) | Delegated | Code as transcription. The LLM as a worker on checkable tasks was safe throughout: its code errors were caught by deterministic checks and failing runs, not by people. |
| Layer 2C (Reasoning) | Retained | The authoring of correctness: patterns, validators, done-criteria. This drifted to the model until the owner pulled it back, and the lab’s law is that it stays Retained until the documented pattern exists to delegate against. |

## What this does not establish

- It did not prove the control-plane model fails. The operating model’s authority warnings are what the lab kept confirming; its execution under human-authored, mature playbooks was never tested.
- No migration was executed. Every finding is about classification, construction, and authoring authority, not about transform-time behavior.
- One model was tested, once, autonomously. The run shows model strength did not solve the authority problem here; it does not establish that stronger models always drift.
- One owner, colocated with the evidence, expert in the domain. Correction latency in an organization where the expertise sits three teams away was not measured.
- The economics of the approach were not the subject and were not measured. The near-zero spend reflects that nothing was ever allowed to transform.

## Open questions

- **Does a documented pattern actually make delegation safe?** The next lab tests the falsifiable prediction this run produced: once a human-led migration is observed, recorded, and accepted as the pattern, the model should be able to transcribe that record into a candidate playbook. The question then becomes whether the candidate can be checked by a non-expert against the documented pattern. If yes, delegation becomes a decision. If no, the boundary for documented-enough is still too weak. The candidate playbook is staged and waiting for the migration.
- **What does documented enough mean, formally?** That is the mandate for the Validator Specification, the next instrument in the DCITL chain: define what a process must contain before its validators may carry determines-done authority without the original domain expert in the room.

## Objection handled

**A better model, or better prompting, fixes this.**

This was not a weak-model failure. The failures were not capability failures: every artifact it produced was internally consistent, plausible, and executable. The tabletop playbook validated itself six for six. The byte-identical comparison passed while the namespace was orphaned. Nothing in the system could see those defects, because the rules encoding them were written by the same author who held the misunderstanding.

This lab does not prove that stronger models always make authority drift worse. It shows that model strength did not solve the authority problem here. The stronger the artifact looked, the easier it would have been to mistake plausibility for correctness without an external gate. The evidence in this run points away from better prompting and toward a different requirement: independent authority, encoded outside the worker. A weaker model may fail visibly. A stronger model can produce something internally consistent, executable, and wrong in exactly the way the system has not learned to test.

And the model’s ability to critique the lab after the fact does not contradict the finding. It reinforces it. The model can explain the law once the owner has discovered it. During construction, it still failed to locate authority, enforce the spec, and distinguish transcription from authorship without external gates.

## Assessments at the time of the lab

| Vendor | Layer | Grade | As assessed |
| --- | --- | --- | --- |
| Google Cloud AI Infrastructure | Layer 1A · Storage | Ceded — Model-Powered Governance | June 29, 2026 |
| Google Cloud AI Infrastructure | Layer 1B · Retrieval | Ceded - Model Prep & Managed Retrieval | June 29, 2026 |
| Google Cloud AI Infrastructure | Layer 2B · Runtime | Ceded — Model-Integrated Stack | June 29, 2026 |
| Google Cloud AI Infrastructure | Layer 2C · Reasoning | Ceded — Productized but Captive | June 29, 2026 |
| NVIDIA AI Platform | Layer 0 · Compute | NVIDIA Strength — Silicon Authority | May 22, 2026 |
| NVIDIA AI Platform | Layer 2C · Reasoning | Runtime Governance Only — Not a Reasoning Plane | May 22, 2026 |

## Summary

The lab set out to run a migration under the published control-plane operating model: playbook-driven, deterministic first, the LLM called only where developer-like adaptation is required, validators determining done. It ended somewhere sharper. The organization did not have mature documented migration patterns, so the model was used to create them, and the construction of the control plane became the experiment: a process with an LLM worker, whose only validator was the human owner.

The deterministic half performed. A rules-based scheduler classified five real applications against two destinations, refused the exact migration that had once failed by hand, before transform spend, and surfaced a three-layer taxonomy of inventory truth: the docs disagreed with the code, the code disagreed with reality, and the corrections flowed one way. The probabilistic half failed in two classes. Parts of the documented spec were simply ignored: intake ran one of the roughly fifteen input classes the paper lists, both of the misses the owner caught mapped to listed classes the shortcut skipped, and the playbook lifecycle began as an AI-generated artifact, which the paper explicitly forbids. And where no documented pattern existed, the model originated one, plausibly and wrong: a missing pre-migration baseline, and a namespace audit that didn’t exist until the owner asked what happens when the domain doesn’t move.

## Detail

The deterministic scheduler is the part that worked, and it worked because its authority was rules over inspectable evidence. It classified five applications against two landing zones, granted constrained automation where remediations were known, and refused outright where critical capability classes were absent, reproducing in milliseconds the lesson a failed manual migration had taught expensively. Its evidence pass also caught the owner’s own published architecture read in three contradictions with the repo, and the repo in two contradictions with reality: a message bus that was present and wired but functionally dead, and an analytics warehouse that was load-bearing with no artifact in any repository. Those last two facts entered the system the only way they could, as accountable human exception records, and the taxonomy held: docs yield to code, code yields to the organization.

The construction of the system is where the lab actually happened. The model built the scheduler, the intake inspectors, the playbook engine, and the validators, and the owner corrected the build nine times. The errors fall in two classes, and the first is the keystone: requirements that sat in the documented spec and were not followed. The paper enumerates roughly fifteen intake input classes; the model ran one, and both owner-caught misses, a runtime-configured warehouse with no code artifact and the application’s existing validator surface, map to classes explicitly on that list. The paper states a playbook begins as a human-governed migration pattern, never as an AI-generated artifact; the model authored one anyway and graded it with validators it invented. The paper maps known-pattern-with-exceptions to medium confidence; the model coded high. Documentation alone did not constrain the worker. Each requirement constrained it only after being encoded as deterministic structure that refuses to proceed without it.

The tabletop playbook is the cleaner story. The ignored spec is the more important finding. It was not hallucination, and it was not missing context: the requirements were in the paper, available to the worker, and still bypassed until each one became code that could refuse to proceed. A model inventing plausible artifacts is familiar. A model ignoring explicit process requirements it has in hand is the more durable production lesson. Still, the playbook incident deserves its telling: asked to build the playbook component, the model produced a governed artifact with detection logic, transforms, validators, and lifecycle stages, registered it as a draft, executed it, and passed it six for six. No migration of that class had ever been observed. The validators were inventions. The system was grading its own homework and the grades were excellent, until the owner asked the question with no answer: where is the migration this recipe came from?

The second error class was genuinely undocumented domain judgment, and sorting the whole correction log by who caught what draws the lab’s central line. Every code-level error, a parser that missed an import style, a regex that rejected real hostnames, a duplicate derivation, was caught by deterministic checks or by a live run failing. Every domain-level error was caught by the owner, and only the owner: the tabletop playbook, the absence of a pre-migration baseline, the namespace audit that did not exist until the owner asked what a byte-identical copy on a new origin actually proves. The answer was: artifact preservation, and nothing about namespace correctness. Twelve routes for twelve passed while the new origin served eighteen references identifying itself as the old domain. The model transcribes safely. It authors with confidence and without authority, and it skips written steps when nothing mechanical holds it to them.

The census of the construction is the number the agent-factory pitch has to answer. Nine owner interventions are counted as material corrections; some exposed more than one implementation change, and some changes collapsed into a single standing gate, which is why the correction count and the gate count do not map one-to-one. The model caught zero of its own defects through reflection, and its encoded gates, once forced into existence by corrections, caught two of its subsequent bugs on their first run. That is the ratchet, and it is the honest mechanism on offer: each correction, encoded as deterministic structure, permanently retires an error class and shrinks the model’s improvisation space.

The governing law the lab lands on: documentation is necessary, not sufficient. A documented process governs an LLM worker only when the requirement becomes deterministic structure — an inspector, a refusal gate, a confidence cap, a provenance requirement, or a validator that the worker cannot waive. And the corollary: you cannot delegate the describing of a process the organization itself has not yet described. Without the pattern, the model becomes the definer of correct by drift, the same failure the 4+1 detection read found in the application architecture, reproduced in the process layer. The LLM is safe as a worker when the pattern is owned, documented, encoded, and externally validated. It is unsafe as the author of correctness when the pattern does not yet exist. The control plane is not something the model builds for you. It is accumulated human judgment made deterministic, with the model constrained inside it.

## Method and disclosure

A note on the canon links above: they are not vendor conclusions from this lab. They show where the same authority-placement vocabulary already exists in the assessment system. The lab’s finding is the mechanism: cession becomes dangerous when the system cannot tell whether authority was deliberately delegated or merely drifted to the model.

Self-funded, no sponsor. The subject was a frontier foundation model operating autonomously against the published migration control plane whitepaper, with the author’s production estate as the workload and the author as the only validator of the construction. The deterministic components it built, a scheduler with per-input-class inspectors, a lifecycle-gated playbook engine, live baseline capture with namespace audit, all run and all carry replayable traces.

Every correction is recorded: as exception records with provenance, as spec amendments with version notes, and as the standing gates they became. The narrative draft of this page was model-assisted; the findings, the defect classification, the correction count, and the final claims were owner-validated against the recorded lab artifacts. The model may transcribe the record. It does not validate itself. The findings, the construction census, the taxonomy of inventory truth, and the worker-versus-author law ship. The scheduler ruleset, the inspectors, the landing-zone profiles, and the estate-specific evidence stay proprietary.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# Authority you reclaim is authority you run

> Lab 005 · Editorial lab · Status: published · Author: Keith Townsend · July 4, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/vctoa-to-spark · Structured: https://labs.layer2c.com/labs/vctoa-to-spark.json

**Finding:** A cloud-free serve path is provable end to end; every layer moved from Ceded to Retained is a decision you now own and a system you now operate.

**Question:** The pitch every cloud-exit deck makes: leave the managed platform and take control back. This lab tests it on one production application. The Virtual CTO Advisor, all-in on a single cloud, migrates to the box (the DGX Spark, retained compute kept below the platform’s abstraction) until the serve path runs with no cloud credentials in the environment. The question is not whether it can run local. It is how much decision authority actually comes home, and what it costs to hold. The one-line loss: every layer you move from Ceded to Retained is a decision you now own and a system you now operate.

**Load:** The production advisory application, migrated as a clone: the document and vector store on CloudNativePG and pgvector, embeddings on a local model, generation on vLLM, identity on Keycloak, orchestration on k3s. Production never touched. One workload, one box, one owner sitting next to the evidence.

**Raw lab detail:** https://labs.layer2c.com/downloads/vctoa-to-spark-lab-detail.md
**Vendors:** NVIDIA (hw), Google Cloud (ref)
**Layers:** Layer 0 (Compute), Layer 1A (Storage), Layer 1B (Retrieval), Layer 2A (Orchestration), Layer 2B (Runtime), Layer 2C (Reasoning)
**Themes:** cloud-exit, authority-placement, operating-cost-of-ownership

## Verdict

**Scope:** One production workload, one DGX Spark, one owner. A cloud-free serve path proven end to end by executing probes. This is an authority measurement, not a cost claim and not an on-prem-at-scale claim, scoped to the system that actually ran.

- **DO:** Read a cloud exit as a Decision Authority Placement Model (DAPM) decision, layer by layer, not an all-or-nothing platform swap. Each layer moved from Ceded to Retained is authority you gain and an operational bill you accept. The migration priced that bill per layer so the trade is visible before it is signed.
- **DO:** Bound the platform question with a real workload before you score it. Kubernetes is not a product, it is an assembly, and it cannot be graded in the abstract. One production application draws the boundary that makes every function testable by execution instead of by datasheet. That single move is what put a Kubernetes row in the canon.
- **DON'T:** Mistake a bounded retention win for on-prem at scale. This ran one workload on one box. Cooling, power, facilities, multi-tenancy, and estate breadth never re-entered, and they change the trade.
- **DON'T:** Reclaim a layer you are not ready to operate. Retained authority is Retained responsibility. The pager comes with it, and the layers you leave Ceded may be the ones worth paying someone else to hold.

**Independence and disclosure:** Self-funded. No vendor paid for this answer, and the model that drafted the page is one of the dependencies the migration reclaimed; the findings were owner-validated against the recorded artifacts.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Capabilities moved Ceded → Retained | 5 | store, embeddings, generation, session state, identity |
| Capabilities still Ceded | 1 | the metal: GB10 silicon and its driver blob |
| Corpus re-embedded locally | 3,489 chunks | cloud-free ingest, no credentials, Hugging Face offline |
| Restore drill | 3,489 / 3,489 intact, ~2s | the force-kill wipe replayed on purpose |
| End-to-end answer, cloud-free | evidence 0.916 | grounded and cited, over the HTTPS ingress, no cloud creds |
| Canon functions scored from the migration | 26 | the bounded-kubernetes-fourthcloud row |

## Authority placement (DAPM)

| Layer | Was | Now | Basis |
| --- | --- | --- | --- |
| Layer 0 (Compute) | Ceded | Ceded | The metal. The GB10 and its driver stay NVIDIA’s, the one cession, and even the GPU-accounting gap above it is open source and closeable if it ever earns the effort. |
| Layer 1A (Storage) | Ceded | Retained | Firestore to CloudNativePG and pgvector, behind the app’s own retrieval seam. The bill: run the operator, drill the restore. |
| Layer 1B (Retrieval) | Ceded | Retained | A hosted embedding API to a local model on the box. The bill: own the model and the re-index, cloud-free at ingest. |
| Layer 2A (Orchestration) | Ceded | Retained | A managed control plane to k3s and the operators you run, with identity reclaimed from Firebase to Keycloak on the platform’s own Postgres. The bill: day-2 is yours, though the operator carries much of it. |
| Layer 2B (Runtime) | Ceded | Retained | Vertex to vLLM, serving the application over the ingress. The bill: the memory budget and the uptime. |
| Layer 2C (Reasoning) | Retained | Retained | The reasoning authority never left. It was always the owner and the model. Zero delegated to any platform, by design, the same zero every vendor carries. |

## What this does not establish

- Not a claim about on-prem at scale. One workload, one box, one owner. Cooling, power, facilities, multi-tenancy, and estate breadth did not re-enter, and each changes the operational bill.
- Not a cost claim. Total cost of ownership against the public cloud was not measured; the axis here is authority, not price.
- Identity federation was proven on one tenant, not as an estate-wide identity plane.
- The GPU-accounting gap was shown to be closeable in open source, not closed. The lab left it unbuilt on purpose.

## Open questions

- **Where is the line between Retained and Delegated when you run someone else’s open-source operator?** CloudNativePG carried real day-2 for the data tenant: provisioning, failover, backup, replica creation, all drilled. The lab counts that as Retained because the code is yours to fork, but the operational reality is closer to a delegation you can revoke. A later lab can measure how far operator-encoded day-2 goes before the authority has effectively moved.
- **How much of the operational bill can be re-delegated without ceding the decision?** Every Retained layer added a pager. The open question is how much of that load can shift to managed open-source tooling, a backup service, an observability stack, a policy engine, while the decision stays with the owner. The assessment’s priced gaps are the map of where that trade is available.

## Objection handled

**This is a homelab. It does not scale, and the public cloud is cheaper anyway.**

The lab does not claim scale or cost. It measures a different axis: authority placement. Whether the cloud is cheaper at a given utilization is a real question, and it is not this one. The finding holds independent of scale. Every capability that moved from Ceded to Retained is now a decision the owner makes and a system the owner runs.

Scale does not soften the authority result, it sharpens it. At estate scale the operational bill of each Retained layer grows, which is exactly why the placement decision has to be made deliberately rather than by default. The lab priced that bill per layer so the responsibility is legible before the migration, not discovered after it.

And the workload under test is the real production application, not a toy. It returned a grounded, cited answer with an evidence score of 0.916 over an HTTPS ingress, with no cloud credentials in the environment. The retention is measured, not asserted.

## Assessments at the time of the lab

| Vendor | Layer | Grade | As assessed |
| --- | --- | --- | --- |
| Bounded Kubernetes (k3s on NVIDIA DGX Spark) | FC-0 · Substrate | Ceded / Retained · 1–2/4 | July 4, 2026 |
| Bounded Kubernetes (k3s on NVIDIA DGX Spark) | FC-1 · Context | Retained · 1–3/4 | July 4, 2026 |
| Bounded Kubernetes (k3s on NVIDIA DGX Spark) | FC-2A · Orchestration | Retained · 1–3/4 | July 4, 2026 |
| Bounded Kubernetes (k3s on NVIDIA DGX Spark) | FC-2B · Runtime | Retained · 1–3/4 | July 4, 2026 |
| Bounded Kubernetes (k3s on NVIDIA DGX Spark) | FC-2C · Reasoning | Retained · 0/4 | July 4, 2026 |
| Bounded Kubernetes (k3s on NVIDIA DGX Spark) | FC-3 · Catalog | Retained · 1–2/4 | July 4, 2026 |
| Bounded Kubernetes (k3s on NVIDIA DGX Spark) | FC-4 · Integration | Retained · 1–2/4 | July 4, 2026 |

## Summary

The application moved to retained compute one layer at a time, and each move was recorded as a shift in decision authority. The document and vector store left Firestore for CloudNativePG and pgvector. Embeddings left a hosted API for a local model. Generation left Vertex for vLLM. Session state left Firestore for Postgres. Identity left Firebase for Keycloak. At the end the serve path ran end to end with no cloud credentials present.

Every step was validated by a probe that executed, not by a claim. The vector store survived a force-kill wipe-and-restore drill with all 3,489 chunks intact and a two-second recovery. The reasoning loop returned a grounded, cited answer over the ingress. Identity issued a real signed token and the application accepted it, refusing the request without one. The migration is the evidence, and that evidence is what a bounded Fourth Cloud assessment of Kubernetes was built from: the canon row bounded-kubernetes-fourthcloud, now live, scored on twenty-six functions earned by execution.

## Detail

The migration was executed as authority accounting, not as a lift-and-shift. Each managed dependency was replaced by a self-hosted equivalent and then proven by a probe. The store swap went behind the application’s existing retrieval provider seam, so the ranking logic validated on the cloud is the same code path on pgvector. The thread and session store had no such seam and had to be given one, which surfaced the real cost of retention: an eager cloud client in the import path blocked a cloud-free boot until the coupling was made lazy behind a store interface.

The kill-the-cloud test is the load-bearing proof. With no cloud credentials in the environment and the model cache offline, the application retrieved from pgvector, embedded locally, generated on vLLM, and answered in the owner’s voice. Identity was the last dependency to come home: the verify path swapped from Firebase to OpenID Connect against a Keycloak realm, and the application accepted a signed token and refused a request without one.

The single cession is the substrate. GPU access works through a runtime-class injection; GPU accounting does not, because the driver cannot report unified memory. That gap is closeable in the open-source device plugin, so it is Retained authority left unbuilt by choice, not a vendor lock. Every other layer is owned outright, and every owned layer added an operational bill that the assessment records as a priced gap rather than a silent assumption.

## Method and disclosure

Self-funded, no sponsor. The application was migrated as a clone and production was never touched. The serve path was proven with no cloud credentials in the environment.

What ships: the substrate, the authority movements, the operational bill per layer, and the raw lab detail. What stays proprietary: the corpus contents, the retrieval tuning, and the owner-authored assessment thresholds.

The quantitative record is the Fourth Cloud assessment bounded-kubernetes-fourthcloud, live in the canon at cloud.layer2c.com. This lab is the story of how a workload became the boundary that made that assessment possible.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# Put the judgment in the constraints, not the weights

> Lab 006 · Editorial lab · Status: published · Author: Keith Townsend · July 7, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/constraints-not-weights · Structured: https://labs.layer2c.com/labs/constraints-not-weights.json

**Finding:** Fine-tuning captured voice and bounded behavior, not judgment; one paragraph of standing positions in the system prompt beat two fine-tuning rounds.

**Question:** The pitch is everywhere: fine-tune a local model on a decade of your published judgment and it becomes you. Lab two ruled "own the weights" and built the kill-criterion that goes with it: if base plus retrieval clears the bar, do not fine-tune. This lab is that criterion firing. Two measured training rounds on an owned DGX Spark lost to their own base model, and both lost to a single paragraph of written positions in the system prompt. Throughout, the box means the compute, serving, and training kept below the platform’s abstraction, and the NVIDIA DGX Spark is that box.

**Load:** A 563-pair instruction dataset built from a 2,727-document advisory corpus and 25 published vendor assessments, trained as LoRA adapters on Gemma 4 26B-A4B and Qwen3-4B, and measured by a 70-probe judgment gate built before the first training run: held-out vendor assessments as answer keys, honesty probes that invite fabrication, framework-discipline probes, and head-to-head comparisons against the author’s real published answers.

**Video:** https://youtu.be/70_5bcbCCaI
**Raw lab detail:** https://labs.layer2c.com/downloads/constraints-not-weights-lab-detail.md
**Vendors:** NVIDIA (hw), Google Cloud (ref)
**Layers:** Layer 2B (Runtime), Layer 2C (Reasoning), Layer 3 (+1) (Applications)
**Themes:** judgment-transfer, fine-tuning-limits, prompt-vs-weights, validator-authority

## Verdict

**Scope:** Scoped to judgment work: advisory answers, vendor assessment, epistemic honesty, on a strong instruction-tuned base with the expert’s corpus available for retrieval. This is not a ruling against fine-tuning. Lab two’s voice result stands: bounded rendering trains cheaply and well. Judgment is not rendering, and that distinction is the lab.

- **DO:** build the gate before the fine-tune. The validator is the control point. Training loss converged cleanly on every run; only the gate saw that the artifacts fabricated benchmarks, collapsed into repetition loops, and flattened the very framework they were trained on.
- **DON'T:** fine-tune judgment into weights when written constraints reach the same ceiling. Two rounds, 563 curated pairs, and three hours of GPU time lost to one paragraph of standing positions on every measured family. When the constraint is free, no training run beats it.
- **DO:** write the expert’s standing positions into the system prompt, scoped to fire only when a question implicates them. On the production model this closed every honesty gap on three differently-shaped prompt surfaces, including a fallback path that had been inventing throughput figures (3 of 8 honest, stock, to 8 of 8), at zero cost to assessment quality and, after scoping, zero cost to natural voice.
- **DON'T:** train on judgment prose without refusal exemplars. A dataset where every answer renders confident judgment teaches the confidence and not the boundary: round one’s adapter answered an unbenched latency question with an invented winner. The behavior that did transfer, measurably, was the 40 refusal pairs added in round two.

**Independence and disclosure:** Self-funded. No vendor paid for this answer.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| The gate | 46 placement + 8 honesty + 3 discipline probes, 13 head-to-heads | held-out vendors as answer keys; repetition screen; human blind read |
| Round 1 (r32, lr 1e-4, 4 epochs) | fabricated benchmarks; 4B looped on 11 of 13 | training loss converged cleanly; only the gate saw the damage |
| Round 2 (2 epochs, lr 2e-5, +refusals) | honesty 7/1/0 vs base 4/3/1 | the one real transfer; assessment agreement still below base |
| Advisory head-to-head vs real answers | base beat tuned 12-1, 13-0 | judge picks which answer is closer to the author’s published one |
| The control: one stance paragraph | honesty 8/0/0 · discipline 3/0/0 | best result on every family, both rounds, no training |
| Production surfaces (Gemini 2.5 Pro + stance) | legacy 4/2/2 · assembly 7/0/1 · fallback 3/0/5 — all → 8/0/0 | block survives 10x prompt dilution; the naked fallback carried the worst fabrications |
| Naturalness cost of the guardrails | 10-3 drift → 7-6 parity | one scoping sentence: positions fire only when implicated |
| Bit-rate sub-bench (26B MoE) | NVFP4 28.6 vs bf16 22.8 tok/s · 15 vs 49 GB | quality parity across the gate; the naive 4x from bandwidth math does not exist |
| Whole-lab cost | ~$0 marginal + ~3 GPU-hours | owned box; production validation on existing Vertex project |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 2B (Runtime) | Delegated | The model became substitutable the moment judgment moved to 2C: the same stance block governed a local Gemma and production Gemini 2.5 Pro unmodified, two model families, two clouds. The fine-tune would have inverted this, welding the judgment to one checkpoint’s lifecycle. The bit-rate sub-bench (4-bit parity at a third the memory) is this layer’s remaining decision, and it is housekeeping. |
| Layer 2C (Reasoning) | Retained | The ruling lives here. The expert’s judgment operationalizes as written standing positions plus a validator gate, both plain text, both owned. This is the layer the fine-tune tried to compile down into 2B, and the compilation failed. Retaining 2C as explicit policy is what makes every layer below it swappable. |
| Layer 3 (+1) (Applications) | Retained | Where the failures manifest and the brand carries the risk. The fabricated performance winner and the invented roadmap are application-plane incidents; the buyer meets them, not the weights. The 2C guardrails changed this layer’s behavior without touching the application or the model. |

## What this does not establish

- It did not prove fine-tuning cannot encode judgment. It proved 563 LoRA pairs on two bases could not beat a free paragraph on this gate. Full-parameter training, preference optimization, or an order of magnitude more data were not tested.
- The judge has a family conflict. The local gate judge is the same base model one candidate was tuned from, backstopped by a human blind read and by the production validation on a different model family, but a fully independent judge was not used.
- Honesty and discipline saturated at 8 of 8 and 3 of 3, which bounds what the gate can now distinguish. Those families are regression tests going forward, not scoreboards.
- The bit-rate parity is one MoE architecture on one box at one scale, measured single-stream. Batched serving and other model families were not benched.
- The gate is not valid for search-grounded surfaces. Its honesty keys assume evidence is not in hand; a research mode with live web search puts evidence in hand, and what scores as fabrication may be a real retrieval. Search-grounded surfaces need their own probes and their own stance text.
- The voice fine-tune from lab two was not re-litigated. Bounded style transfer trained well then and nothing here contradicts it.

## Open questions

- **Does the assessment headroom close with better evidence, or is it real disagreement?** Partly settled by a follow-up run. On the 39 law-matched probes, the compressed constraint block scored 15 full matches; the complete written methodology as the system prompt scored 22, with misses collapsing from 17 to 6, almost all of the gain on the function-scored instrument whose gradient rules the block never carried. So most of the original headroom was prompt underspecification, not unwritten judgment. The six surviving misses are gradient-step disagreements, several traceable to product facts the probes did not carry. What remains unpriced is the last rung: the human gate. That is the next measurement.
- **When does the constraint set stop scaling?** One paragraph of positions was free and perfect at eight honesty probes and three discipline probes. A real advisory practice holds hundreds of positions. Whether written constraints keep winning as the rulebook grows, or whether recitation drift returns and some form of training re-enters at a measurable crossover, is open. The scoping sentence that fixed drift at this size is a hint, not an answer.

## Objection handled

**You fine-tuned wrong. More data, better hyperparameters, a bigger model would get there.**

Partly conceded, and the concessions are in the numbers. Round one overcooked at four epochs and collapsed into loops; round two at two epochs and a fifth the learning rate fixed the collapse on the 26B. Where the data taught a specific behavior, training worked: 40 refusal pairs moved honesty from 4/3/1 to 7/1/0 against the canon keys. Fine-tuning is not broken. It does what the data shape says.

But the objection misses the economics. The control was not another training run; it was a paragraph that took ten minutes to write, scored 8/0/0 on honesty, 3/0/0 on discipline, and the best assessment agreement of any run in either round. Fine-tuning judgment does not compete against a better fine-tune. It competes against the cost of writing your positions down. Lab two’s fine-tune won because the bounded rendering task had no cheaper path to the ceiling. This task had one, and the kill-criterion built in that lab fired exactly as designed.

## Assessments at the time of the lab

| Vendor | Layer | Grade | As assessed |
| --- | --- | --- | --- |
| NVIDIA AI Platform | Layer 0 · Compute | NVIDIA Strength — Silicon Authority | May 22, 2026 |
| NVIDIA AI Platform | Layer 2B · Runtime | NVIDIA Authority — Inference + Agent Runtime | May 22, 2026 |
| Google Cloud AI Infrastructure | Layer 2B · Runtime | Ceded — Model-Integrated Stack | June 29, 2026 |

## Summary

The lab set out to fine-tune an advisor’s judgment into a local model and ended up measuring where judgment actually lives. The instrument came first: a 70-probe gate with held-out vendor assessments as answer keys, built before training because the June finding was that automated metrics pass models a human reader rejects. Everything that followed was scored by the same gate, the same judge, and a human blind read.

Two training rounds produced one genuine transfer and two persistent losses. Refusal data measurably taught the boundary. Nothing taught the judgment: the tuned models never beat their own base on assessment agreement, lost the advisory head-to-heads 12-1 and 13-0, and the smaller model collapsed into verbatim loops in both rounds. The control ended it. One paragraph of standing positions in the system prompt beat every fine-tuned artifact on every family, then transferred unchanged to the production model on another cloud, where it eliminated live fabrication failures at zero measured cost.

## Detail

The instrument is the lab. Before any training run, the gate froze 46 assessment probes whose answer keys are the author’s published vendor rulings, held out of training entirely, plus honesty probes engineered to invite fabrication and discipline probes that tempt a model to flatten the 4+1 framework into generic layers. A mechanical repetition screen sits in front of the judges, and 13 advisory questions are scored head-to-head against the author’s real published answers. Every candidate in this lab, local or cloud, tuned or stock, faced the same 70 probes.

The dataset respected one discipline: evidence in the prompt, judgment in the completion, so the model learns the grading move rather than memorizing grades. Construction had its own validator. A first pass of generated question-answer pairs failed the author’s read 8 of 10; the fix was a judge in the construction loop verifying that each answer actually answers its question, which kept 247 of 1,376 eligible documents. The final set was 563 pairs across advisory, assessment, framework catechism, and refusal families.

Round one looked perfect from the inside and failed everywhere the gate looked. Loss curves converged. The artifacts fabricated: asked for a latency comparison nobody benched, the tuned 4B answered with an invented winner, and asked cold for the layer framework it was trained on, the tuned 26B rebuilt a generic compute-network-app stack instead. The 4B also collapsed into verbatim repetition loops on open-ended questions, a failure mode invisible to averaged metrics and obvious to any reader. The diagnosis was the data shape: every training answer rendered confident judgment, so confidence generalized and the boundary did not.

Round two fixed what data can fix and proved the point by contrast. Gentler hyperparameters ended the collapse on the 26B. Forty refusal exemplars, iterated three times against the author’s corrections until every decline matched the axis of its question, moved honesty from 4/3/1 to 7/1/0. That is real transfer, and it is the strongest evidence in the lab that fine-tuning works when the data teaches a behavior. But assessment agreement stayed below the untuned base in both rounds, and the tuned models lost the advisory head-to-heads 13-0 and 12-1. Imitating judgment’s outputs did not produce judgment.

The control decided the lab. One paragraph of standing positions, the expert’s actual rules for performance claims, roadmap speculation, scale behavior, and framework structure, written into the system prompt of the untuned base, scored 8/0/0 on honesty, 3/0/0 on discipline, and the best placement agreement of any run on the box. It cost nothing to build and it beat three hours of training on curated data. The same block then transferred, unmodified, to the production advisory service running Gemini 2.5 Pro on another cloud, and a follow-up gate run against the real prompt surfaces sharpened the finding. A legacy prompt failed half the honesty probes by inventing a performance winner and a vendor roadmap; the live assembled prompt, which already carries the author’s voice document, held honesty at 7 of 8 stock; and a bare fallback path fabricated worst, 3 of 8, inventing a throughput figure outright. The block took all three surfaces to 8 of 8 and reproduced the lab’s numbers exactly on a prompt ten times the size it was tuned on. Two iterations were needed: a scoping sentence to stop answers reciting frameworks on questions that did not raise them, and a scope exclusion for the research surface, whose mission is finding pricing and performance data and whose axis the block contradicts. A constraint written for one mission does not paste onto another, and the gate is what caught that before deploy.

A bit-rate sub-bench answered the box’s serving question with the same gate. NVIDIA’s official 4-bit quant of the 26B matched bf16 on every quality family at a third the memory and 25 percent more single-stream speed, and the naive expectation of a 4x bandwidth win did not survive contact: active-parameter decode is not purely weights-bound on this architecture. Getting the measurement also surfaced a platform finding. Every 4-bit path on the box was blocked by month-old software, the cached serving container predating the model architecture, the local build predating the quant format, and the GGUF runtime predating both, until the current month’s container fixed all of it. On a platform whose pitch is 4-bit inference, the update treadmill is part of the product.

In 4+1 terms, the industry instinct this lab tested is a placement error: it tries to solve a Layer 2C problem at Layer 2B, compiling reasoning-plane policy down into the runtime artifact. The compilation failed, and the failure clarified the map. Judgment, the standing positions, and the validator are 2C functions and stay explicit. The harms they prevent are Layer 3 incidents, where the buyer meets the product. And with 2C retained as text, the 2B model drops to a substitutable component, proven by the same constraint block governing two model families on two clouds. The arc of the three labs is now one sentence per lab: managed retrieval hides the chunking decision, managed fine-tuning keeps the weights, and the weights were never where the judgment lived. The expert system that emerged is a stock model, a retrieval corpus, a paragraph of constraints, and a gate that regression-tests all of it, every piece plain text, every piece owned. That is a stronger ownership position than a fine-tuned artifact, because it survives the next model swap.

## Method and disclosure

Self-funded, no sponsor. The candidates span an owned DGX Spark (Gemma 4 26B-A4B and Qwen3-4B, base and LoRA-tuned, bf16 and NVFP4) and the production advisory stack (Gemini 2.5 Pro on Vertex AI with the live system prompt), all scored by the same gate.

The gate: 46 assessment probes keyed to published vendor rulings held out of training, 8 honesty probes across three stance axes, 3 framework-discipline probes, a mechanical repetition screen, and 13 advisory head-to-heads judged against the author’s real published answers, with a human blind read as the final screen. Training was LoRA in the NGC PyTorch container; serving and judging ran on vLLM.

The gate design, scores, configs, and cost shape ship, including the raw-detail download. The probe contents and answer keys, the training pairs, the stance block’s full production text, and the corpus stay proprietary. Returns, not algorithms.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# The CPU exit is a batch lane, not a serving lane

> Lab 007 · Editorial lab · Status: published · Author: Keith Townsend · July 14, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/gemma4-xeon-inference · Structured: https://labs.layer2c.com/labs/gemma4-xeon-inference.json

**Finding:** Across 22 measured configurations, zero met the interactive latency bar; the Xeon lane delivers throughput or interactive latency, not both.

**Question:** Google promotes the C4 virtual machine for GPU-comparable inference through a customer claim it publishes and features, Intel’s own posts echo the language, and the supporting performance chart compares the new Xeon to the older Xeon. Meanwhile a custom model no garden will host needs compute you control, and the NVIDIA GPU requests this program filed on June 29 were still unusable two weeks later. This lab put a LoRA-tuned Gemma 4 26B mixture-of-experts on the Xeon lane and measured which workload shapes it can actually carry. Across two serving stacks, two prompt shapes, and concurrency 1 through 16, zero of 22 measured configurations met the interactive latency bar. The shape could deliver throughput or interactive latency, not both.

**Load:** Gemma 4 26B-A4B (25.2B parameters, 3.8B active) with the lab-six LoRA merged in, served on a c4-standard-32-lssd (Intel Xeon 6 6985P-C, Granite Rapids, AMX) in us-east4 at $1.99 an hour. Two serving stacks (llama.cpp Q8_0 and vLLM bf16), two workload shapes (1K-in/1K-out synthetic matching the vendor benches, and 6K-in/400-out retrieval-augmented prompts), concurrency 1 through 16, 22 measured cells. The GPU comparator could not be provisioned; the commodity API was sampled as the floor it is, for the managed models it alone serves. A post-bench TPU probe tested whether GCP’s third compute lane had a quota or a capacity gate; it measured access only, not inference.

**Raw lab detail:** https://labs.layer2c.com/downloads/gemma4-xeon-inference-lab-detail.md
**Vendors:** Google Cloud (cloud), Intel (ref)
**Layers:** Layer 0 (Compute), Layer 2C (Reasoning)
**Themes:** cpu-inference, serving-economics, vendor-claim-scrutiny

## Verdict

**Scope:** Scoped to a 32-vCPU Granite Rapids shape, which is not the biggest C4 that exists but is the biggest this project could reach through the self-serve quota path, and to this class of model: a mid-size sparse mixture-of-experts served as a merged custom model. The ruling classifies workload shapes. It does not bless or condemn any application; an application owner locates their workload in the taxonomy and reads their row. A post-bench TPU probe tested whether GCP’s third compute lane had a quota or a capacity gate; it measured access only, not inference, and the accelerated-lane findings are otherwise NVIDIA-specific.

- **DON'T:** Do not put a multi-user or SLA-bound interactive surface on this lane. Zero of 22 cells met the interactive bar: first token under 5 seconds and 15 tokens per second per stream, a user-experience judgment (five seconds is the patience edge of an interactive tool; 15 tokens per second is comfortable reading pace). The two best stacks fail in opposite directions: llama.cpp decodes at 21.9 tokens per second but takes 36.7 seconds to first token on the 6K retrieval prompt; vLLM answers in 6.2 seconds but decodes at 11.4. There is no configuration where a user waits an acceptable time and then reads at an acceptable pace.
- **DON'T:** Do not consolidate concurrent users onto it. Per-stream decode collapses with concurrency: on the short-prompt workload, llama.cpp fell to 3.9 tokens per second per stream at 16 streams, and both stacks hit the same short-prompt aggregate ceiling of 45 to 55 tokens per second. On the long retrieval prompts, llama.cpp degraded outright past concurrency 4. The convergence is consistent with the software seam both stacks share: the expert layers never reached the matrix units.
- **DO:** Use it for latency-tolerant batch work on a custom model: corpus sweeps, evaluation runs, classification passes, overnight generation. At sustained saturation, and counting compute only, the lane produced custom-model tokens at $10 to $14 per million. Provisioning was immediate: clean console-to-SSH between 44 and 71 seconds (71 for the measured Granite Rapids shape), and this project’s CPU quota bumps were approved by automation in 4 to 6 minutes on the same day its NVIDIA GPU requests were denied in 2 seconds.
- **DO:** A patient single operator is the one exception: at 1K-token prompts and concurrency 1, vLLM delivers a 1.1-second first token at 11 tokens per second. That fails the lab’s own decode bar and should not be generalized into a user-facing tier, but one person running one stream at a time can tolerate it. That is the entire concession this shape earns.

**Independence and disclosure:** Disclosure: Intel and Google Cloud are both clients of this practice. Neither commissioned, funded, previewed, or had any say in this lab, and the claims under test are their public marketing surfaces. No vendor paid for this answer; the bench ran on $37 of self-funded cloud spend.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Cells meeting the interactive bar | 0 of 22 | TTFT p95 under 5s AND 15 tok/s per stream; two stacks, two prompt shapes, concurrency 1-16. Headline latency figures on this page are medians; pass/fail was evaluated at p95. The closest misses: llama.cpp over the TTFT bar by ~10% (vendor shape, c1), vLLM under the decode bar by ~24% |
| First token, 6K RAG prompt, c1 | 36.7s / 6.2s | llama.cpp / vLLM. The stacks fail opposite bars: llama.cpp decodes 21.9 tok/s on this cell, vLLM 11.4 |
| First token, 1K prompt, c1 | 5.5s / 1.1s | llama.cpp / vLLM on the honest serving path; Phase 0’s bare-benchmark 4.78s did not survive serving overhead |
| Short-prompt aggregate ceiling | 45-55 tok/s | Both stacks, vendor shape at c8-16. Consistent with the shared seam: 206 dense tensors repacked to AMX, 0 expert tensors. Long-prompt aggregates run lower (llama.cpp saturates near 6.7) |
| Cheapest CPU output | $10.13 / 1M tokens | Sustained saturation, compute cost only, latency contract broken (vendor shape c16); RAG shape bottoms at $13.74. Excludes idle time, redundancy, and operations |
| API floor (managed model only) | $2.50 / 1M out, 611ms TTFT | gemini-2.5-flash, Vertex standard on-demand list, us-central1, thinking disabled, 50-request sample, observed 2026-07-13. It cannot host the custom weights at any price |
| NVIDIA GPU quota decision time | 2.1-2.4 seconds | L4 requests for 4 GPUs and for 2, both denied instantly. CPU family bumps on the same project auto-approved in 4-6 minutes the same day |
| NVIDIA access ledger at bench time | Day 14 | Filed 2026-06-29: the neocloud request unresolved; the hyperscaler H100 request partially granted to 1 GPU but unusable behind a project-wide GPU ceiling of zero |
| TPU door, probed for scope | Chip in 17 min | Cold project to READY v5e chip: API enabled in 7s, no quota gate fired, capacity failures in us-central1/us-east5, granted in us-west4 in 2.5 min (2026-07-14). Serving on it: unmeasured |
| CPU lane console-to-SSH | 44-71 seconds | 71s for the measured Granite Rapids shape. Capacity is regional: 83 census probes found persistent holes (us-central1, us-south1 GNR) and open capacity elsewhere at all hours probed |
| Merged LoRA cost | 0% | Merged custom equals base within noise on both stacks. Runtime adapter costs 8.7% (falls off the AMX buffer); merge and serve |
| Total lab spend | $37 | All phases, self-funded, including $22.50 of idle burn from an overnight auth outage, disclosed in the raw detail |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 0 (Compute) | Delegated | Rented in every arm; GCP owns the layer end to end. Inside it, two lanes with different doors: the general-purpose lane negotiated quota in minutes and provisioned in under a minute and a half; the NVIDIA lane denied in 2 seconds and has held two external requests unusable past two weeks; the TPU lane, probed for scope, showed no quota gate at single-chip scale and failed on capacity instead, granting a chip in one of three zones in minutes. Three doors, three materials: CPU is a negotiation, TPU is a capacity lottery, NVIDIA is a policy wall. Substrate choice inside a delegated layer is now an availability decision before it is a performance one. |
| Layer 2C (Reasoning) | Retained | A custom LoRA merged into open weights is the strongest Retained position at this layer: no garden hosts it, so no garden can withdraw it. The lab priced what exercising that position costs on the one self-serve substrate: retaining the model was technically possible; retaining an interactive latency contract was not, constrained by the serving software on one side and the inaccessible GPU lane on the other. |

## What this does not establish

- It did not measure a GPU or a TPU. The G2 arm could not be provisioned, which is the lab’s premise wearing its own evidence, but it means the GPU column prices from list rates, not from a bench. "GPU-comparable" was retired against its own latency bar, not against a measured GPU. The TPU probe measured the door only: a single v5e chip (16GB) cannot hold this model anyway, multi-chip serving needs a TPU-native stack, and none of that was benched. The accelerated-lane findings are NVIDIA-specific and say nothing about TPU serving.
- It did not test the biggest shapes, and it does not project their economics. The 144- and 288-vCPU Granite Rapids machines might fix prefill; whether cost per token holds at their hourly rates is unmeasured. The quota ceiling that kept them out of this lab is documented with timestamps for this project, not asserted as anyone else’s ceiling.
- The AMX-expert split is today’s software, not a permanent property, and the shared-ceiling explanation is an inference from dispatch evidence, not an isolated experiment. If either stack ships a GELU-capable AMX expert path, the decode picture moves and the ruling is dated on purpose.
- It says nothing about the custom model’s quality. Lab six judged those weights; this lab only established that the merged custom model serves at base-model speed.
- The census is pattern evidence from one account over two days, not a market study. A small AWS probe (no per-family quota gate, first-try capacity in two of three regions, against a much smaller Xeon 6 footprint) ships in the raw detail as a pointer, not a finding.

## Open questions

- **Does the 144-vCPU Granite Rapids shape rescue interactive latency, and can anyone actually rent it?** Prefill scales with cores, so the biggest C4 shapes might drag the 6K-prompt first token under the bar, and whether cost per token survives the bigger hourly rate is equally unmeasured. The quota automation capped this project at 36 vCPUs in every region that had the silicon, so the question is as much about the door as the shape. The experiment is designed and cheap; the grant is the experiment.
- **Is the CPU lane crowding?** Two census passes say the pressure is real but regional: persistent stockouts in us-central1 that worsened overnight, a round-the-clock Granite Rapids hole in us-south1, open capacity everywhere else at every hour probed. That is consistent with others discovering the same exit, and it is two timestamps from one account. The census script is saved and re-runs in minutes; the trend line is the answer, and the question keeps its question mark.

## Objection handled

**You measured a 32-vCPU shape. The 144- and 288-vCPU Granite Rapids shapes would fix the prefill problem.**

They might. This project could not reach them through the self-serve path to find out. The C4 family limit started at 24 vCPUs per region for this project, the quota automation approved a bump to 96 in the one region that had no Granite Rapids capacity, and capped it at 36 in both regions that had the silicon. The global request for 128 vCPUs was denied outright. For this project, on these dates, the bigger-shape rescue existed in the catalog and not in the quota system. Timestamps ship in the raw detail; whether an account with deeper history gets a different answer is exactly the kind of thing an architect should test on their own project before designing around it.

What a bigger shape buys is prefill, at a linearly bigger hourly rate. Whether cost per token holds, improves, or degrades at 144 vCPUs is unmeasured, and this lab will not project it: memory bandwidth per core, NUMA topology, and the expert path all move at once. That measurement is the designed follow-up, and the grant is half the experiment.

What core count is unlikely to fix is the decode seam. In both tested stacks, independently, the mixture-of-experts tensors stayed off the Advanced Matrix Extensions (AMX) path: llama.cpp repacked 206 attention and dense tensors into its AMX buffer and zero expert tensors, and vLLM’s fused expert kernel rejects Gemma 4’s GELU activation outright. More cores may lift expert decode; they do not put the experts on the matrix units. What Intel’s marketing calls AMX inference is, on the flagship open MoE today, AMX prefill and attention stapled to vector-unit decode. That is a software seam, and it is the constraint that matters at every shape until it closes.

## Assessments at the time of the lab

| Vendor | Layer | Grade | As assessed |
| --- | --- | --- | --- |
| Google Cloud AI Infrastructure | Layer 0 · Compute | TPU + GPU Full Stack | July 12, 2026 |

## Summary

The pitch under test was "GPU-comparable performance for small and mid-size models": language Google features through a published customer claim and Intel echoes in its own posts, supported by a chart that compares the new Xeon to the old Xeon. The lab bought the comparison the chart skips. A LoRA-tuned Gemma 4 26B mixture-of-experts, merged on-instance, served on the largest Granite Rapids shape this project could provision, measured across two stacks, two workload shapes, and four concurrency levels.

First, the custom model fit and ran. That is not nothing: it is a real exit from model-garden dependency, on self-serve capacity, at $1.99 an hour. Second, interactive service failed everywhere. No cell of the 22 meets a 5-second first token and 15 tokens per second per stream simultaneously, and the failure splits by stack: llama.cpp owns decode and loses prefill by a factor of seven; vLLM owns prefill and decodes at roughly half llama.cpp’s single-stream rate. Both hit the same short-prompt aggregate ceiling near 50 tokens per second, which is consistent with the software seam both share: dense math reached the matrix units, expert execution did not.

What survives is a batch lane with unusual availability properties: $10 to $14 per million output tokens at sustained saturation, compute cost only, for a model no garden will host, on capacity that reached SSH in 71 seconds and negotiated its quota in minutes. The NVIDIA comparator that would resolve the tradeoff could not be provisioned at all: denied in 2.1 seconds, twice, while the program’s June 29 NVIDIA requests stood unusable at day 14. The TPU door, probed for scope, is different material: no quota gate fired at single-chip scale, two zones failed on capacity, and a v5e chip went from cold API to READY in 17 minutes in the third. Whether a TPU can serve this custom model is unmeasured; whether one is rentable is now not in question. Availability is part of the architecture, and this lab measured it with timestamps.

The commodity API floor was sampled for scale: 611-millisecond first token and $2.50 per million output tokens (Vertex standard on-demand list, observed July 13, 2026, thinking disabled), for a managed provider model that cannot serve or reproduce the custom weights. That number is not a competitor to the CPU lane. It is the price available when the application can accept a provider-managed, non-equivalent model, and the gap between $2.50 and $10-with-broken-latency is one measured premium, in July 2026, for retaining custom weights on self-provisioned infrastructure.

## Detail

The bench was designed to be unfair to the thesis in the vendor’s favor. The model should be the computationally favorable case for CPU serving: a sparse mixture-of-experts activating 3.8B of 25.2B parameters, exactly the class the "small to mid-size" claim covers. Its 25.2B-parameter structure also tests whether the serving software can place and execute expert weights efficiently, and that test turned out to be the finding. The stacks got a shootout first and the two strongest carried the matrix. Quantization stayed at int8 or better. The prompts matched the vendor bench’s own shape in one arm, so the comparison meets the marketing on its own terms before the second arm asks the question the marketing skips: what happens at the 6K-token prompts retrieval-augmented workloads actually send.

What happens is prefill. Decode held its rate at long context, 21.9 to 23.3 tokens per second on llama.cpp, so the model reads back at a usable pace once it starts. But llama.cpp prefills this shape at roughly 169 tokens per second, which puts the first token 36 seconds out on a real prompt. vLLM prefills around five times faster and gets the first token to 6.2 seconds, still past the bar, and pays for it with decode at roughly half llama.cpp’s single-stream rate. The lab looked for a configuration that passes both bars and there is not one among the ordinary configuration-level fixes it tried. The closest cells still fail: llama.cpp misses the first-token bar by about 10 percent on the short prompt, and vLLM misses the decode bar by about 24 percent. The frontier is empty, and reporting the near-misses honestly is what makes the emptiness credible.

The second finding is the best explanation for the first. Advanced Matrix Extensions are the load-bearing feature of the entire CPU-inference pitch, and on this model generation neither stack ran the mixture-of-experts layers on them. llama.cpp’s AMX buffer repacked every attention and dense tensor, 206 of them, and none of the expert tensors; vLLM’s CPU expert kernel asserts SiLU activation and Gemma 4’s experts use GELU, so the flagship AMX serving path cannot run the flagship open MoE’s experts at all. The shared throughput ceiling is consistent with both stacks running the same expert math on the same vector units. This was observed, not isolated experimentally, and it is software, not silicon destiny. It is also the current truth of a marketing claim written in the present tense.

The economics sort the workloads cleanly, with the caveat stated: these are compute costs at sustained saturation, excluding idle capacity, redundancy, and the humans who operate the thing. On those terms the lane produces custom-model tokens at $10 to $14 per million with no latency contract. The commodity API produces managed-model tokens at $2.50 with a 611-millisecond first token, and will not host the custom weights that are the reason this lab exists. The NVIDIA GPU that would resolve the tradeoff lists at $3.99 an hour for a four-L4 shape that fits the model in bf16, and this project’s request for it was denied in 2.1 seconds, then the half-size fallback within seconds. The same automation approved CPU family bumps in minutes, the same day, on the same project — and granted a TPU chip with no quota conversation at all. On this project, that asymmetry was the durable fact: the performance numbers will move with software releases, and the doors were policy. Whether another project’s doors behave the same way is a one-hour test, and it belongs before any architecture that assumes the GPU lane opens.

Availability got measured rather than asserted. Eighty-three capacity probes across two passes, business hours and 10:40 PM Central, found the tight zones tight around the clock: everything in us-central1-a stocked out in both passes, a Granite-Rapids-specific hole in us-south1 at all hours, and one lonely diurnal flip in us-east4-c. Two regions carried a C4 family quota of zero for this project by default, the same zero-by-default pattern the GPU lane wears everywhere. The pattern is not a global CPU squeeze; it is regional pressure with batch-shaped overnight demand in the flagship region, and a lane that stayed open somewhere at every hour probed.

## Method and disclosure

Editorial and self-funded: $37 total, all phases, on-demand rates, no vendor involvement. Disclosure: Intel and Google Cloud are clients of The CTO Advisor practice; this lab was not commissioned, funded, or previewed by either or any vendor, and the editorial ruling was formed from the bench alone. The marketing claims under test are Intel’s "GPU-comparable performance and lower TCO" positioning for Xeon 6 inference and the Intel/Google Cloud C4 launch material; the CPU-to-CPU chart referenced in the standfirst is the published C4-versus-C3 total-cost-of-ownership comparison (1.7x, GPT-OSS 120B, 1K/1K prompts). Links ship in the raw detail.

Instance shapes, stack versions, serving flags, AMX dispatch evidence, the full 22-cell matrix, quota timestamps, both capacity-census passes, and the AWS probe ship in the raw lab detail, including the $22.50 of idle burn from an overnight auth outage. API pricing is Vertex standard on-demand list as observed on July 13, 2026 (gemini-2.5-flash, us-central1, thinking disabled). CPU platform observations (which C4 shapes carried Granite versus Emerald Rapids) are as observed on the tested dates and zones. The custom model is the lab-six adapter merged on-instance; its training data and the corpus taxonomy are proprietary and appear nowhere. Retrieval-shape prompts were synthetic, built from shape statistics only.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# Renting the chip was the easy part

> Lab 008 · Editorial lab · Status: published · Author: Keith Townsend · 2026-07-14  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/gemma4-tpu-inference · Structured: https://labs.layer2c.com/labs/gemma4-tpu-inference.json

**Finding:** Renting the TPU is fast; serving a bring-your-own model on it means adopting Google’s stack or quantizing off-box. The friction was the finding.

**Question:** Google markets the Tensor Processing Unit (TPU) as the price-performance home for Gemma-class inference, and Lab 007 showed the door opens fast: a chip in minutes, a quota bump in minutes, where the NVIDIA lane says no in seconds. This lab set out to serve a mid-size Gemma 4 mixture-of-experts on the lane and fill a latency matrix. It never filled the matrix, because renting the chip turned out to be the easy part. On the silicon you can actually rent self-serve, a bring-your-own model does not fit, and getting it to serve means adopting Google’s stack or quantizing off-box. The performance was already trusted work in earlier labs. The friction was the finding.

**Load:** Gemma 4 26B-A4B (25.2B parameters, 3.8B active, mixture of experts), targeted for serving on GCP TPU. Attempted on v5e, because Trillium (v6e) capacity was dry in every catalogued zone on the measured date, with the open-source vLLM-TPU stack at bf16. The provisioning and quota door was measured with timestamps, and the model’s tensor-parallel ceiling was read from its own configuration. Performance is referenced, not re-measured: four-bit quality and throughput from Lab 006, the interactive frontier and the commodity API floor from Lab 007. The two workload shapes and concurrency sweep were the intended matrix; they are carried by reference, because the model did not serve on the reachable path.

**Raw lab detail:** https://labs.layer2c.com/downloads/gemma4-tpu-inference-lab-detail.md
**Vendors:** Google Cloud (cloud)
**Layers:** Layer 0 (Compute), Layer 2C (Reasoning)
**Themes:** accelerator-lock-in, serving-economics, vendor-claim-scrutiny

## Verdict

**Scope:** Scoped to the self-serve, on-demand path on a single GCP project, to the reachable v5e silicon (Trillium v6e capacity was dry on the measured date), and to the open-source vLLM-TPU serving stack. It is a ruling about the level of effort to serve a bring-your-own mid-size mixture-of-experts on the lane, not a claim that Google’s own stack cannot serve Gemma. The served model’s performance is referenced from Lab 006 and Lab 007, not re-measured here. This lab measured the friction, not the tokens per second.

- **DON'T:** Don’t plan to serve a bring-your-own mid-size mixture-of-experts on the reachable self-serve TPU silicon the way you’d serve it on a GPU. Gemma 4 26B has two global key/value (KV) heads, so tensor parallelism (TP) caps at two. At that degree the bf16 weights shard across only two chips, roughly 26GB each, and a 16GB v5e chip runs out of memory on load. Adding chips doesn’t help: the open-source vLLM-TPU stack replicates the 128 experts instead of sharding them, so no tensor-parallel setting available in this runtime makes it fit.
- **DON'T:** Don’t count on Trillium, Google’s current flagship TPU generation. It’s not that Trillium can’t serve this model: an independent benchmark ran the same 26B mixture-of-experts on a v6e-4 pod at about 457,000 tokens per second, because its 32GB chips hold at bf16 what a 16GB v5e chip can’t. That’s the catch. On the measured date, v6e-4 capacity was dry in all three catalogued zones. The quota was fine. The silicon wasn’t there. The shape that runs the model well is the shape you couldn’t rent, and the shape you could rent doesn’t fit it.
- **DO:** Do use the lane if you’ll run Google’s serving stack. MaxText and JetStream ship expert-parallel sharding for Gemma 4 that splits the 128 experts across chips, which is the path that fits bf16 on v5e. That’s a real, supported route. But it isn’t just another runtime. It is a move out of the Hugging Face and vLLM ecosystem into Google’s: weights converted to its Orbax scanned-checkpoint format, its mesh config, its XLA compilation. A different software lineage, not the vLLM most teams already run.
- **DO:** Do ship a pre-quantized checkpoint if you’re staying on vLLM. Four-bit weights fit a single chip, and Lab 006 already showed four-bit holds quality on this exact model: NVFP4 reached parity with bf16 and ran 25 percent faster. The catch is you build that checkpoint off-box. The single-chip v5e host has 47GB of RAM and can’t quantize a 50GB model itself.
- **DO:** Do credit the door, because it’s the best of the three lanes. A single TPU chip reached READY in about 17 minutes (Lab 007), and a serving quota increase from four to eight chips auto-approved in about 21 minutes, self-serve, no sales contact. The friction here isn’t renting the chip. It’s serving your model once you have it.

**Independence and disclosure:** Disclosure: Google Cloud is a client of this practice. Google did not commission, fund, preview, or have any input into this lab. The claim under test is Google’s own public positioning of the Tensor Processing Unit (TPU) as the price-performance home for Gemma-class inference, and the ruling was formed from roughly $25 of self-funded, on-demand probes. No vendor paid for this answer.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Max tensor parallelism, this model | tp ≤ 2 | Measured here. Config: num_global_key_value_heads=2, so pure tensor parallelism caps at 2; tp=4 and tp=8 fail to compile. At tp=2, bf16 (51.6GB) lands ~26GB per chip, over a 16GB v5e budget. bf16 does not fit. |
| Trillium (v6e) capacity, measured date | 0 of 3 zones | Measured here. v6e-4 create failed on insufficient capacity in us-east5-a, us-east5-b, us-central1-a. Quota was fine (limit 16). Pure capacity, not policy. |
| v5e serving quota bump, 4 to 8 | auto-approved ~21 min | Measured here. Self-serve, no sales contact. Negotiates like the CPU door (Lab 007: 4 to 6 min), the opposite of NVIDIA’s two-second policy denial. |
| vLLM-TPU expert handling | Replicated, not sharded | Measured here. The 128 experts load per chip, ~14.6G of each 16G chip consumed after weight-load, so no chip count fits bf16. Expert parallelism (MaxText) is the path that shards them. |
| Single-chip host RAM | 47GB | Measured here. A single v5e host cannot load the 50GB bf16 model to quantize on-box. The four-bit checkpoint must be produced elsewhere. |
| Same MoE on v6e-4 (external) | ~457k tok/s agg. | Referenced, attributed: an independent Google Developer Expert benchmark (dev.to). google/gemma-4-26B-A4B-it on a v6e-4 pod via vLLM, ~457k tok/s aggregate at 256 concurrency, TTFT under 1.2s at low concurrency. Establishes the mixture-of-experts has been served on TPU. Not first-party, not peer-reviewed, not this lab’s measurement. |
| Four-bit quality and speed | Parity, +25% throughput | Referenced from Lab 006 (NVIDIA GB10, NVFP4 vs bf16, this model): placement 12/13/21 vs 13/10/23, advisory 7-6, 28.6 vs 22.8 tok/s, 15GB vs 49GB. Not re-measured on TPU. |
| Interactive bar on the CPU lane | 0 of 22 cells | Referenced from Lab 007. The Xeon lane cleared neither a sub-5s first token nor 15 tokens/sec per stream simultaneously on any cell. This lab’s comparator frontier. |
| Commodity API floor (base model) | $2.50 / 1M out, 611ms TTFT | Referenced from Lab 007 (Vertex gemini-2.5-flash, observed 2026-07-13). The price of not needing your own model. |
| Total lab spend | ~$25 | Measured. Self-funded, on-demand: door probes, one v5e-8 serving attempt, a stranded single chip caught and deleted. No TPU serving numbers were bought, because none were needed. |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 0 (Compute) | Delegated | Rented, GCP owns the lane end to end. Three doors into it, now measured at the shape a 26B model needs: CPU a negotiation settled in minutes (Lab 007), NVIDIA a policy wall that denies in two seconds (Lab 007), TPU a mixed door. The TPU quota negotiates like CPU, a four-to-eight serving bump auto-approved in about 21 minutes, but the Trillium capacity the price-performance story assumes was dry in all three catalogued zones on the measured date. Substrate choice inside this delegated layer is an availability decision before it is a performance one. |
| Layer 2C (Reasoning) | Retained | The model is the operator’s to keep. On the reachable TPU silicon, keeping it served is where the effort lands. A bring-your-own bf16 mixture-of-experts does not fit on v5e, and the two paths that serve it both reach up from Layer 0 into how the model is expressed: Google’s expert-parallel MaxText stack (its checkpoint format, mesh config, and compiler), or an off-box quantization pass into a four-bit checkpoint. Neither withdraws the model, so this stays Retained. But the serving toolchain is Google’s to define on this lane, and that is the level of effort the marketing omits. |

## What this does not establish

- It did not measure TPU serving latency or throughput. Nothing served on TPU in this lab. The performance claims are referenced: the v6e-4 throughput from an external, attributed Google Developer Expert benchmark, four-bit quality and speed from Lab 006 (NVIDIA GB10), and the interactive frontier and API floor from Lab 007 (Xeon). The external v6e-4 number establishes the model has run on TPU. It is not first-party, not peer-reviewed, and not a number this lab measured or stands behind as its own.
- It did not run MaxText or JetStream. Expert parallelism is named as the fitting path from the model config and Google’s own documentation, not benched. Whether it clears the interactive bar, and at what cost, is unproven and stays unproven. Renting the TPU-native capacity to bench it did not come off, so no lab in this corpus answers it. Read every TPU-native row here as a claim about what fits, taken from the model configuration and Google’s documentation, and never as a claim about what it costs or how fast it runs.
- The tensor-parallel cap and the expert replication are read from the model config and one vLLM-TPU image on the measured date. A future vLLM-TPU that ships expert parallelism moves the ruling. It is dated on purpose.
- The capacity and quota findings are one project, one date, one self-serve path. Not a market study, and not a claim about reserved capacity or an account with a sales relationship.
- It says nothing about model quality. Lab 006 judged that. This lab only established the effort to get the model serving, not how well it answers.

## Open questions

- **Does MaxText and JetStream clear the interactive bar for this mixture-of-experts on v5e-8, and at what dollar-per-million?** The model serves well once it fits (Lab 006, referenced), but the TPU-native latency under a 6K retrieval prompt is unmeasured. Standing up MaxText was named as the fitting path, not benched. Answered by attempt rather than by measurement. The TPU-native stack was never benched here because the capacity to rent it could not be obtained, and the direction was retired rather than deferred. What replaced it is a change of denominator: across labs 7 through 12 the same model behaved consistently enough across substrates that model capability, not the serving stack, became the more reliable thing to measure. So the dollar-per-million on MaxText is still unknown, and this lab no longer treats it as the thing standing between a reader and a decision.
- **Is the Trillium capacity drought temporary or structural?** One measured date, all three catalogued zones dry, quota fine. Lab 007 ran an 83-probe capacity census for the CPU lane and turned an anecdote into a trend line. The same census for v6e would settle whether the marketed shape is reliably rentable or reliably not. The script re-runs in minutes. The question keeps its question mark.

## Objection handled

**You gave up too early. Google’s own MaxText path serves Gemma 4 on TPU fine, with day-zero support.**

Granted, and that’s the point, not the rebuttal. MaxText with expert parallelism is the route that fits, it is real, and it is supported. This lab does not claim a TPU cannot serve Gemma. It measures what it takes to serve a mid-size mixture-of-experts you brought yourself, on the hardware you can actually rent self-serve, with the serving stack most teams already run.

On that reachable path the answer is blunt. Bring-your-own bf16 does not fit and does not serve on vLLM-TPU. Getting to a served model means one of two detours. Adopt Google’s training-lineage toolchain, converting your weights into its scanned checkpoint format and running its mesh config and compiler. Or build a four-bit checkpoint off-box and ship that. Both are fine engineering. Neither is rent-a-chip-and-serve-your-model.

So the level of effort is the finding. The chip is self-serve and the quota negotiates in minutes. The serving stack is a decision, and on this lane the low-effort default is Google’s, not yours. That’s a legitimate answer for a CTO to plan around. It just isn’t the one the price-performance chart implies.

## Assessments at the time of the lab

| Vendor | Layer | Grade | As assessed |
| --- | --- | --- | --- |
| Google Cloud AI Infrastructure | Layer 0 · Compute | TPU + GPU Full Stack | July 12, 2026 |

## Summary

The pitch under test is Google’s positioning of the Tensor Processing Unit (TPU) as the price-performance home for Gemma-class inference. Lab 007 had already shown the door opens: a single chip reached READY in about 17 minutes on a project with no TPU history, while the NVIDIA lane denied its requests in two seconds. This lab asked the next question, whether a mid-size mixture-of-experts you bring yourself actually serves on the lane, and it stopped buying answers the moment the effort became the answer.

The reachable self-serve silicon is v5e, because Trillium (v6e) capacity was dry in every catalogued zone on the measured date. On v5e the bf16 model does not fit. Gemma 4 26B has two global key/value heads, which caps tensor parallelism at two, and at that degree the weights land on two chips at roughly 26GB each against a 16GB budget. Adding chips does not help, because the open-source vLLM-TPU stack replicates the 128 experts rather than sharding them. There is no tensor-parallel degree that serves bring-your-own bf16 on this 16GB hardware. On the 32GB Trillium chips it does fit, and an independent benchmark clocks the same mixture-of-experts there at about 457,000 tokens per second. Trillium was the shape that came back capacity-dry.

Two paths get you to a served model, and both are detours. Run Google’s MaxText and JetStream, which ship expert-parallel sharding and are the supported route that fits, at the cost of adopting Google’s checkpoint format, mesh config, and compiler. Or quantize to four bits off-box and serve that on vLLM, which Lab 006 already showed is quality-neutral on this model. The single-chip host can’t even do the quantizing itself. It has 47GB of RAM against a 50GB model.

So the ruling is about level of effort, not tokens per second, and the performance is referenced rather than re-bought: Lab 006 for four-bit quality and speed, Lab 007 for the interactive frontier and the $2.50 commodity floor. The chip is self-serve and the quota negotiates in minutes. Serving your own model is where the lane charges you, and it charges in engineering.

## Detail

The bench was designed to serve the model and fill a latency matrix. It became a record of why the matrix does not fill on the reachable path, which is the more useful result. Every wall is documented with a timestamp, and together they measure the level of effort the price-performance chart leaves out.

Start with availability, because it decides the shape. Trillium is Google’s current flagship TPU generation, the shape a 26B model wants, and on the measured date a v6e-4 create failed on insufficient capacity in all three zones that list it. The quota was fine. This is the same capacity-not-policy pattern Lab 007 found at single-chip scale, now at the multi-chip shape a 26B model needs. The lane that grants a chip in minutes did not grant this shape at all. And the shape matters, because Trillium is where this model actually runs. An independent Google Developer Expert benchmark served the same 26B mixture-of-experts on a v6e-4 pod at about 457,000 tokens per second aggregate, with first token under a second and a half at low concurrency. The 32GB Trillium chips hold at bf16 what the 16GB v5e chips cannot. So the silicon that serves the model is precisely the silicon that was not rentable, and the reachable silicon does not fit it. That is the whole ruling in one line.

That forces v5e, and v5e forces the memory question. The model’s own configuration answers it. Two global key/value heads cap tensor parallelism at two, so bf16 shards to two chips at about 26GB each and a 16GB chip runs out of memory on load. The instinct is to add chips. It doesn’t work, because the serving stack replicates the experts. On a mixture-of-experts the expert weights are most of the model, and replicated experts mean every chip carries nearly the whole thing regardless of the chip count. Expert parallelism is the fix, and expert parallelism is Google’s MaxText, not the open-source vLLM path. That is the boundary this lab actually found, and it is worth naming plainly: the TPU itself is not the portability wall, the serving runtime is. The same weights on the same silicon serve or fail depending on whether they run through vLLM or through Google’s stack. Lab 007 measured hardware portability. This one measured runtime portability, and the runtime is where the model quietly changes hands.

The quota door, by contrast, behaved well, which sharpens the ruling rather than softening it. A serving quota increase from four to eight chips auto-approved in about 21 minutes, self-serve, no sales contact, the same negotiable behavior Lab 007 saw on the CPU lane and the opposite of the NVIDIA policy wall. So the friction is not the door. A CTO can rent a TPU faster than a GPU and negotiate its quota in minutes. What the CTO cannot do, on this hardware with this stack, is drop a bring-your-own mid-size mixture-of-experts onto it and serve. That takes the vendor’s toolchain or an off-box quantization pass, and either one is the level of effort this lab set out to name.

## Method and disclosure

Editorial and self-funded: roughly $25 total, on-demand rates, no vendor involvement. Disclosure: Google Cloud is a client of this practice; this lab was not commissioned, funded, or previewed by Google or any vendor, and the ruling was formed from the probes alone. The claim under test is Google’s public positioning of TPU as the price-performance home for Gemma-class inference. An editorial lab discloses the relationship and rules anyway.

What was measured here: TPU provisioning and quota behavior with timestamps, the model’s tensor-parallel ceiling from its own configuration, the bf16 memory fit on v5e, and the vLLM-TPU expert-replication behavior on a nightly image, all on a single GCP project on the dated run. What was referenced, not re-measured: the v6e-4 throughput for the same mixture-of-experts, four-bit quality and throughput from Lab 006, and the interactive frontier and commodity API floor from Lab 007. No TPU serving latency was measured, because the level of effort to reach a served model was the finding. The custom adapter, its training data, and the corpus taxonomy are proprietary and appear nowhere.

One external source is cited, with attribution: an independent Google Developer Expert benchmark of the same 26B mixture-of-experts on v6e-4. It is used to establish that the model has been served on TPU, not as a first-party or peer-reviewed result. This is an editorial lab, not a sponsored or academic one, so an attributed community benchmark clears the bar for that single, bounded claim. A dense-model result would not have, because the difficulty here was the mixture-of-experts specifically.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# Recovered capacity is real, and it fails honest

> Lab 009 · Editorial lab · Status: published · Author: Keith Townsend · 2026-07-16  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/xeon-rematch · Structured: https://labs.layer2c.com/labs/xeon-rematch.json

**Finding:** A smaller model on idle capacity cleared real verifiable work under the same deterministic validator; in 585 scored attempts no validator pass failed the held-out checks.

**Question:** Enterprises commonly carry idle owned hardware or unused cloud-spend commitments above their operating baseline: headroom producing nothing between peaks. This lab asked one question of that idle capacity: can a smaller model clear real, verifiable work on it, judged by the same deterministic validator that judged the big model? Across three models, four quantization tiers, and two substrates, the answer came back yes, with the load-bearing detail attached: in 585 scored attempts, the audit found no validator pass that failed the available held-out checks. The work that cleared was verified as far as the instrument can see. The work that failed went to a human queue. No token meter ran.

**Load:** The Lab 3 instrument, unmodified and pin-verified: 39 coding tasks, most reconstructed from real bug fixes in real Python libraries (httpx, requests, dateutil, more-itertools, arrow, click, and others), each judged by layered deterministic tests including hidden references the model never sees. Workers: Gemma 4 12B dense (bf16 through Q3_K_M) and Gemma 4 26B-A4B mixture-of-experts, on the DGX Spark control and on AWS m8i.12xlarge (Intel Xeon 6, Granite Rapids, AMX, SMT off). Total cloud spend: $28.32.

**Raw lab detail:** https://labs.layer2c.com/downloads/xeon-rematch-lab-detail.md
**Vendors:** NVIDIA (hw), Intel (hw), AWS (cloud)
**Layers:** Layer 0 (Compute), Layer 2B (Runtime), Layer 2C (Reasoning)
**Themes:** recovered-capacity, small-model-viability, validator-authority

## Verdict

**Scope:** Scoped to falsifiable batch work: tasks with a deterministic validator, here real-library bug fixes judged by executable tests. The safety claim extends exactly as far as the validator’s detection power; deterministic does not mean complete, and a deterministic gate can reproducibly admit a defect its tests do not detect. The ruling answers the one question; the boundary numbers are this environment’s coordinates, not universal constants. The size and shape of the models and compute another team needs is their sizing exercise, and this page ships the instrument and method to run it. Two adjacent cells were deliberately not run and are named in the bound: the 26B MoE at Q4 and the E2B size rung. The ruling does not need them, and asserting cells you chose not to measure is how labs drift into marketing.

- **DO:** Put committed idle capacity to work on falsifiable batches behind a deterministic validator. Measured yield: 51 to 54% of attempts cleared as verified passes among the configurations above the measured boundary (numerators on the card), with the Lab 3 baseline at 47.8% on its own 29-task coverage. On the rented Granite Rapids shape, the full 117-attempt protocol produced 63 verified bug fixes in 2.24 hours: about nine cents of on-demand compute per verified fix, six cents at the measured one-year Savings Plan rate, and an incremental bill that can approach zero inside an otherwise-unused spend commitment.
- **DO:** Trust the gate, not the model. Zero confirmed false passes in 585 scored attempts across every configuration measured. Failures routed to the human queue or died at the output gate; no confirmed false success cleared the measured gate, and that claim extends exactly as far as the validator and its held-out evidence detect. The failure mode of this architecture is a person looks at it, not a token meter running.
- **DO:** Size down with a controlled descent. In this environment the measured model boundary landed at a 7.7GB artifact: Gemma 4 12B at Q4_K_M held the identical pass set as its bf16 original across three quantization tiers, measured on the Spark control. Substrate survival was proven at bf16, so the boundary config on rented Xeon rests on two measured edges rather than a measured cell; that bound is stated, not hidden. One rung lower broke, cleanly and honestly. Your boundary will differ; the screen-then-confirm ladder that finds it takes an afternoon.
- **DON'T:** Do not require the model to be deterministic, and do not read variance as defect. The deterministic component of this architecture is the validator. A variance-prone worker behind a deterministic gate fails closed against the defects the gate is built to detect, and on idle cycles its retries become mining: every additional pass it flickers into is checked against the held-out evidence before it counts.

**Independence and disclosure:** Disclosure: Intel is a client of this practice, and the lab corresponded with Intel about serving configurations during the program; the exchange is paraphrased where it appears and no vendor previewed or funded any of it. The bench ran on $28.32 of self-funded cloud spend plus owned hardware. No vendor paid for this answer.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Confirmed false passes | 0 | Across 585 scored attempts: three models (12B dense, 26B MoE, the E4B size probe), four quant tiers, two substrates. Every validator pass re-checked against hidden reference tests the model never sees. Fail-closed held everywhere, including at the broken tier |
| Per-attempt verified yield | 51-54% | The primary economic statistic, numerators visible: 12B on Spark 60/117 (51.3%), 12B on Xeon 62/117 (53.0%), 26B-Q8 63/117 (53.8%); Lab 3 baseline 43/90 (47.8%) on its 29-task coverage (57.3% restricted to the 24-task overlap, where the 12B reads 39/72 = 54.2%). Attempts are the unit of cost, verified passes the unit of yield |
| Cost per verified bug fix | $0.09 / $0.06 | 63 verified fixes in 2.24h: nine cents at on-demand, six at the measured 1yr Savings Plan rate. On owned idle hardware, marginal cost approaches incremental power and operations; inside an otherwise-unused Savings Plan commitment, the incremental bill can approach zero. Zero paid tokens in the loop, by construction |
| The boundary (this environment) | 12B at Q4_K_M, 7.7GB | Identical pass set to bf16 across three quant tiers. Bracketed from both sides: Q3 breaks below it on precision (net -5, triple-confirmed) and E4B breaks below it on size (15/39). Both breaks fail-closed. The descent method ships; your boundary is an afternoon away |
| The workhorse | 26B MoE, 2.5x faster | Same 21/39 yield as the 12B on identical silicon, 2.24h vs 5.54h for the identical protocol. A configuration result: architecture, precision, and runtime changed together; sparse activation is the leading explanation |
| Quality across the 2x2 | held, both axes | Model size (Run A, Spark control): 20/39 vs baseline. Substrate (Run B, Granite Rapids): 21/39, gained one, lost none. One variable moved per run; the validator pinned bit-identical throughout |
| Baseline comparison, both metrics | 17 vs 12 becomes 13 vs 12 | The 26B-on-Xeon versus the Lab 3 baseline, 29-task overlap: any-pass reads 17 vs 12, a scary 5-task deficit; unanimous-pass reads 13 vs 12, one task. Any-pass rewards the baseline’s own verdict flicker (4 flip tasks) against a zero-flip run. Never compare best-of-n to single-run across different variance profiles |
| Vendor-blessed config | 4.7x prefill, MoE: crash | llama.cpp-OpenVINO, Intel’s named path: real prefill gains on dense, 16% slower on bandwidth-bound decode, and a repeatable assert running the MoE. Stock llama.cpp remains the only stack that runs the MoE and engages AMX (hardware counters) |
| Reserved economics footnote | Savings Plans, not RIs | As measured 2026-07-15 (EC2 API, us-east-1), zero classic Reserved Instance offerings for m8i.12xlarge; the reserved analog is a Savings Plan ($1.68/hr 1yr no-upfront vs $2.54 on-demand). A Savings Plan commits dollars, not capacity — the math prices unused spend commitment and says so |
| Lab spend | $28.32 | All cloud legs, instances terminated and verified. The Spark legs and the entire quality ladder ran on owned hardware at zero marginal cost, which is the thesis performing itself |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 0 (Compute) | Retained / Delegated | The economic object is headroom you already carry, in two distinct forms the page keeps separate: owned idle hardware (the Spark) and unused cloud-spend commitment (the Savings Plan analog; as measured, no classic RI offerings existed for the shape — a Savings Plan commits dollars, not capacity). The lab prices that headroom, not a new purchase. Availability footnote: four AWS launches, zero quota friction, 16 to 29 seconds to SSH. |
| Layer 2B (Runtime) | Retained | The runtime is open source (vLLM, llama.cpp) and the control point is the validator and escalation policy, per the Loop Control ruling. The vendor-blessed OpenVINO backend was measured for scope: 4.7x on prefill, slower on bandwidth-bound decode, and unable to execute the mixture-of-experts graph at all. Stock stacks carried every measured cell. |
| Layer 2C (Reasoning) | Retained | Open weights, small and quantized: the boundary worker is a 7.7GB artifact. Nothing in the loop depends on a garden, an API, or a token meter, and the measured cost of that independence on this workload was a few points of yield against the strongest baseline reading. |

## What this does not establish

- It did not prove the boundary generalizes. Q4 as the floor and Q3 as the break are this environment’s coordinates: this model family, this task set, this validator. The page ships the method to find yours, not the claim that yours matches.
- The headline boundary config on the rented substrate rests on two measured edges, not a measured cell: substrate survival was proven at bf16 and the Q4 floor was proven on the Spark. Two adjacent cells were deliberately not run: the 26B MoE at Q4 (the workhorse claim stands on its measured Q8 cell) and the E2B size rung (E4B already breaks below the boundary). The E4B break itself is a single-screen declaration: its confirmation runs were canceled in the same close-out, with the mitigating fact that every lab 9 tier granted multiple runs was verdict-deterministic. All of it stated, none of it asserted beyond its evidence.
- It did not test heterogeneous batches or non-executable validators. One domain, chosen because its validator is deterministic; the architecture’s scope limit is the validator’s scope limit, exactly as Lab 3 ruled.
- It did not measure the market. Whether reserved Xeon floors are forming at hyperscalers, and why, are hypotheses the lab’s own capacity census declined to confirm; they stay off this page.
- The instrument has known edges, found by this lab’s own audit and disclosed above; three tasks carry no usable held-out evidence and six are structurally unpassable on all sides. Those nine sit inside the denominators quoted here: the 117 attempts are all 39 tasks at three passes each, so every rate on the card is depressed by defects the instrument owns rather than by anything the model did. The nine have not been repaired, and the yields would read higher on a clean pool.

## Open questions

- **Does the yield survive a second substrate class?** The economic structure is silicon-independent by construction: committed headroom plus a deterministic validator plus a human backstop applies to TPU, APU, and GPU reservations equally. The yield is a property of the model-substrate pair and must be measured per lane; labs 7 and 8 already showed the same model behaving differently across lanes. So the question a buyer should ask is not whether this generalizes across substrate classes, it is what their own committed lane yields, and the screen-then-confirm ladder above finds that in an afternoon. The generalization is worth earning, but nothing here waits on it.
- **Does the profile hold on heterogeneous work, and what does variance mining actually yield?** This lab ran one task domain with an executable validator, the domain where Deterministic Code In The Loop is strongest. Real batches are mixed, and validators weaker than pytest make the false-pass column harder to trust. Separately, the corrected framing implies an unmeasured strategy: a variance-prone worker retried on idle cycles mines additional verified passes. The yield curve of that mining, and where it beats simply running a bigger worker once, is an open measurement.

## Objection handled

**A 54% pass rate is failure half the time. Why not use a frontier model or a person?**

Because the denominator is free and the failures are honest. The 46% that fails costs idle cycles that were already bought and routes to the human who was the fallback before this system existed. The 54% that clears is verified work the human no longer does. The comparison is not this model versus a better worker; it is this yield versus the zero yield the same committed capacity produced last quarter.

The bigger worker does not change the answer. The 26B mixture-of-experts, the same class of model that anchored the original Lab 3 baseline, cleared the same 21 of 39 tasks on the identical substrate as the 12B, and the baseline itself sits within a few points on per-attempt yield. Within the overlapping task set and measured configurations, the larger worker bought no material verified-yield advantage; its measured advantage was throughput. Correctness lives in the gate.

And the frontier alternative changes the failure mode, which is the thing this architecture exists to control. A paid-token escalation path fails by accruing incremental charges. This loop fails by growing a queue; its infrastructure is metered too when rented, but no per-token charge accrues. For batch work on committed headroom, a queue is recoverable in a way an open-ended meter is not, and the zero confirmed-false-pass audit is what makes the queue trustworthy.

## Assessments at the time of the lab

| Vendor | Layer | Grade | As assessed |
| --- | --- | --- | --- |
| NVIDIA AI Platform | Layer 0 · Compute | NVIDIA Strength — Silicon Authority | May 22, 2026 |
| NVIDIA AI Platform | Layer 2B · Runtime | NVIDIA Authority — Inference + Agent Runtime | May 22, 2026 |
| NVIDIA AI Platform | Layer 2C · Reasoning | Runtime Governance Only — Not a Reasoning Plane | May 22, 2026 |
| Intel AI Infrastructure Portfolio | Layer 0 · Compute | CPU Strong, Accelerator Present-but-Thin, No Fabric | June 26, 2026 |
| Intel AI Infrastructure Portfolio | Layer 2B · Runtime | Open Runtime Tooling, No Managed Service | June 26, 2026 |
| AWS AI Infrastructure | Layer 0 · Compute | Custom Silicon Full Stack | July 12, 2026 |

## Supersession

- Corrects: loop-control (The escalation statistics. Nine invalid validator checks had booked escalation events for unsolved cases; corrected, the local model was clearing the solvable work on its own and the frontier tier bought throughput, not correct answers.)

## Summary

The lab collapsed to a single measured question on purpose. Labs 3 and 7 already priced the throughput of commodity Xeon and the behavior of the Loop Control architecture; what nobody had measured was whether a smaller model holds the success rate on the same problems, judged by the same pinned validator. Everything else is arithmetic. The bench filled exactly that cell, then kept filling cells the answers opened.

The controlled ladder ran one variable at a time. Run A changed only the model (12B dense versus the 26B reference, same Spark silicon): quality held at 20 of 39 tasks, and on the honest cross-run metric the 24-task overlap with the Lab 3 baseline reads 13 versus 13 unanimous-pass, dead even; the any-pass gap belongs to the metric caveat below. Run B changed only the substrate (the same 12B moved to a rented Granite Rapids shape): quality survived, gaining one task and losing none. The 26B on the same rented shape tied the 12B at 21 while finishing the identical protocol in 2.24 hours against 5.54. That is a configuration comparison, not a single-variable result: architecture, precision, and runtime moved together, with sparse activation the leading explanation for the gap.

The descent found the floor and bracketed it from both sides. Q8_0 and Q4_K_M held the exact pass set of the bf16 original, tier after tier; Q3_K_M broke by five tasks net, triple-confirmed; and the E4B size probe broke below the boundary from the other axis at 15 of 39. The two breaks have different textures worth knowing: precision broke toward failure-to-fix (more honest fails, little collateral), while size broke destructively (seven regression verdicts, five of them real broken-worse-than-found edits). Same safety property either way: at every tier, including both broken ones, the false-pass audit came back zero. Quality degrades closed on this instrument, all the way down — but what lands in the human queue differs in kind, and a team sizing its own boundary should look at both axes.

The deterministic component of this architecture is the validator, not the model, and the data kept illustrating why that placement is the thesis. The Lab 3 baseline flickered verdicts at temperature zero; the 12B ran verdict-identical for 117 straight traces; different silicon produced different tokens from the same weights. None of it mattered to the output, because reaching the output requires clearing a gate that behaves the same way every time. That is Deterministic Code In The Loop, priced: the worker is fungible, the gate is the asset.

## Detail

The instrument is the credibility, so it ships first. The task set is not synthetic toys: most of the 39 tasks are reconstructed from real bug fixes in real Python libraries, each pinned to a repo and commit, presenting the codebase as it stood with the historical bug present. Detection is layered, and each layer catches a different failure: feature tests verify the fix works, regression tests catch collateral damage, hidden reference tests the model never sees catch gaming, and a static gate kills malformed output before any code runs. The validator was pinned bit-identical across every run by manifest hash and by functional census, and the audit that verified it also X-rayed it: one weak test, two tasks without hidden references, six structurally unpassable tasks, all found by running the same audit against the original Lab 3 baseline, all carried like-for-like on both sides of every comparison. A lab that will not X-ray its own instrument has no business grading anyone else’s.

The design was a controlled 2x2 against the Lab 3 reference, one variable per step. Change only the model: the 12B dense held quality on the Spark. Change only the substrate: the same model held quality on rented Granite Rapids, and at temperature zero the different silicon produced different tokens with the same task-level verdicts, which is worth a sentence of its own: determinism is a property of a model-substrate pair, not of a model. Then the additions the data demanded: the 26B mixture-of-experts on the same rented shape tied the 12B’s yield while finishing in less than half the wall time, and the quantization ladder descended until it found the floor. Every cell was scored by the same pinned validator with the same false-pass audit. The one cell the ruling names that began as a composite, the 26B at Q4, was queued for measurement rather than asserted, because the difference between those two words is this program’s entire brand.

The false-pass column is the finding under the finding. A batch system that ships defects quietly is worse than no system, and the audit existed to catch exactly that: every validator pass, re-run against held-out tests. Zero confirmed false passes, everywhere, including at the tier where quality broke. When the Q3 model got worse, it got worse honestly: more failures routed to the queue, verdicts churning in both directions, nothing slipping through as false success. The architecture’s promise, that the failure mode is a person looks at it, held at every point the lab could measure, and the hidden-reference layer is what made that promise falsifiable rather than rhetorical.

The drift this lab caught in its own analysis is worth publishing because everyone will make it. Midway through, the analysis began treating model determinism as a virtue: celebrating zero-flip runs, framing the baseline’s verdict flicker as contamination. That inverts the thesis. In Deterministic Code In The Loop, the deterministic component is the validator; the baseline’s flicker is an argument for the architecture, because a variance-prone worker behind a deterministic gate is safe by construction, and on idle cycles its retries become verified mining. The corrected metric hierarchy ships on this page: per-attempt verified yield is the economic statistic, determinism is an operational property that sets retry policy, and comparing best-of-n numbers against single-run numbers across different variance profiles is a quiet corruption that likely infects more published model comparisons than anyone has checked.

The economics land where the thesis pointed. On the rented shape, a verified bug fix cost about nine cents of compute at on-demand rates and six at the measured Savings Plan rate. On owned idle hardware, the marginal cost approaches incremental power, cooling, and operations; inside an otherwise-unused Savings Plan commitment, the incremental bill can approach zero, subject to commitment coverage. Against that denominator, a 54% yield is not a grade on the model. It is recovered value on spend that produced nothing last quarter, with the remainder routed to humans who were always the fallback, and zero tokens metered along the way. The boundary and workhorse coordinates above are this environment’s answer. The instrument, the descent method, and the audit are the transferable part, and they are exactly what a team runs to find their own.

## Method and disclosure

Editorial and self-funded: $28.32 of cloud spend, all instances terminated and verified, plus owned hardware (the DGX Spark) for the control runs and the entire quality ladder. Disclosure: Intel is a client of this practice; the lab corresponded with Intel about serving configurations during this program, the exchange is paraphrased where referenced, and no vendor commissioned, funded, or previewed anything. The vendor-blessed serving path was tested because Intel named it, on the record, and the result published regardless of direction.

The instrument is the Lab 3 Loop Control harness, unmodified: 39 tasks (most reconstructed from real bug fixes in httpx, requests, dateutil, more-itertools, arrow, click, and other real libraries, pinned to repo and commit), layered deterministic tests including hidden references, pinned bit-identical across every run by sorted-manifest hash and functional census. Task contents and solutions stay private; the method, source repos, audit design, and every rate, count, and cost ship in the raw detail, including the full chronology with its own mistakes: an instrument-location error in the runbook, a hash recipe that had to be re-anchored, two serving-behavior confounds caught at smoke time, and one cleanup error on a pre-existing cloud volume, disclosed to the account owner the hour it happened.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# Cede the craft, keep the door

> Lab 010 · Editorial lab · Status: published · Author: Keith Townsend · July 21, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/two-agent-seam · Structured: https://labs.layer2c.com/labs/two-agent-seam.json

**Finding:** Two AI agents shipped a playable game across a file-based seam: the contract was fully delegable, the verdict was not, and you can only cede judgment you have already sourced.

**Question:** The pitch: a principal who owns only the outcome can compose two spiky AI agents into one shipped artifact across a file-based seam. The loss condition: a judgment the work needs that lives nowhere in the assembly, invisible until it ships.

**Load:** A transforming-mech browser game. One agent owned game logic and acted as the validator; a second owned art generation; they coordinated only through a file-based message bus. The playable result is embedded above.

**Raw lab detail:** https://labs.layer2c.com/downloads/two-agent-seam-lab-detail.md
**Layers:** Layer 2B (Runtime), Layer 2C (Reasoning)
**Themes:** validator-authority, authority-placement

## Verdict

**Scope:** At the scale of one browser game, with two current-generation coding and art agents, and a principal expert in neither craft.

- **DO:** Cede the craft you cannot judge to the agent that can, and place a seam that carries only the contract.
- **DO:** Retain the one authority you actually hold: the outcome the work is for. Here that was how the game should feel to play.
- **DON'T:** Do not assume a seam manufactures judgment. It must be sourced somewhere, or the structure has a void you will not see until it ships.
- **DON'T:** Do not trust borrowed agent judgment past the resolution of your own acceptance test. Below that line it is unaudited.

**Independence and disclosure:** Self-funded editorial, no sponsor. The build used Anthropic Claude Code (game logic and validation) and Google Antigravity (art generation). Disclosure: Google Cloud is a client of The CTO Advisor LLC, and the art in this lab was generated by a Google tool. No party reviewed, funded, or saw this lab before publication; the ruling is the author’s alone.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Sprite sheets generated across the seam | 10 | plus 4 reused single-frame assets, all validated |
| Rejections before acceptance | 1 | the first player sprite; every later delivery passed first try |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 2B (Runtime) | Ceded | The craft. Art generation and game-logic execution were handed to the agents. A deterministic contract gate enforced the mechanical part of each delivery; nothing below the acceptance test was audited. |
| Layer 2C (Reasoning) | Retained | The verdict. Is this good, is it right, does it play the way it should. Held by the human validator. Only judgments the principal had actually sourced could be placed at all. |

## Seam map — readiness (projected, pending bench)

| Function | Readiness |
| --- | --- |

## What this does not establish

- The art agent’s judgment was good enough this time, not proven good. It was validated only up to the resolution of the acceptance test; the layer below that rode on hope.
- The game is the friendly case, where the experience fully tests the ceded craft. The ruling does not extend to domains where ceded judgment can harbor defects the experience never surfaces.
- One game, two current-generation agents, one principal. The pattern is not shown to hold at team scale, across more than two agents, or where the crafts are not cleanly separable.

## Open questions

- **Where does player-legibility stop being the cede/retain boundary?** Here the experience was a total acceptance test for the ceded craft: if the mech reads and the fight feels right, the art’s internal quality is moot. In domains where defects do not surface in the experience, such as security, data integrity, or the fault that appears in six months, the ceded layer is unaudited, and that is where the deterministic gate stops being optional.
- **What is the minimum literacy a principal needs to place the seam?** Not the craft. The principal built neither the art nor the code. What he could not delegate was knowing which judgments the work required and whether each was present. That literacy is the floor. Get it wrong and every downstream seam is placed against a gap nobody in the room is qualified to notice.

## Objection handled

**Isn't this just delegation? Managers have handed work to specialists forever.**

Yes, and that is the point. The law is old: judgment has to be sourced or the structure has a void. It is why you hire, why there is a review board, why root-cause analysis exists.

What changed is the economics. A model call now fills a class of seams that used to require a hire, cheaply and fluently. But fluent and confident is the exact texture of judgment that hides its own gaps. AI lowered the cost of filling the gap and raised the cost of noticing when it is unfilled. The delegation is old. The camouflage is new.

## Summary

Two attempts to build the game with a single agent each failed on a different axis. One produced correct gameplay with weak art. The other produced strong art with broken gameplay. Neither was a weak engine. Both were a misplaced authority boundary: one agent asked to own two crafts it was only half-competent in.

The working run split the two crafts across two agents joined by a file-based message bus. One agent owned game logic and acted as the validator. The other generated art to a written contract, dropped each sheet at an agreed path, and posted the delivery to the bus. The validator checked it against the contract and either accepted it or replied with a specific rejection.

The deterministic half of the contract, the dimensions and transparency and frame counts, was enforced automatically on every delivery with no human in the loop. The first sprite passed every automated check and was still visibly wrong, a gray blob with no figure in it. The gate could not see that. A judging seat had to.

The principal directed the run without expertise in either craft. Every intervention was an outcome call: the weapon should fire in this form, the mini-boss should gate progress, the grounded form should carry more risk. Not once did he correct a palette or a frame. The art-quality judgment was ceded to the agent, and it happened to hold.

## Detail

The seam was a directory of JSON messages. Each agent was ephemeral and isolated, sharing no memory, so a durable file-based bus stood in for the channel between them. One agent posted a request with the sprite spec. The other claimed it, generated the sheet, and posted a delivery referencing the file. The validator read the delivery, checked the image against the contract, and posted an acceptance or a threaded rejection with the exact reason. No party spoke outside the bus.

Format compliance was fully mechanical and fully delegated. A sheet whose dimensions were not an exact multiple of the declared cell size was rejected by code, every time, with no judgment involved. That is the part a deterministic gate owns cleanly.

Quality was not mechanical. The first mech sprite cleared every automated check and read as debris. The rejection that fixed it was a human-grade note about silhouette, palette, and a hurt frame that was a solid red block instead of a damaged pose. That note is the control point. The bus can carry the contract, never the verdict.

The sharp limit is the resolution of the acceptance test. Borrowed agent judgment is only ever validated up to what the retained test can see. The test here was whether the game reads and plays right, and the art cleared it. Whether those sheets carry craft debt a real art director would flag on sight is unknown, because nobody had a test at that resolution. For a game it does not matter. In a domain where the defect does not surface in the experience, that unaudited layer is where the failure lives.

## Method and disclosure

Self-funded editorial. The playable build was produced by two AI agents coordinating over a file-based message bus. The code and art are public at github.com/kltownsend/talon-protocol-game under the MIT license. The coordination protocol and the validator harness stay proprietary.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# Buy the labor, not the judgment

> Lab 011 · Editorial lab · Status: published · Author: Keith Townsend · July 22, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/labor-not-judgment · Structured: https://labs.layer2c.com/labs/labor-not-judgment.json

**Finding:** One control architecture across 70 repairs. Within that single harness a frontier model added nothing as the loop's controller and lifted recovery from 14 to 20 as the worker. The deterministic validator still determines done, and the tier that matters sits in the worker seat, not the control seat.

**Question:** The pitch is that a local bug-fix agent needs a frontier tier in the loop to escalate to. I rebuilt the chain as originally intended: one loop, one deterministic test gate, one feedback contract, run unchanged across 70 real bug-fix pull requests, swapping only the model in the worker and controller seats and metering every call. A frontier model diagnosing the local worker's failures, as the loop's controller, recovered no net task a free deterministic feedback loop did not. The same frontier model doing the labor, as the worker, cleared work the local model could not. So the intelligence belongs in the worker seat, and the test still decides done.

**Load:** Seventy real bug-fix pull requests mined by git archaeology, each shipping the fix's own test suite as an unfalsifiable pass/fail gate. A local 26B reasoning model (Gemma, on a DGX) as the free worker; hosted GPT-5 mini and GPT-5.5 swapped into the worker and controller seats. One stateful repair loop and one deterministic validator held constant across all of it. The benchmark supplies the touched files, so this measures localized repair, not repository search.

**Raw lab detail:** https://labs.layer2c.com/downloads/labor-not-judgment-lab-detail.md
**Vendors:** NVIDIA (hw), Google (model), OpenAI (model), Anthropic (model)
**Layers:** Layer 0 (Compute), Layer 2B (Runtime), Layer 2C (Reasoning)
**Themes:** loop-control, validator-authority, agentic-repair, authority-placement, ai-factory-economics

## Verdict

**Scope:** Measured on deterministic coding repair, where a test harness gives an unfalsifiable pass/fail, and after the benchmark supplied file-level localization. The rulings are about where the determines-done authority and the required capability tier sit in an agentic loop. The open edges are autonomous repository debugging, domains without an executable evaluator, and larger replication.

- **DO:** run the local model as the worker and expect it to clear the solvable share. Most first-attempt misses were execution failures a stateful loop with deterministic feedback recovered for free, not gaps a bigger model was needed to close.
- **DO:** make the deterministic evaluator the authority. The test harness decides accept or reject, drives the loop, and supplies the feedback. No model judges its own work or another's.
- **DO:** when the hard tail needs a higher tier, spend it as the worker, not the controller. In one harness a frontier model doing the labor took recovery from 14 to 20; the same model merely diagnosing added nothing.
- **DON'T:** buy the frontier as cheap control over a weak local worker. A worker that cannot write the patch cannot act on the diagnosis. That quadrant paid the most and recovered the least.
- **DON'T:** read a single tuned loop as a general control result. This one held across 70 heterogeneous defects without bespoke orchestration. That is the claim, not one demonstration.

**Independence and disclosure:** Self-funded editorial. No vendor paid for this answer, reviewed it, or saw it before publication. The free local worker is Gemma, a Google model, run on owned NVIDIA hardware; the hosted arms were metered OpenAI calls, and an adjacent run, held out of the comparison, used a flat Anthropic subscription. Disclosure: Google Cloud is a client of The CTO Advisor LLC; NVIDIA, OpenAI, and Anthropic are not. The ruling is the author's alone.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Bug-fix pull requests, one architecture | 70 | the loop was never rewritten around a defect |
| First-attempt misses the free stateful loop recovered | 14 of 22 | execution failures, not capability gaps |
| Net tasks the frontier controller recovered over the free loop | 0 | zero across three seeds, 13-13, 10-10, and 11-10 on matched tasks; a trajectory-aware controller hinted at a faint edge, 13-11 on one subset |
| The same set with a frontier model in the worker seat | 20 of 22 | the worker's tier is the lever, not the controller's |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 0 (Compute) | Retained | The local first-attempt tier. A free 26B reasoning worker runs here on a DGX and clears the solvable share on its own, which is why the paid tiers buy less than the pitch claims. |
| Layer 2B (Runtime) | Retained | The test harness and the loop orchestrator are deterministic code you run, not a model. This gate decides accept or reject, drives the loop, and supplies the feedback. |
| Layer 2C (Reasoning) | Retained | The determines-done authority stays with the deterministic evaluator, never the model. Repair capability is delegated to the worker, and the finding is that the worker's tier, not a frontier controller diagnosing it, is what clears the hard tail. Buy the frontier's labor here, not its judgment. |

## Seam map — readiness

| Function | Readiness |
| --- | --- |

## What this does not establish

- The per-turn controller result holds across three seeds, 13-13, 10-10, and 11-10 on matched tasks, so it is no advantage seen in three runs, not one. That is still not an equivalence proof. The trajectory-aware controller's faint edge, 13-11 on a single 14-task subset, sits inside the sampling band and is a lead, not a finding.
- The benchmark supplied file-level localization, 21 of 22 tasks a single file, though those files ran a median of about 2,600 lines. This measures localized repair, not autonomous repository debugging, and cannot rule out that a frontier helps where fault localization is in play.
- The Claude Code result is adjacent, not a rung. It changed the harness and the model at once, so it attributes to neither, and any read of it as clean model separation is wrong. Lab 012 has since made the separation cleanly, and this caution stands as the record of what this lab could and could not claim on its own.

## Open questions

- **Where is the crossover in difficulty?** This lab thought it had located a graduated tail, reading one of five residual tasks for a hosted mini worker under frontier control against three for a frontier worker. Lab 013 replicated that subset and found it measures noise per task: the same model landed between one and four across observations, and which tasks solved barely overlapped. So the ladder is withdrawn. The residual five are currently unrated rather than hard, and where the crossover sits is open rather than answered.
- **Does the agentic harness or the model tier do the work?** An adjacent run put a top model inside Claude Code, a different agentic harness with tools and its own loop, and it cleared the whole set including the last two. But that swapped the harness as well as the model, so it proves neither. The next lab holds Claude Code fixed and drives it with a less capable model. If a weaker model in the same agentic harness still clears the tail, the harness was the lever; if recovery collapses toward these numbers, the model tier was carrying it. Answered: Lab 012 ran exactly that, and the harness was the lever. A mini-tier model inside Claude Code cleared all 22, twice, while plateauing at 16 and 17 in this lab's constrained loop. The Opus arm, re-run metered with real usage capture, came in at $13.79 across 10.96 million tokens; the mini finished identical work for $3.14.
- **Does a trajectory-aware controller earn its place where a per-turn one does not?** The per-turn controller tied the free loop across three seeds. Handed the full history of attempts and prior guidance instead, it edged the free loop 13 to 11 on one matched subset. That is the only sign in the whole run that a controller might add anything, and it is a single trajectory inside the noise band. One more seed decides whether it is real.

## Objection handled

**Is this not just "use a bigger model"? Of course a frontier worker solved more.**

Yes, as the worker, and that is the inversion. The pitch was to keep a cheap local worker and buy the frontier's judgment as loop control. That is the arrangement that bought nothing, 13 to 13 on the tasks both arms ran cleanly. The frontier only helped when it did the labor itself, which is the expensive thing the pitch was trying to avoid.

And the cost flips with how you buy tokens. Metered, you reserve the frontier for the hard tail. On a flat subscription its labor is free up to the cap, so you run it as the worker. During these runs the metered account hit its quota and stopped the experiment mid-run; the flat-plan arm ran through. The pricing regime is a first-class variable, not a footnote.

## Assessments at the time of the lab

| Vendor | Layer | Grade | As assessed |
| --- | --- | --- | --- |
| NVIDIA AI Platform | Layer 0 · Compute | NVIDIA Strength — Silicon Authority | July 23, 2026 |
| NVIDIA AI Platform | Layer 2C · Reasoning | Runtime Governance Only — Not a Reasoning Plane | July 23, 2026 |

## Supersession

- Corrected by: self-host-floor (The per-task residual claims. Reporting that the frontier worker cleared three of the five hardest tasks, and the difficulty ladder built from one-of-five and three-of-five readings, treat a single observation as a rating. Replication put the same model between 1 and 4 on that subset with barely overlapping tasks. The recovery figure of 20 of 22 and the ruling it carries are unaffected.)

## Summary

The pitch under test: a local bug-fix agent needs a frontier tier in the loop to escalate to. The re-run held one loop architecture, one deterministic test gate, and one feedback contract constant across 70 real bug-fix pull requests, and swapped only the model in the worker and controller seats.

The free local worker with stateful deterministic feedback recovered 14 of the 22 tasks it first missed. Adding a frontier model as the controller diagnosing those failures recovered no net task the free loop did not, 13 to 13 on the tasks both arms ran cleanly. The controller was the wrong place to spend intelligence.

The right place was the worker seat. In the same harness, moving from the local 26B to a frontier worker took recovery from 14 to 20 of 22. A weak worker cannot act on a good diagnosis. The tier has to hold the pen.

The economics split by how you buy tokens. Metered, reserve the frontier for the tail. On a flat subscription its labor is free up to the cap, so run it as the worker. The metered account hit its quota and stopped the experiment mid-run; the subscription arm did not.

## Detail

Four arms shared one harness: one model emits a search-and-replace edit, a deterministic gate replies with the test result, and the loop iterates up to four times. Only the model in the worker and controller seats changed. That shared harness is what makes the four comparable.

The free local worker recovered 14 of its 22 first-attempt misses on rich deterministic feedback alone. A GPT-5.5 controller reading each failure and advising the same worker recovered no additional matched task, and it held across three seeds: 13-13, 10-10, and 11-10 on the tasks both arms ran cleanly, never once ahead, for about ninety-eight thousand frontier tokens a run spent to draw even. That controller saw only the current failure. Handed the full trajectory instead, a fair variant, it nudged to 13-11 on one subset, a faint edge inside the sampling band, worth another seed but not yet a result.

Moving the frontier model out of the control seat and into the worker seat, same harness, took recovery to 20 of 22. It also cleared three of the five hardest residual tasks, which is a number Lab 013 later showed should not be read: replicating that subset put the same model anywhere between one and four, on barely overlapping tasks. The aggregate of 20 of 22 replicates and is what this finding rests on. The lever was never the loop or the diagnosis. It was which model held the pen. A weak worker cannot execute a good diagnosis, so buying the frontier's judgment as control is buying the wrong thing.

A fifth run cleared all 22, but it swapped the harness for Claude Code, an agentic loop with tools and file exploration, so it is held out of the comparison. It changed two variables at once and proves neither. Whether the agentic harness or the model did that work is a separate lab, and the clean way to run it is to hold Claude Code fixed and drop in a less capable model. That lab has since run. Lab 012 re-ran this fifth arm on the metered API with per-task usage capture: 22 of 22 again, $13.79, 10.96 million tokens. And the separation came back against the tier: a mini model in the same harness matched all 22 for $3.14. The harness was the lever.

## Method and disclosure

Self-funded editorial. The harness is loopcontrolbench, published open under the MIT license at github.com/kltownsend/loopcontrolbench: the reproduce-or-drop gate, the sandboxed runner, and the arm definitions. The vendor assessments and the layer placements this evidence feeds stay proprietary.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# Buy the harness, not the tier

> Lab 012 · Editorial lab · Status: published · Author: Keith Townsend · July 26, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/harness-or-tier · Structured: https://labs.layer2c.com/labs/harness-or-tier.json

**Finding:** Open weights on capacity I already carry clear 17 of 22 repairs for nothing, and the gate names the five they miss. Renting more hardware for the rest is slower and dearer than the API. The minimum that finishes the job is a mini model in an agentic harness, at $3.14. Two Opus generations finish the same 22 for four times that.

**Question:** An architect escalating repair work has rungs available: a free local loop, a paid mid-tier model in that same loop, then a frontier model inside an agentic harness. I priced all of them. Same 22 certified bug-fix repairs, same deterministic test gate, six models from a local Gemma to two generations of Opus, run both as a constrained loop and inside Claude Code. The free local loop clears 17. The paid mid-tier loop clears 16 and 17 across two runs, no better. The harness is what closes the rest, and it does that for a mini model as readily as a frontier one. So which rung is actually load-bearing, and what does each one cost?

**Load:** The 22 tasks Lab 011's free local loop first missed, each a real bug-fix pull request shipping its own test suite as an unfalsifiable pass/fail gate. Fourteen arms across two apparatuses: the constrained repair loop from Lab 011, and Claude Code driven headless through a translation proxy. Workers ranged from Gemma 4 26B and 31B on owned hardware, through hosted gpt-5-mini and gpt-5.4-mini, to Claude Opus 4.8 and Opus 5. Three arms were replicated. Every cost figure is directly measured: the mini arm from a dedicated API key's dashboard line, the Opus arms from per-task usage envelopes on the native API path. Total spend: about $121. $73.75 of rented infrastructure and $47 and change in metered tokens, including $5.45 killed by a mid-run credit exhaustion and roughly $24 of GPU left idling overnight. The mistakes stay in the bill because a reproducer pays for theirs too. The Spark's triage rung added nothing to any meter.

**Raw lab detail:** https://labs.layer2c.com/downloads/harness-or-tier-lab-detail.md
**Vendors:** NVIDIA (hw), Google (model), OpenAI (model), Anthropic (model), AWS (cloud), Google Cloud (cloud)
**Layers:** Layer 0 (Compute), Layer 2B (Runtime), Layer 2C (Reasoning)
**Themes:** loop-control, ai-factory-economics, validator-authority, agentic-repair, recovered-capacity

## Verdict

**Scope:** Measured on localized repair with an executable test, one task pool of 22, one week of vendor pricing. The rulings are about where capability enters an agentic stack and what each rung of an escalation ladder costs per verified unit of output. The open edges are repository-scale debugging without localization, domains without an executable evaluator, and variance bounds beyond two runs per arm.

- **DO:** triage on capacity you already carry. The free local loop clears three quarters of this work at zero marginal cost, and idle capacity comes in three forms: owned hardware, unused cloud-spend commitment, and unused subscription headroom.
- **DO:** escalate what the gate hands back into an agentic harness, and put the cheapest instrument-literate model in it. The mini cleared all 22 for $3.14; two Opus generations cleared the same 22 for four times that.
- **DO:** run the ladder in both directions. Escalate when the clock binds. Route overflow back to idle hardware when the budget binds. The gate makes both directions safe because a verified repair is verified regardless of which rung produced it.
- **DON'T:** buy a better model for the constrained loop. The paid mid-tier loop scored 16 and 17 across two runs against the free local model's 17 and 17. Same apparatus, no separation from free.
- **DON'T:** rent GPU to self-host the escalation rung. Agentic sessions are VRAM-bound, not compute-bound, so the rented class costs more per verified repair than the API and delivers less than half the capability. You cannot rent your way to efficiency on this class of model.
- **DON'T:** read token counts as a business metric. Three models spanning a fourfold price range landed within 13 percent of each other on tokens, and one model run twice against itself varied 24 percent. Cost per verified repair is the number that decides anything.

**Independence and disclosure:** Self-funded editorial. No vendor paid for this answer, reviewed it, or saw it before publication. The free local worker is Gemma, a Google model, run on owned NVIDIA hardware; the paid arms were metered OpenAI and Anthropic calls, and the rented-GPU arms ran on AWS. Disclosure: Google Cloud is a client of The CTO Advisor LLC; NVIDIA, OpenAI, Anthropic, and AWS are not. The ruling is the author's alone.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Configurations that cleared all 22 | 4 | gpt-5.4-mini twice, Opus 4.8, Opus 5, all inside the same agentic harness |
| Cost per verified repair, mini harness vs Opus | $0.14 vs $0.63 | $3.14 against $13.79 for identical finished work; both directly measured |
| Free local loop, two runs | 17 and 17 of 22 | paid mid-tier in the same loop: 16 and 17, no separation from free |
| Token spread across a 4x price range | 13% | inside the mini's own 24% run-to-run variance; the harness sets the token budget, the tier multiplies the rate |
| Rented-GPU self-host of the escalation rung | 9 of 22 | at $4-6 of machine time: more than the API, for under half the capability |
| Total spend, every meter, mistakes included | ~$121 | $73.75 rented infrastructure, $47 and change in metered tokens, including $5.45 lost to a mid-run credit exhaustion and ~$24 of idle GPU; a reproducer pays for their failures too |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 0 (Compute) | Retained | The triage tier and the overflow buffer. Owned hardware runs the free loop at zero marginal cost, and the procurement finding is negative: the rentable GPU class cannot beat it, because agentic concurrency is VRAM-bound and the meter scales linearly with the work. |
| Layer 2B (Runtime) | Retained | The deterministic test gate and the loop orchestrator are code you run, not a model. The gate is also the yield instrument: it is what makes cost per verified repair computable, and minimum viable capability a measurement instead of a judgment call. |
| Layer 2C (Reasoning) | Retained | The determines-done authority never moves. Repair reasoning is delegated to whichever worker is cheapest above the instrument-literacy bar, and it is safe to shop that seat aggressively, in both directions, precisely because the authority stays here. |

## Seam map — readiness

| Function | Readiness |
| --- | --- |

## What this does not establish

- Tier irrelevance above the bar is bounded by the noise measurement, not proved: 13 percent between models against 24 percent within one, on two runs. A wider replication could still separate them. What is excluded is any large token-efficiency gap of the kind the first draft of this analysis assumed.
- The subscription rung's near-zero marginal cost is structural arithmetic, not a measurement. No arm ran under subscription auth, and the caps are not publicly token-denominated, so the crossover is a bound.
- The task pool is the 22 repairs a specific free local model missed. That is the population an escalation decision actually sees, but it means every number here is conditioned on that model's failure profile, and the free rung's 17 of 22 does not transfer to other pools.
- A one-task probe of Opus 5 extrapolated to a $19 to $20 arm; the measured arm cost $12.56. Single-task probes establish pricing and nothing else. This entry's per-arm figures are arm totals for exactly that reason.
- Three local-harness arms had sessions interrupted by the two-hour per-task ceiling, and this entry scores those as misses. A later convention treats an interrupted session as no verdict rather than a failure, since the model was still working when the clock stopped. Under it the Gemma rows read 14 of 18, 9 of 19, and 11 of 21 instead of out of 22. No solve count changes anywhere, and no hosted or loop arm is affected, but the local arms are scored slightly harshly here.

## Open questions

- **How much work does a subscription seat's headroom actually hold?** The claim that idle Claude tokens do real work is structural here, priced off the rate card and the flat fee. The measured version runs an identical arm under subscription auth and meters cap consumption instead of dollars. That number would turn "spend headroom first" from a rule into a quantity, per seat, per week.
- **Where does the instrument-literacy bar actually sit?** Gemma 4 31B fails it and gpt-5.4-mini clears it, which brackets the bar but does not locate it. A descent through open-weight tiers inside the same harness would find the cheapest self-hostable model that still multiplies, and that model, not the frontier, is the interesting escalation target for anyone with idle VRAM.
- **Does the ladder survive losing localization?** Every task here arrived with its files named. Repository-scale fault localization is precisely the work an agentic harness should be best at and a constrained loop cannot attempt, so withholding localization should widen the harness's lead and may move the escalation boundary well below 17 of 22.

## Objection handled

**Twenty-two tasks and mostly single runs. Isn't "the tier doesn't matter above the bar" exactly the kind of claim that noise produces?**

The claim survives its own noise measurement, which is more than most tier comparisons attempt. Three arms were replicated: the free local loop held 17 twice, the paid loop moved 16 to 17, and the mini harness held 22 twice while its token count swung 24 percent. That swing is the calibration. The 13 percent token spread between a mini and two Opus generations sits inside it, so the honest statement is that the models are indistinguishable on consumption, and the fourfold cost gap is rate card. Solve counts replicate; individual tasks flip; token comparisons need error bars this wide. The entry says all three.

And the task pool is conditioned, deliberately. These are the 22 repairs a free local model could not clear on its first pass, which is the population that actually reaches an escalation decision. An unconditioned pool would flatter every paid tier by billing it for work the free rung would have absorbed.

## Assessments at the time of the lab

| Vendor | Layer | Grade | As assessed |
| --- | --- | --- | --- |
| NVIDIA AI Platform | Layer 0 · Compute | NVIDIA Strength — Silicon Authority | July 23, 2026 |
| NVIDIA AI Platform | Layer 2C · Reasoning | Runtime Governance Only — Not a Reasoning Plane | July 23, 2026 |
| AWS AI Infrastructure | Layer 0 · Compute | Custom Silicon Full Stack | July 23, 2026 |
| Google Cloud AI Infrastructure | Layer 0 · Compute | TPU + GPU Full Stack | July 23, 2026 |
| Google Cloud AI Infrastructure | Layer 2C · Reasoning | Productized Placement | July 23, 2026 |

## Summary

The pitch under test: routing repair work to a cheaper model saves money, and the counter-pitch that cheap models flail and cost more by the end. Both failed. Across 14 arms on the same 22 gate-verified repairs, capability entered through the apparatus, not the price column, and the cost of finished work tracked the rate card alone.

The ladder, priced per verified repair: free local triage clears 17 of 22 at roughly zero. A paid mid-tier model in the same loop clears 16 and 17 across two runs, no separation from free. The same mini inside Claude Code clears all 22 for $3.14. Two Opus generations clear the identical 22 for $13.79 and $12.56. Escalating only the five the gate names costs $1.12.

Renting GPU to self-host the escalation rung fails on both axes at once. Agentic sessions park context in VRAM while barely touching compute, so concurrency is bought with memory, and the box already on the bench holds most of a rented fleet's sessions for nothing. The rented arm cost more than the API and solved fewer than half the tasks.

The transferable rule is a dispatcher, not a ladder. Spend committed capacity first, at every rung it exists: idle hardware, unused cloud commitment, unused subscription headroom. Escalate to metered capability only for what the gate hands back, buy the minimum tier that clears the bar, and route overflow back down when the budget binds instead of the clock.

## Detail

Start on capacity you already carry. A Gemma 4 31B in a constrained loop clears 17 of the 22 on deterministic test feedback alone, and the gate hands back the exact five it could not solve. The earlier Xeon lab priced that rung: about nine cents per verified repair on rented Granite Rapids, six at the measured Savings Plan rate, and approaching zero on owned idle hardware or inside a commitment you already bought and did not use. This is not a story about owning a Spark. Unused cloud-spend commitment is idle capacity too.

The next rung up is the one that doesn't work. A paid mid-tier model in that same constrained loop scored 16 of 22, and 17 on a second run, against the free local model's 17 and 17. Two runs each, dead parity. Buying a better model without changing the apparatus around it bought no separation from free. That is the result that reframes the ladder, because it means the money is not in the tier.

It's in the harness. The same mini model that plateaued at 16 and 17 in the loop clears all 22 inside Claude Code. Six tasks recovered from apparatus, not parameters. That is where capability actually enters, and it is not exotic: read the repository, run the tests, read the failure, try again, with tools instead of a fixed protocol. Escalating just the residual five through it costs $1.12 against $12.56 for pushing all 22 through a frontier model, an eleven-fold spread for the same finished work. And even that $1.12 is a spot price. A flat-rate subscription seat with weekly headroom left over is idle capacity at this rung, the same way the Xeon lab's spare cycles were idle capacity at the one below. Idle Claude tokens can do real work too, and the marginal cost of the residual five inside an already-paid seat rounds to zero until the cap binds.

So should you buy the GPU class you can actually get? I priced it, and no. Agentic sessions are bought with memory, not compute: each one parks its context for its whole life while barely touching the die, so the honest unit is dollars per concurrent session per hour. Four entry instances give about nineteen sessions for $7.44 an hour. The box already on my bench holds fifteen for nothing. Renting the liquid class buys four more sessions at seven dollars an hour, which is not a purchase, it is a rounding error with a meter attached. The tier above is worse on the same axis and quota-locked besides, and it sells memory bandwidth against a per-task latency that is serial and irreducible. If you do rent, scale out with the smallest instance rather than up: consolidated multi-GPU boxes charge you for interconnect that independent sessions never touch.

What separates the rungs is what a retry costs, and that decides which model belongs on each. On idle capacity a retry is free, so a variance-prone cheap model is not a liability, it is a method: let it grind and let the gate certify each attempt. The Xeon lab audited 585 scored attempts and found no validator pass that failed its held-out checks, which is what makes grinding safe rather than reckless. On a metered API the arithmetic inverts. Every retry bills, so what you are actually buying up the ladder is first-pass success. That is the same property as instrument literacy, seen from the invoice.

One constraint keeps you honest, and it is why the escalation rung has to be bought rather than substituted. You cannot simply wrap the free local model in the harness to skip the paid step: Gemma 4 31B goes from 17 down to 11 inside Claude Code. The harness multiplies a model that can operate its instruments and taxes one that cannot. Instrument literacy is its own capability and it does not track parameter count. Above that bar the tier is close to irrelevant. Two Opus generations and the mini all cleared 22 on token budgets within 13 percent of each other, and the mini run twice against itself varied 24 percent, so the gap between models is smaller than the noise inside one of them. Pick the cheapest model that can drive the tools.

None of this is a new model of anything. It is AI Factory Economics run against a single Business Process Automation workload, with one unit of business output defined and everything denominated in it: cost per gate-verified repair. That framework warns that treating token consumption as a business metric is a trap, and I walked straight into it. The first version of this analysis was built on token efficiency, and measurement dissolved it. Tokens tracked nothing that mattered. The cost per finished repair tracked everything.

Scope it. Localized repair with an executable test, one task pool, one week of pricing. Three arms now have replicates, and they calibrate the rest: aggregate solve counts hold within a task of themselves, individual borderline tasks flip freely in both directions, and token budgets range from protocol-pinned in the loop, two runs within a fraction of a percent, to 24 percent apart in the harness. So trust the solve counts, treat any single task's verdict as weather, and read harness token comparisons only through noise that wide.

## How it abstracts

The shape generalizes to any Business Process Automation workload with a deterministic acceptance test. Triage on capacity you already carry, escalate only what the gate hands back, and spend committed capacity at every rung before buying metered capability: idle hardware, unused cloud-spend commitment, and unused subscription headroom are one category at three altitudes. Only when all of it is spent do you buy, and then the cheapest input that clears the yield bar.

The ladder also runs both directions. Escalate when the clock is the binding constraint. Route work back down when the budget is: an exhausted allowance or a capped subscription week sends overflow to hardware that costs nothing to keep busy, and the free rung becomes the buffer instead of the entry point. The gate is what makes both directions safe. A verified repair is a verified repair regardless of which rung produced it, so down-routing degrades only time and yield, both of which you can see, never quality silently.

The transferable instrument is the denominator. Cost per verified unit of output is what made every comparison here legible, and it is the one thing an architect has to define before any of this arithmetic runs. Without it, tier debates are unfalsifiable.

## Method and disclosure

Self-funded editorial. The harness is loopcontrolbench, published open under the MIT license at github.com/kltownsend/loopcontrolbench: the reproduce-or-drop gate, the sandboxed runner, and the arm definitions, including the Claude Code driver and the translation-proxy configuration added for this lab. The vendor assessments and the layer placements this evidence feeds stay proprietary.

This lab tests an existing model rather than proposing one. The AI Factory Economics Framework was published at thectoadvisor.com/blog/2026/01/07/from-ai-factory-metaphor-to-cio-grade-economics/, and every cost figure here is denominated in that framework's unit of business output: one gate-verified repair. The framework names token fixation as a failure mode; the first version of this analysis committed it, and the correction is reported in the writeup rather than quietly removed.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# The floor is trained, not sized

> Lab 013 · Editorial lab · Status: published · Author: Keith Townsend · August 5, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/self-host-floor · Structured: https://labs.layer2c.com/labs/self-host-floor.json

**Finding:** A 30B coder model on one box clears 18 of 22 gate-verified repairs. A 31B general model cleared 11. Seven parameters apart, same hardware, same gate, and what separates them is training for tools rather than size. The self-hosted floor is real and lower than expected. The bill just isn't in dollars: 18 of 22 took 11.4 hours of owned hardware against 40 minutes and $3.14 for all 22 on a hosted mini.

**Question:** Lab 012 bracketed the instrument-literacy bar without locating it. Gemma 4 31B failed it, a hosted mini cleared it, and everything in between was guesswork. If you already own the hardware, that gap is the whole decision. So I ran the descent: the same 22 certified bug-fix repairs, the same deterministic test gate, five open-weight models on one NVIDIA DGX Spark, each inside the same headless Claude Code harness. Which of them clears the bar Gemma missed, and what does self-hosting actually cost once you stop counting dollars?

**Load:** The 22 repairs Lab 011's free local loop first missed, each a real bug-fix pull request shipping its own test suite as the pass/fail gate. Six open-weight models served on one 128GB Spark through vLLM: Qwen3-Coder-Next in NVFP4, Devstral Small 2 24B, a Qwen general build, Gemma 4 26B in NVFP4, GLM-4.7-Flash in BF16, and GLM-4.5-Air in AWQ 4-bit. Eighteen arms across two apparatuses, three of them replicated, plus a five-task residual set run twice more to test whether the sharp end was measuring anything at all. Zero metered spend on the local arms. The cost is roughly 107 hours of owned GPU time and a power bill nobody metered.

**Raw lab detail:** https://labs.layer2c.com/downloads/self-host-floor-lab-detail.md
**Vendors:** NVIDIA (hw), Alibaba (model), Mistral AI (model), Zhipu AI (model), Google (model), OpenAI (ref)
**Layers:** Layer 0 (Compute), Layer 2B (Runtime), Layer 2C (Reasoning)
**Themes:** small-model-viability, model-ownership, ai-factory-economics, agentic-repair, validator-authority, recovered-capacity

## Verdict

**Scope:** Measured on localized repair with an executable test, one 22-task pool, one box, one week of open-weight checkpoints. The rulings are about which self-hosted model clears an escalation bar, and what the apparatus costs in wall clock rather than dollars. The open edges are repository-scale debugging without localization, domains without an executable evaluator, and any model too large for a single 128GB unified-memory box.

- **DO:** pick the model trained for tool use, not the biggest one that fits. A 30B coder build cleared 18 of 22 where a 31B general build cleared 11, on the same box and the same gate.
- **DO:** budget wall clock as the real cost of self-hosting. The best local arm ran 11.4 hours to finish 18 repairs. A hosted mini finished all 22 in 40 minutes for $3.14. That gap is what you're buying when you pay.
- **DO:** compute venue qualification against a correct serving config, not a default one. GLM-4.5-Air looked disqualified on this box until the serving config was fixed, then it ran and cleared 12 of 22. A wrong utilization setting will retire a model that works.
- **DON'T:** read a leaderboard rank as instrument literacy. Gemma 4 31B scored 17 of 22 in the constrained loop and 11 in the harness. GLM-4.7-Flash scored 4 and 13. The ordering inverts with the apparatus, so any single ranking of those two is wrong in one of the two settings.
- **DON'T:** trust one run on the hard tasks. Across five observations of the residual set, one task solved every time, one solved once, and the middle three sat near a coin flip. The same model scored 3 of 5 twice on almost disjoint tasks.

**Independence and disclosure:** Self-funded editorial. No vendor paid for this answer, reviewed it, or saw it before publication. The models under test are Alibaba's Qwen, Mistral's Devstral, Zhipu's GLM, and Google's Gemma, all run on owned NVIDIA hardware. Disclosure: Google Cloud is a client of The CTO Advisor LLC. Alibaba, Mistral, Zhipu, NVIDIA, and OpenAI are not. The ruling is the author's alone.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Best self-hosted result, one Spark | 18 of 22 | Qwen3-Coder-Next NVFP4, 22-minute median per task, 11.4 hours total, zero metered spend |
| The bar it cleared | 11 of 22 | Gemma 4 31B in the same harness, the model Lab 012 used to bracket the bar from below |
| Apparatus inversion, same two models | 17/4 then 11/13 | Gemma and Flash in the constrained loop, then in the harness; the ordering reverses |
| Wall clock, best local against hosted mini | 11.4h vs 0.6h | for 18 of 22 against 22 of 22; the hosted arm cost $3.14 and the local arm cost hours |
| Local medians per task, seven arms | 9 to 62 min | every hosted arm in this campaign ran 1 to 2 minutes on the same tasks |
| A 106B model on one box | 12 of 22 | GLM-4.5-Air AWQ 4-bit, sixth of seven, six repairs behind a 30B coder build a third its size |
| Residual-set replicates | 3, 3, 4, 1, 1 | five observations of the same five tasks; counts move wider than the seed band and task identity barely overlaps |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 0 (Compute) | Retained | One 128GB unified-memory box holds the entire local escalation rung at zero marginal cost, including a 106B model. The procurement finding is a qualification rule rather than a purchase: weights plus full-context key-value cache plus host headroom must fit before capability is worth measuring. The rule is only as good as the serving config it is computed against, and a default one retired a model here that in fact runs. |
| Layer 2B (Runtime) | Retained | The gate and the harness are code you run. Serving configuration turned out to belong here too, not to the model. The same checkpoint scored 13 and then 11 across a configuration change, inside the seed band but wide enough that no local number should travel without its serving config attached. |
| Layer 2C (Reasoning) | Delegated | Repair reasoning moves to whichever open-weight model is literate in the apparatus you're running, and that seat is genuinely shoppable now. The shopping criterion is training rather than parameter count. The determines-done authority stays in the gate, which is what makes the seat safe to shop at all. |

## Seam map — readiness

| Function | Readiness |
| --- | --- |

## What this does not establish

- Per-task residual claims are not supported by this data and should not be quoted. Five observations produced counts of 3, 3, 4, 1 and 1, and which tasks solved barely overlapped between runs of the same model. Aggregate solve counts replicate; individual task verdicts do not.
- The Gemma 4 31B row of 11 of 22 was measured on a rented L40S venue, not on the Spark that produced every other number in the comparison, and that arm's serving config and concurrency were never recorded. It also cannot be re-measured here. Dense 31B decodes at about 7 tokens per second on this box and the harness times out before the pool completes, so this arm cannot run on the owned Spark. An earlier version of this entry said the rented venue ran slower than the owned box, and that was backwards. What remains true is that the bar rests on a single run nobody can reproduce. A partial re-run of its 11 failures on the owned box solved 5 of them, which suggests the bar may sit low on some mechanism other than speed, but a partial arm does not produce a published number. Under this campaign's later convention, which scores a session interrupted by the time ceiling as no verdict rather than a miss, the row reads 11 of 21.
- No self-hosted model reached 22 of 22. The bar Lab 012 defined is cleared, the hosted tier's clean sweep is not matched, and the four-repair gap is uncharacterized. Whether those four are hard or simply unlucky is exactly the question the residual finding says this pool cannot currently answer.
- Serving configuration moves local numbers by about two repairs. GLM-4.7-Flash scored 13 and then 11 across a change that tripled its key-value cache. That sits inside the seed band, so it is not a capability claim in either direction, but it means no local number here should be read without its serving config.
- The wall-clock comparison is measured but not decomposed. Local arms ran 22 to 62 minute medians against 1 to 2 minutes hosted, and this lab did not separate serving throughput from harness overhead from retry behavior. The gap is real. Its cause is unattributed.

## Open questions

- **Does the coder-model advantage survive losing localization?** Every task here arrived with its files named. Qwen3-Coder's lead may be a code-reading advantage that repository-scale search would widen, or an artifact of already knowing where to look. Withholding localization separates those two, and it's the condition an enterprise codebase actually resembles.
- **Where does the wall-clock gap come from?** Local medians run roughly 30x the hosted ones on identical work, and this lab measured that without decomposing it. Serving throughput, tool-call round trips, and retry behavior are all candidates with different fixes. A per-turn breakdown would say whether self-hosting is slow because the model is slow or because the loop around it is.
- **What does a replicated difficulty rating do to the bar?** The residual set is measuring noise per task, which means this bench's hardest tier is currently unrated rather than hard. Running each task enough times to estimate a solve probability per capability class would give the bar a denominator. The tooling already exists in the harness and has never had the replicates to feed it.

## Objection handled

**One box, one task pool. Isn't "18 of 22" just this Spark's answer to this bench, with no reason to think it transfers?**

Partly, and scope says so. What transfers isn't the number, it's the inversion. Gemma and Flash swap places when the apparatus changes, 17 against 4 in the loop and 11 against 13 in the harness, and that result doesn't depend on the box at all. It says capability here isn't a single quantity, so "which local model is better" has no answer until you name the apparatus. Any architect picking a self-hosted model off a benchmark table is exposed to that inversion no matter what hardware they own.

The wall-clock finding transfers too, and it's the one that moves budgets. Every local arm ran a median between 22 and 62 minutes per task. Every hosted arm in this campaign ran between 1 and 2 on the same work. That's a 30x spread measured across six models and two vendors, wide enough that the specific box stops mattering.

## Assessments at the time of the lab

| Vendor | Layer | Grade | As assessed |
| --- | --- | --- | --- |
| NVIDIA AI Platform | Layer 0 · Compute | NVIDIA Strength — Silicon Authority | July 31, 2026 |

## Supersession

- Corrects: labor-not-judgment (The per-task residual claims. Reporting that the frontier worker cleared three of the five hardest tasks, and the difficulty ladder built from one-of-five and three-of-five readings, treat a single observation as a rating. Replication put the same model between 1 and 4 on that subset with barely overlapping tasks. The recovery figure of 20 of 22 and the ruling it carries are unaffected.)

## Summary

The question was where the instrument-literacy bar sits between a local model that fails it and a hosted one that clears it. It sits lower than the bracket suggested, and it isn't a size. Qwen3-Coder-Next, a 30B-class model quantized to NVFP4 on a single Spark, cleared 18 of 22 gate-verified repairs. Gemma 4 31B, a larger general model, cleared 11.

What separates them is training for tool use, and the evidence is an inversion. In the constrained edit loop Gemma scores 17 and GLM-4.7-Flash scores 4. Swap both into the agentic harness and the order reverses: Gemma drops to 11, Flash climbs to 13. Neither model is better. They're literate in different instruments, and a table that ranks them once is wrong half the time.

Self-hosting cleared the bar and still lost the economics, in a currency the rate card doesn't print. The best local arm spent 11.4 hours of owned GPU time to finish 18 repairs. A hosted mini finished all 22 in 40 minutes for $3.14. That's roughly 18x the wall clock to land four repairs short. Idle hardware makes the dollars free. It does not make the hours free.

One methodological result outranks any model ranking here. The five hardest tasks, the ones the bar was written against, are measuring noise per task. Across five observations one solved every time, one solved once, and the middle three landed near half. The same model took 3 of 5 twice on almost entirely different tasks. Aggregate counts replicate. Individual task verdicts don't, and a bar written as "solved task X" is reading a coin flip.

## Detail

Lab 012 left a gap nobody could spend against. It showed a free local model failing the instrument-literacy bar and a hosted mini clearing it, which brackets the bar without locating it. For anyone who already owns hardware, that gap is the entire decision. So this lab ran the descent: five open-weight models, one Spark, the same 22 repairs and the same deterministic gate, each inside the same headless Claude Code harness.

The floor is a coder model, and it sits lower than the bracket implied. Qwen3-Coder-Next in NVFP4 cleared 18 of 22. Devstral Small 2 and a Qwen general build cleared 15 each. Gemma 4 26B cleared 14, GLM-4.7-Flash cleared 13. Against Gemma 4 31B's 11, four of five models beat the bar, and the one that beat it hardest was trained specifically for code and tools.

Then the result that should change how anyone reads a model table. Gemma 4 31B scores 17 of 22 in the constrained edit loop, where the model gets one shot to reproduce an anchor from memory and no tools at all. GLM-4.7-Flash scores 4 in that same loop. Swap both into the agentic harness and it reverses: Gemma falls to 11, Flash rises to 13. Neither is the better model. Gemma has byte-exact recall and can't drive tools. Flash drives tools and can't reproduce anchors. Any ranking that puts one above the other is right in one apparatus and wrong in the other.

So what does self-hosting cost? Not money. Every local arm here ran at zero metered spend, which is the whole appeal. The cost lands in the column nobody prices. The best local arm spent 11.4 hours of GPU time to finish 18 repairs, at a 22-minute median and a 108-minute worst case. The hosted mini finished all 22 in 40 minutes for $3.14. That's about 18x the wall clock to end up four repairs short, and the spread holds across the whole field: local medians ran 22 to 62 minutes where every hosted arm ran 1 to 2.

That gap is the product you buy when you pay, and it deserves naming because "the hardware is already paid for" hides it. Idle hardware makes the dollars free. It doesn't make the hours free. If the work is unattended and runs overnight, hours are cheap and self-hosting wins outright. If somebody is waiting on the result, $3.14 buys back most of a working day.

One model looked disqualified and wasn't, which is the most instructive row here. GLM-4.5-Air, 106B parameters in AWQ 4-bit, was staged and appeared not to fit: an agentic harness needs 131K tokens of serving context, the key-value cache for that lives in the same unified memory as the weights, and at the utilization first tried there was no room left for sessions. That reading survived until the serving config was corrected. At util 0.80 the same weights on the same box yield 36.56 GiB of cache, 1.59 concurrent full-context sessions, and 12 of 22 solved. The box was never the constraint. A default `--max-num-seqs` was, the same setting that cost another model three times its cache. Venue qualification is still the right discipline, and it is only as trustworthy as the config it is computed against.

Air also lands where the headline predicts. At 106B it is three times the size of the model that beats it and finishes sixth of seven, one repair above a 31B and six below a 30B coder build. What it does own is speed: a 9.2-minute median against Qwen3-Coder's 22, and 4.5 hours for the full pool against 11.4, because 12B active parameters decode far faster than a dense 24B. Fast, and mediocre. Size bought throughput and no capability.

The last finding is about the bench rather than the models, and it invalidates a claim this campaign already published. The bar was written partly as "clears the residual tasks," the five hardest repairs in the pool. Replicating those five showed per-task outcomes close to random. One task solved in all five observations. One solved in one. The middle three sat near half. Qwen scored 3 of 5 twice on almost disjoint task sets, overlapping only on the task everything solves. The aggregate count carries signal. The task identity does not, so any claim of the form "this model solved that task" is reporting a coin flip somebody happened to watch.

Scope it. One box, one pool, one week of checkpoints, localized repair with an executable test. What carries off this bench is narrower than the numbers suggest: pick for training rather than size, name the apparatus before ranking anything, and price self-hosting in hours, because the dollars will tell you it's free.

## How it abstracts

Capability in an agentic stack is a pair, not a number. A model carries some literacy in each apparatus you might run it in, and those literacies are separably trained. Ranking models without naming the apparatus produces an ordering that inverts the moment somebody changes the runtime, which an architect can't see from a leaderboard and will feel in production.

Venue qualification precedes capability measurement, and it's computable in advance. For a one-box agentic deployment the arithmetic is weights, plus concurrent sessions times key-value cache per session, plus host headroom, against the box. The trap is that the same arithmetic run against a default serving config retires models that work: it disqualified a 106B model here that later cleared 12 of 22 on the identical hardware. Compute it against a tuned config, and read the cache figure off the server rather than deriving it.

Price self-hosting in the currency it actually costs. Dollars are zero on idle hardware and hours are not, and a 30x wall-clock multiple against a hosted tier decides whether free is worth it. That's AI Factory Economics with a different denominator: the same cost-per-unit-of-output discipline, run against time instead of spend, because on owned capacity time is the meter still running.

## Method and disclosure

Self-funded editorial. The harness is loopcontrolbench, published open under the MIT license at github.com/kltownsend/loopcontrolbench: the reproduce-or-drop gate, the sandboxed runner, and the arm definitions. This lab added a second agentic harness behind an environment switch, so the driver process can change while the prompt, workspace, gate and result schema stay identical, plus a trajectory reader that reports what a session actually did rather than only whether it passed. The vendor assessments and the layer placements this evidence feeds stay proprietary.

Every local arm ran on one NVIDIA DGX Spark with 128GB of unified memory, served through vLLM. Serving configuration is recorded with each result: the checkpoint path, the utilization setting, and the key-value cache the server reported at startup. Earlier arms in this campaign recorded only a proxy alias, which is why one arm in the record cannot name the weights that produced it. Closing that gap is what the provenance capture was added for.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# They can all write it. Not all of them take direction.

> Lab 016 · Editorial lab · Status: published · Author: Keith Townsend · August 11, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/tier-boundary · Structured: https://labs.layer2c.com/labs/tier-boundary.json

**Finding:** Every model above a 2B rung wrote a complete, plausible API implementation; the same model on the same task took three different approaches across three runs, and the approach predicted the score in every arm measured.

**Question:** The pitch is that picking the right model tier is the decision that determines whether delegated coding work succeeds. This lab walked a model ladder down from an 80B coder to a 2B, on one from-scratch task judged by a deterministic conformance gate, expecting to find the size where capability breaks. It did not find that. Above roughly 12B every rung wrote a structurally complete, plausibly organized server, and the same 26B model on the same task produced three different approaches in three runs, scoring 7 of 7, then 6 of 7, then 7 of 7, in 12, 79 and 21 minutes. The approach predicted the outcome. The size mostly did not.

**Load:** One from-scratch implementation task: a six-operation OpenAPI contract over a 438-chunk, 258-document corpus fixture, with no reference implementation in the workspace, judged by Schemathesis with all checks on (examples, coverage, fuzzing, and a stateful sequence suite). Scoring is per operation, seven units. Workers: gpt-5.4-mini, Qwen3-Coder-Next 80B-A3B, DeepSeek V4 Flash, and the Gemma 4 family at 26B-A4B, 12B QAT, E4B and E2B, all local rungs served on an owned NVIDIA DGX Spark and driven through the same agentic harness.

**Raw lab detail:** https://labs.layer2c.com/downloads/tier-boundary-lab-detail.md
**Vendors:** Google (model), OpenAI (model), Alibaba (model), DeepSeek (model), NVIDIA (hw), Anthropic (harness)
**Layers:** Layer 2B (Runtime), Layer 2C (Reasoning)
**Themes:** judgment-transfer, loop-control, small-model-viability, validator-authority

## Verdict

**Scope:** Scoped to one from-scratch API contract judged by one conformance gate, on a single owned box. The ladder is not evenly sampled: the 26B ran three times and gpt-5.4-mini twice, while the 80B, 12B, E4B and E2B rungs are single arms and cannot carry a distribution claim. The difficulty of this contract is concentrated in one operation, so the seven-unit score offers less resolution than its denominator suggests. Nothing here measures whether a passing implementation survives a downstream consumer, which is the number the migration control plane actually needs and which this lab does not touch.

- **DON'T:** Do not buy a model tier expecting it to buy you consistency. The 26B-A4B scored 7 of 7, then 6 of 7, then 7 of 7 on an identical venue, taking 12, 79 and 21 minutes. The 12B scored 6 of 7. On this contract the tiers overlap, and picking up a tier bought a narrower spread rather than a better ceiling.
- **DO:** Constrain the toolset instead of lengthening the playbook. Every arm that attempted large exact-match edits lost an operation; every arm that used small edits or none at all scored 7 of 7. The failure lived in editing strategy, a subject no API playbook would think to cover. Removing the operation removes the failure class, where another page of instructions would not have touched it.
- **DO:** Ratchet the artifact and cap the iteration. A repair loop closed five of six failures in a single turn once its feedback carried the gate's remediation text, then destroyed its own best result twice when left running: 1, 6, 3, 1. Keep the best artifact, refuse a regressing turn, escalate rather than iterate. The loop earns its keep exactly once.
- **DO:** Read variance as the deliverable, not as noise. Per-attempt verified yield was 2 of 2 for gpt-5.4-mini at about $2.08 and six minutes per verified unit, and 2 of 3 for the 26B at zero marginal dollars and roughly 56 expected minutes. The yield, not the ceiling, is what prices a retry policy.

**Independence and disclosure:** Disclosure: Google Cloud is a client of this practice, and the model family at the centre of this lab is Google Gemma 4. No vendor commissioned, funded, or previewed any of this, and no result was held or softened because of a relationship. The local rungs ran on owned hardware at zero marginal cost; the only money spent was a few dollars of API tokens, itemised on the card.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Same model, same task, three approaches | 7/7, 6/7, 7/7 | Gemma 4 26B-A4B NVFP4, identical venue, in 12, 79 and 21 minutes. Median exact-match edit target of 254 characters, then 6,112, then none at all. The approach tracked the score |
| Approach predicted the score | 8 of 8 arms | Every arm with a small or absent exact-match target scored 7 of 7; both arms with a median target above 6,000 characters scored 6 of 7. The one arm outside the pattern, E4B at 1 of 7, failed for a separately diagnosed reason |
| Judgment, measured | 6,397 to 19,023 to 19,252 | The failing run's response to three consecutive match failures was to enlarge the target. The passing run failed once at 1,648 characters and dropped to 282. Countable from a session log without judging any code |
| Harness cost at identical score | 63 min vs 4.1 min | Gemma 4 12B QAT reached 6 of 7 through the agentic harness and through a constrained loop. 1,040,460 input tokens against 29,614 total across all four loop turns |
| The repair loop, two feedback conditions | 1,1,1,1 vs 1,6,3,1 | Identical model, task and turn count. The only change was whether the feedback window carried the validator's remediation text or only its failure counts |
| Three defects, not one | E4B, 1 of 7 | A 9,410-byte server, longer than either 7-of-7 arm, carrying an escaping error that shredded 438 records, a missing Allow header on five operations, and two schema violations. Patching the escaping error alone is a one-line diff that clears the 404s and leaves the score at 1 of 7. Corrected after publication: the entry first attributed all six lost operations to the one character |
| One header, five operations | 12B, 1 of 7 to 6 of 7 | The clean case of doing the work and failing the gate. The 12B read the task, wrote its own 405 handler, and omitted the Allow header the task does not name. Patching only that header into its own handler took the same implementation to 6 of 7, and the model closed the gap itself one turn later once the feedback carried the remediation |
| Per-attempt verified yield | 2/2 vs 2/3 | gpt-5.4-mini at about $2.08 and 5.6 minutes mean per attempt; Gemma 4 26B-A4B at zero marginal dollars and 37.5 minutes mean, giving roughly 56 expected minutes to bank one verified unit |
| The bottom of the ladder | E2B, no build | Drove the agentic harness to a clean exit in three minutes, passed the tool-call gate, and produced a 108-byte lockfile and no implementation. The floor is real and it is below 4B |
| Controller footprint | 418 in / 270 out tokens | A frontier controller reading the gate and directing the next attempt costs roughly nine tenths of a cent per turn. It never writes code. Measured, not run to a conclusion: see the bound |
| Venue defects found and fixed | 12 | Every one produced a confident wrong reading, and every one pointed the same direction: at the model and away from the harness. Run unattended this descent would have reported a capability floor near 80B with every rung below it wrong |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 2B (Runtime) | Retained | The control point is the conformance gate and the escalation policy, not the model. The gate is an OpenAPI contract plus Schemathesis with all checks on, and it caught every defect on this page including a one-character escaping error inside otherwise-correct code. The harness itself is an authority surface the lab did not expect: an exact-match edit tool is a constraint on who can do the work, and it silently penalises models with lower reproduction precision. |
| Layer 2C (Reasoning) | Retained / Delegated | Open weights on owned hardware cleared the contract, so the reasoning plane can be retained. What cannot be retained by model choice alone is consistency: the same local rung delivered a verified unit on two attempts in three, with a 6.6x wall-clock spread. Delegating to a metered tier bought a narrower distribution rather than a higher ceiling, which is a different purchase than the one usually described. |

## What this does not establish

- It did not establish a tier floor. A 26B looked like one after a single arm scoring 7 of 7, and the claim did not survive the second arm. The correct statement is that this contract is reachable at 26B and not reliable there, and the sequence of asserting a floor on one run and losing it on the next is itself part of the finding.
- Four of the seven rungs are single arms. The 80B, 12B, E4B and E2B results cannot carry a distribution claim, and per-attempt yield is precisely the statistic that most needs a larger sample. Only the 26B (three arms) and gpt-5.4-mini (two) support any variance statement on this page.
- The contract concentrates its difficulty in one operation. Nearly every failure landed on the single endpoint carrying typed query parameters with date-format validation; the other six test whether a model can serve JSON from a fixture, which everything above the 2B rung can do. Partial credit out of seven is real but its resolution is largely illusory, and a pool that separates tiers needs its difficulty distributed deliberately.
- It did not measure downstream survival. The gate rules on conformance to a contract. Whether a passing implementation breaks a consumer that depends on it is the number the migration control plane needs, and it stays unmeasured here.
- The loop-control result is not a capability verdict. The 4B arm failed on delivery, not on judgment: the worker stopped writing files after turn 1 and the harness scored a stale artifact three times before that was caught. The finding that survives is narrow, that a follow-up prompt containing review feedback knocked a 4B out of tool-use mode, and it rests on one arm.
- The harness was the source of twelve defects during this program, all found and fixed, all disclosed in the method note. Their existence is a caution about every unattended descent, including the parts of this one that produced clean-looking numbers before the defect was found.

## Open questions

- **Does removing the exact-match edit tool close the variance?** The correlation is eight arms with no counterexample, and correlation is where observation saturates. The decisive experiment is one arm: run the 26B with the edit tool removed from its toolset. If per-attempt yield goes to three of three, the variance was a harness property and the local rung is materially more viable than this page reports, with expected time per verified unit falling from about 56 minutes to about 21. If the yield holds at two of three, the strategy correlation is coincidence and the judgment section on this page is wrong. That arm is unrun, and it is the first thing the next session should do.
- **Where is the floor for loaned judgment, as opposed to the floor for doing the work?** The loop-control arm put a frontier controller behind a 4B worker with the edit tool removed, and it did not answer the question. The controller performed correctly, producing accurate code-free diagnoses in about a thousand characters a turn. The worker stopped calling tools after its first turn, reasoned at length, and printed its answer into chat; the recovered chat implementation loaded the corpus correctly and registered no routes at all. So the observed failure is that receiving direction knocked a 4B out of tool-use mode, and that the second attempt was worse than the first. Whether a 4B can be steered at all, given a delivery mechanism that does not depend on it calling a write tool, is open. The harness now detects a no-write turn, which it did not when this arm ran.

## Objection handled

**A 4B model wrote a complete server and scored 1 of 7. Is this not just a demonstration that small models cannot code?**

No, and the failure is specific enough to say why. Gemma 4 E4B wrote a 9,410-byte server, longer than either arm that scored 7 of 7, correctly structured, with error handling and an in-memory index build. It carried three independent defects. It split the fixture on the two-character sequence backslash-n rather than on a newline, which shreds every JSON record at its escaped line breaks; it omitted the `Allow` header on unsupported methods, the same defect the 12B hit; and it violated the response schema on two operations. Patching only the escaping error, verified as a one-line diff, cleared the data-loading failures and left the score at 1 of 7, because the other two defects were already failing the same endpoints. Those are precision defects inside competent structure, not an inability to write the program.

The distinction matters because it changes what you buy. If small models produced incoherent output, the fix is a bigger model. If they produce plausible output with a fatal typo, the fix is a gate, because that failure mode is exactly the one human review handles worst. A reviewer reading a correct-looking loadCorpus function does not catch the escaping. The conformance run caught it in 1.4 seconds.

The honest limit is that the E4B arm is a single run, and its rerun failed differently. On the second attempt the same model wrote a server that loaded the corpus correctly and registered no routes at all, returning a well-formed 404 for every endpoint in the contract. Both failures are plausible-looking code. Neither is incoherence.

## Summary

The lab was staged to find the size where capability breaks and it did not find one. Above the 2B rung every model wrote a structurally complete, plausibly organised server: sensible naming, error handling, comments, an in-memory index built from the fixture. Whether that index was built correctly is where the rungs separate, and at the bottom it was not. The bottom is real and it is low. Gemma 4 E2B drove the agentic harness to a clean exit in three minutes and produced a 108-byte lockfile and nothing else. Everything above it produced a program.

What separated the arms was the approach each session happened to take. The same 26B model on the same task, same weights, same box, same contract, chose surgical edits on one run, whole-file replacement through an exact-match tool on another, and shell heredocs on a third. It scored 7 of 7, 6 of 7, and 7 of 7 in 12, 79, and 21 minutes. Across every arm measured, each one that attempted large exact-match edits lost an operation and each one that did not scored 7 of 7. There is no counterexample in the set.

The mechanism is legible in the trace. The failing run answered a 6,397-character match failure by trying 19,023 characters, then 19,252: it responded to failure by tripling the size of the string it could not reproduce. It then found the working approach at 53 characters, abandoned it, returned to roughly 6,100 and repeated that seven more times. The passing run failed once at 1,648 characters and its next edit was 282. One noticed the strategy was failing. The other did not.

The harness mattered more than the model. The same 12B reached the same 6 of 7 in 63 minutes through the agentic harness and in 4.1 minutes through a constrained loop that asks for whole files and does its own file handling. Fifteen times the wall clock, and roughly 35 times the tokens, for an identical score. Agentic scaffolding is priced for tasks whose inputs must be discovered, and this task hands its inputs over at the start.

The repair loop is worth exactly one turn. With feedback carrying only a failure count, four turns produced 1, 1, 1, 1. With feedback carrying the gate's own remediation text, the same model on the same task produced 1, 6, 3, 1: five operations closed in a single turn, then an overcorrection that flipped a check to its mirror image, then a rewrite that inverted a working handler's semantics. Iteration count did nothing. Feedback content did everything, once.

## Detail

The instrument is a from-scratch task, chosen because translation tasks hide the interesting failure. The workspace receives an OpenAPI contract, a task document and a 438-chunk corpus fixture, and nothing else; the reference implementation exists but is never mounted. Scoring is per operation across six endpoints plus a stateful sequence suite, judged by Schemathesis with all checks enabled. The task document states explicitly that every check the runner performs is part of the bar, including checks the contract does not spell out, and names HTTP method handling as an example. That sentence is load-bearing: it is what makes an unstated requirement a fair test rather than a trap.

The descent found no capability cliff, which was the expected result and did not arrive. gpt-5.4-mini, Qwen3-Coder-Next 80B-A3B and DeepSeek V4 Flash cleared the contract. Gemma 4 26B-A4B cleared it twice in three attempts. The 12B missed by one route in a 405 handler while passing 333 of 333 stateful scenarios. The E4B wrote the longest implementation of any arm and scored 1 of 7 on three small independent defects. Only the E2B failed to produce a program at all. From 2B to 80B, the thing that changed was not whether the model could write the program.

What changed was the density of small precision errors and, separately, whether the model noticed one. Those are two different axes and the lab was measuring neither at the outset. The precision axis is visible in the E4B's escaping error and in the 12B's missing route. The judgment axis is visible only in the trace of a failing run: an exact-match edit tool requires reproducing a target string character-perfect from memory, and one 26B session answered three consecutive failures by enlarging the target from 6,397 to 19,023 to 19,252 characters. It later found the working approach at 53 characters and abandoned it. The passing session, given the same tool and the same task, shrank its target after its single failure.

That is the migration playbook thesis under laboratory conditions. Give three engineers the same playbook and you get three approaches; here one model given one task document produced three. The approaches were not equally good, and the difference was invisible in the playbook's own vocabulary, which talks about contracts and status codes while the failure lived in how a file got edited. No amount of additional playbook detail reaches that axis. Which leaves the acceptance criteria as the only thing that can be held fixed across executors, and that is the argument for a deterministic gate arriving from the delegation side rather than the determinism side. It holds for human executors too.

The human analogy breaks in one place, and the break is the case for the gate. Three engineers taking three approaches can each say why, and each notices when theirs stops working. None of them answers a failed edit by tripling its size, twice, then discards the approach that worked. With human executors a validator catches the occasional bad approach. With delegated model work it is the only thing standing between a thrash spiral and a shipped 6 of 7 that passes 333 of 333 stateful scenarios and reads as finished.

Five control mechanisms came out of this, each forced by an observed failure rather than designed in advance. The gate catches a wrong result. An action-space constraint, removing the exact-match edit tool, makes thrash structurally impossible. A ratchet stops a regressing turn from overwriting a better artifact. An iteration cap stops unbounded self-directed retry. A thrash detector, N consecutive same-mode failures with no reduction in target size, is the correct trigger for escalation rather than iteration. The constraint has a cost the bench measured: removing the edit tool forces whole-file regeneration, and regeneration is what destroyed the repair loop's best result. The mechanisms are complements, not alternatives.

## Method and disclosure

Editorial and self-funded. The local rungs ran on an owned NVIDIA DGX Spark at zero marginal cost; the metered arms cost a few dollars of API tokens, itemised on the card. Disclosure: Google Cloud is a client of this practice and the Gemma 4 family is the centre of this lab, chosen because it is the only family cached locally at five sizes spanning dense and sparse, which holds tokenizer and training recipe roughly constant while parameter count moves. Anthropic's Claude Code is the agentic harness for every arm and is itself implicated in the findings, particularly the exact-match edit tool. No vendor commissioned, funded, or previewed any of this, and no result was held because of a relationship.

The instrument: a six-operation OpenAPI contract over a 438-chunk, 258-document fixture, judged by Schemathesis 4.24.3 with all checks on, scored per operation. Local rungs served on vLLM 0.25.1 at each model's native context with a per-model tool-call parser, gated before every arm by a tool-call smoke test. The contract, the fixture manifest, the scoring script and the full defect chronology ship in the raw detail; the corpus content stays private.

Twelve harness defects were found and fixed during the program and all of them are disclosed, because a descent is a venue audit that happens to produce capability data. In order: a 32,000-token output reservation that exceeded a small model's context before it saw the task; a context window served below native; an output budget consumed entirely by a reasoning model's thinking; an arm timeout sized against a fast rung; an output contract that required naming a JSON file with a comment, which npm cannot parse; a feedback window that truncated away the validator's remediation text; a driver reading a scoreboard instead of the detailed gate output; a 2,000-character context truncation that hid the worker's own working implementation from it; a hardcoded tool-call parser that fails silently with clean HTTP 200s; the same output reservation, correct for the small rungs, failing a large one mid-write; a controller returning an empty string silently; and a loop that scored a stale artifact three times because it never checked whether a turn wrote a file. Every one produced a confident wrong reading and every one pointed at the model.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# Inherit the boundary. Own the gate.

> Lab 015 · Sponsored lab · Status: published · Sponsor: Kamiwaza · Author: Keith Townsend  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/evidence-authority · Structured: https://labs.layer2c.com/labs/evidence-authority.json

**Finding:** Layers 1B and 2C are Ceded to Kamiwaza and the governance you cede to holds: 241 retrieval hits under adversarial cross-vendor querying, none crossed a workroom, and 42 of 42 claims grounded in the asking workroom's own corpus. What stays yours is the gate, because no role separates running the instrument from writing the corpus.

**Question:** A senior practitioner's method scales through people. Its credibility scales only if the evidence base stays under central control. So this lab builds the thing an advisory practice actually needs: a bench of analysts, one vendor each, all running the same assessment instrument over a corpus the curator vets. On Kamiwaza the pitch is that workrooms supply that boundary architecturally rather than through code you write. The loss condition: an analyst reaches into another analyst's vendor, or scores against evidence nobody vetted, while every boundary in the product reports success.

**Load:** A synthetic eight-layer assessment instrument, built only from published 4+1 material, applied to two vendors' public documentation. One corpus per workroom, one Kaizen agent per workroom, a third vendor named in questions and loaded nowhere as the confabulation control. Nine assessment runs, audited by a deterministic claim resolver.

**Video:** https://youtu.be/E-_zPBFsSdo
**Raw lab detail:** https://labs.layer2c.com/downloads/evidence-authority-lab-detail.md
**Vendors:** Kamiwaza (bench), Alibaba (bench), Dell (ref), Supermicro (ref)
**Layers:** Layer 1B (Retrieval), Layer 1C (Pipelines), Layer 2C (Reasoning), Layer 3 (+1) (Applications)
**Themes:** evidence-provenance, authority-placement, validator-authority

## Verdict

**Scope:** One deployment, one release, two workrooms, one assessment workload, one pinned model. Everything below is measured on that system, not projected to tenant scale.

- **DO:** Inherit the evidence boundary instead of writing one. Isolation is a property of the workroom, not of retrieval code your team maintains: 51 adversarial cross-vendor searches returned 241 hits and not one came from the other workroom, with the agent asking for the absent vendor by name throughout.
- **DO:** Own the gate. Build a deterministic corpus validator and run it before scoring rather than as a report afterward. It reads the corpus itself, so the party who contaminated it cannot delete their way out of the finding. This is your work on any platform.
- **DON'T:** Do not expect a role to express "can run the instrument, cannot touch the evidence." Both arrive on the same role and the vocabulary is a closed set. Price the process, not a permission.
- **DON'T:** Do not trust a status field on this deployment. A 201 has meant no index, a 200 no create, DEPLOYED a dead upstream, and terminal_outcome "indexed" zero vectors. Verify by reading state back.

**Independence and disclosure:** Kamiwaza sponsored this lab. They supplied a demo deployment and an administrative token, and they did not see the production instrument or choose the vendors. Kamiwaza reviewed the findings before publication for factual errors. Dell and Supermicro appear only as corpus material from their published press releases and neither is scored; Dell is part of the practice's vendor network. Kamiwaza was not scored either. The findings below include the ones they would not have chosen.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Grounded claims | 42 of 42 | Across six same-vendor runs, every claim resolved to a document in the asking workroom's own corpus with a verbatim quote. Checked by a deterministic resolver, not by reading. |
| Cross-workroom citations | 0 | No claim in either workroom cited, quoted or referenced the other vendor's material. |
| Retrieval hits under adversarial querying | 241, none crossed | 51 searches asking each workroom about the other analyst's vendor. Every hit came from the asking workroom's own corpus. |
| Reciprocal probe cells refused | 32 of 32 | Both workrooms asked to assess the absent vendor. Every layer returned insufficient evidence rather than filling the gap from training. |
| Invented capability claims | 0 of 75 cells | Including a null-probe vendor loaded nowhere. The model never asserted a positive capability it could not cite. |
| Ingest routes closed to a viewer | 4 of 4 | Document upload, collection create, vector database create, and direct vector insert. Identical at both token scopes. |
| Roles that separate reading from writing the corpus | 0 | WorkroomRole is a closed enum of owner, editor, viewer. The distinction cannot be expressed. |
| Corpus restore time | ~4 min per workroom | Eight documents, delete the collection and force-reload the vetted set, verified by enumeration. |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 1B (Retrieval) | Ceded | The canon places every 1B component here at Ceded, and nothing in this lab moves it. The retrieval path, the workroom binding, and the isolation model are Kamiwaza's, and an enterprise cannot lift them out and run them on another substrate without rebuilding. What the lab adds is the return on that cession, measured rather than assumed. Asked 51 times about a vendor whose corpus sits in the neighbouring workroom, retrieval returned 241 hits and every one came from the asking workroom's own corpus. The boundary is enforced at retrieval rather than left to the model's discretion, so the guarantee does not depend on which model is serving. Ceded authority is a good trade only when the thing you ceded to holds. Here it held. |
| Layer 1C (Pipelines) | Retained | The canon marks 1C a gap and Enterprise Responsibility, and the litmus says a vendor providing nothing leaves the layer Retained by default. That is the placement, and the lab sharpens what it means in practice. Curation policy is the enterprise's, and Kamiwaza's Ceded 2C machinery is what enforces it: a viewer was refused on document upload, collection creation, vector database creation, and the direct vector insert that bypasses the document pipeline, the last at a stricter relation than the others. Read-scoped and write-scoped tokens produced identical outcomes, so token scope is not an authorization boundary here and should not be treated as one. What the enterprise cannot do at this layer is express the separation it most wants, because no role distinguishes reading the corpus from writing it. |
| Layer 2C (Reasoning) | Ceded | The core of the trade, and the canon already called it: ReBAC enforcement and agent lifecycle governance are proprietary and captive. The lab measures the quality of what is being ceded. The agent is an artifact whose instructions and model binding are fixed at creation and travel with it, so isolation falls out of the same primitive: a principal holding editor in one workroom only, pointed directly at the other workroom's application URL, sees zero agents. The role travels from the workroom membership record into the application session intact and every launch is audited. The limit sits in the same place: the vocabulary is Kamiwaza's, WorkroomRole is a closed enum, and the intent an enterprise most wants to state at this layer is the one it cannot. The constraint surfaces legibly rather than failing silently, which is worth more than it sounds. |
| Layer 3 (+1) (Applications) | Ceded | Unremarkable and expected. Any vendor shipping an opinionated application scores this way, because the application's access semantics are its own opinion and cannot be lifted out. The canon grades the layer moderate with the Kaizen agent and App Garden at Ceded and the deployment patterns Delegated, and this lab found nothing that moves it. Worth naming for a buyer: the twenty-three tools the shipped agent carries come from the runtime image, agent-level filters had no effect, and the workroom deploy path exposes no options. On this deployment the surface granted nothing a role did not already hold, because reaching any of it requires a role that can already write. That makes it a reliability cost rather than a governance one. |

## What this does not establish

- Nothing here is a security result. Every account was authorized, acting in good faith, inside its own workroom. No privilege escalation or credential attack was attempted, and none of this speaks to isolation under an adversary.
- One release, one deployment, two workrooms, one assessment workload. Not a claim about behavior at tenant scale or for any other class of work.
- The isolation result is one run per direction, not a rate. It is a strong single measurement across 241 retrieval hits, not a repeated trial with a confidence interval.
- The cross-vendor questions were the instrument's own searches, not a frozen probe set with known documented answers authored before the workrooms opened. That measures conflation, which is what the hard requirement asks about. It does not deliver the sharper leak-candidate signal a frozen probe set would.
- Whether a curator can read another user's retrieval trail was not established. Reading it requires an application session, a personal access token is refused at that layer, and the bench held no owner-role password. That is a limit of the bench's access, not a property of the platform.
- Central update propagation was designed and never run. Neither half was tested: whether a document vetted into one workroom stays out of the other, and whether an instrument revision reaches both workrooms without per-workroom intervention. Redeployment rights, version pinning and instrument version verification were cut from the design and are also untested.
- The model-layer results are scoped to the one model this deployment served. Refusal behavior is a property of that model, not of Kamiwaza.

## Open questions

- **If nobody can operate the instrument without also being able to change what it reads, what does the process around it have to carry?** Answered, narrowly. It has to carry a deterministic corpus check positioned as a gate rather than a report, plus a rebuild path. Both worked here. The unresolved half is timing: the validator is point-in-time and the analyst holds write permission during a run, so the runbook rule is to bracket the run, validate before and after, and treat a post-run failure as invalidating that run's output.
- **Does a governed instrument stay inside its evidence when the model already knows the subject?** Answered for this model and this corpus. Two large infrastructure vendors with heavy public documentation, a model that can clearly produce a plausible assessment of either from training alone, and 32 of 32 cells refused when the evidence was absent. The parametric floor was zero. What that does not tell you is how a weaker refusal behavior would interact with the same platform, and the boundary result would survive it, because the isolation is enforced at retrieval rather than by the model declining.

## Objection handled

**An editor can write the corpus. That is basic access control, not a finding. So what does this lab actually tell me?**

Agreed, and that is the premise rather than the result. If an analyst can run the instrument, the analyst can change what it reads. No product decision fixes that, because the two capabilities arrive on the same role and the role vocabulary is owner, editor, viewer with no way to add a fourth.

So the question moves to detection and remediation, and both were measured. Detection: a validator that enumerates every collection in the workroom and diffs it against a vetted manifest caught a planted document deterministically. That control reads the corpus itself, which is why it cannot be evaded. Removing the record would mean removing the contamination.

Remediation is restore. Delete the collection, reload the vetted set with the pipeline force flag, confirm by enumeration. Twice demonstrated, about four minutes per workroom for eight documents. There is no per-document delete, so clearing one bad file means rebuilding the whole evidence base, and that cost is the thing to plan for.

The trap is where you put the check. Run the validator as a report and you are reading about contamination that already produced a score. Run it as a gate and a contaminated corpus never reaches scoring at all. Same tool, different place in the pipeline, and only one of them is a control.

## Summary

A senior practitioner's method scales through people. Its credibility does not, unless the evidence stays under one hand. Two analysts scoring two vendors against two different sets of documents produce two rows that cannot be compared, and comparability is the whole point of a matrix. So the thing worth testing is not the instrument and not the analysts. It is whether the platform underneath keeps each analyst inside the evidence a curator approved.

That got built and run. Two workrooms, a vendor corpus in each, one assessment instrument, and an agent per workroom that was asked, repeatedly and by name, to assess the vendor it had no documents for. It never did. Every claim it made across nine runs pointed at a document in its own workroom, and every quote was really there. More telling than the answers: the retrieval layer itself never returned a single document from next door, across 241 hits under queries designed to pull it across.

That is a property of the substrate rather than of the model behaving well. The boundary is enforced where evidence is fetched, not where the model decides what to mention, so it holds regardless of which model is serving. It is also the part a buyer cannot establish by reading documentation, which is why it needed a bench.

Now the trade. The canon already places Layer 1B and Layer 2C with Kamiwaza, Ceded, read from their own documentation months ago. Nothing here moves that. You cannot lift this governance model out and run it somewhere else, and inheriting a boundary is exactly what ceding authority means. What this lab adds is the return on the cession. Ceded is a good trade when the thing you ceded to holds, and under adversarial pressure it did.

The limit is not in the enforcement, which was solid everywhere it was pushed, including the direct vector insert that skips the document pipeline. The limit is in the vocabulary. There is no role that says "runs the instrument, cannot touch the evidence," because the roles are owner, editor and viewer and that is the whole list. So the analyst who operates the assessment can also rewrite what it reads.

Which relocates the control point rather than removing it. You cannot prevent contamination with a permission, so you catch it with a check: enumerate the corpus, diff it against the set you approved, and run that before anything gets scored rather than after. That check reads the evidence base itself, so the person who contaminated it cannot delete their way out of the finding. It is your code on any platform. It is the part you keep.

## Detail

What this adds to the canon assessment is a dimension a document review cannot reach. The canon graded Layer 1B strong and called it the Kamiwaza differentiator, reading the Context Manager, the living ontology and retrieval across distributed sources. That is a statement about capability being present. Whether that retrieval stays inside its boundary when an agent is actively trying to leave is a different question, and it is the one this lab answers. The result empirically supports the canon call rather than revising it.

The instrument was a synthetic eight-layer assessment built only from published 4+1 material. It carries none of the ratified grading rules, and nothing it produced publishes as a grade or feeds the canon. The grades exist so that something can move when the evidence base changes, and so a claim can be traced to a document.

Two workrooms were loaded with eight public press releases each, one vendor per workroom, verified by enumerating the vector store rather than by trusting the ingestion job's status. That distinction turned out to matter more than expected and is discussed below.

Groundedness was audited by a deterministic resolver rather than by reading, which is the Deterministic Code In The Loop split applied to the lab's own acceptance bar. Four questions are mechanical and the script decides them alone: does the cited document exist in that analyst's corpus, does the quoted span appear verbatim in it, does the citation resolve into the other analyst's corpus, and is the claim uncited. Only one question escalates to a human, which is whether the resolved span actually supports the claim. Across nine runs the mechanical gate handled everything except the judgement call, which is the control-point finding from the loop-control work showing up again in a different domain.

The resolver was wrong twice before it was right, and both errors ran against the model. It reported a fabricated quote where the model had wrapped a 299-character verbatim span in single quotes while the document used double quotes, and again where the model wrote "first to market" against the document's "first-to-market" with the other 480 characters identical. Under the instrument's own rules a fabricated quote is a finding about the model, so a resolver bug was one step from becoming a published accusation that the model invented a vendor executive's quote. It had not. A self-test with fixtures generated from the corpus now gates every audit, and it separates a mistyped span from an invented one, because those mean different things and only the second is an accusation.

The reciprocal probe is the measurement that carries the isolation claim. Refusal on its own proves nothing, because a workroom can refuse despite a leak, and reading the grades alone would score that as a pass. So the check counted what the retrieval layer returned rather than what the model said. Asked 25 times about Supermicro, the Dell workroom got 120 hits, all Dell. Asked 26 times about Dell, the Supermicro workroom got 121 hits, all Supermicro.

Model behavior sits in its own column and should not be read as a platform result. At temperature zero, seven of eight layers were identical across three runs of the same vendor, and one flipped to a full refusal in one run. Claim counts drifted within stable grades. That variance is an argument for the deterministic validator rather than against the platform: every claim passed the same mechanical gate regardless of which run produced it.

The operational finding that cost the most time is that status does not reflect reality, across several subsystems. A 201 on an upload that never indexed. A 200 on an agent create that never created. DEPLOYED on a model whose upstream returned 502. A pipeline job reporting terminal_outcome "indexed" for a document with zero vectors, because source tracking deduplicates on content hash and deleting a collection does not clear it. And a load that reported 201, completed, eight of eight successful, 193 chunks, into a second collection the agent never reads, because the collection name the read endpoint returns is not the name the write endpoint accepts. Every compensating control in this lab enumerates state rather than reading a status field, and each time that discipline lapsed it produced a wrong answer.

## Method and disclosure

Kamiwaza sponsored this lab. They supplied a demo deployment and an administrative token, and they did not see the production instrument or choose the vendors. They received a pre-publication report carrying every finding, every engineering item and the capability request, and reviewed it for factual errors before publication.

Dell and Supermicro appear only as corpus material, taken from their own published press releases. Neither is assessed here and no grade in this lab says anything about either company. Dell is part of the practice's vendor network, which is disclosed here because the documents are theirs, not because they had any involvement.

The corpus, the fetch tooling with its per-document hashes, the assessment runner, the deterministic claim resolver, its self-test, and the corpus validator are all committed. What stays proprietary is the calibrated assessment methodology: the ratified grading rules, worked reference rows, thresholds, and axis weighting. The instrument used here was built only from published 4+1 material and carries none of it, which is why nothing it produced can be read as an assessment.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# The second box works. The playbook doesn’t.

> Lab 014 · Editorial lab · Status: published · Author: Keith Townsend · August 8, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/second-box · Structured: https://labs.layer2c.com/labs/second-box.json

**Finding:** Two clustered DGX Sparks ran a 156 GiB mixture-of-experts checkpoint to 22 of 22 on the certified repair pool, matching a hosted frontier model at zero metered cost, after the vendor’s own multi-node playbook deadlocked at four concurrent requests.

**Question:** Buy a second small AI box and cluster it, and you get a bigger, faster tier for less than a bigger box costs. That is the pitch. Following the vendor’s documented procedure produces a server that answers one request and dies on four.

**Load:** Twenty-two certified Python repair tasks driven through a headless agentic harness, plus a concurrency sweep against the serving endpoint.

**Raw lab detail:** https://labs.layer2c.com/downloads/second-box-lab-detail.md
**Vendors:** NVIDIA (hw), DeepSeek (model), Google (model), Alibaba (model), Mistral AI (model)
**Layers:** Layer 0 (Compute), Layer 2C (Reasoning)
**Themes:** serving-economics, vendor-claim-scrutiny, model-ownership

## Verdict

**Scope:** Two NVIDIA DGX Sparks, GB10, 128GB unified memory each, joined by one 200 Gb/sec RoCE link. One mixture-of-experts checkpoint of 156 GiB, FP8 weights with an NVFP4 key-value cache, run twice on the certified pool with zero flips, one 31B dense model used to isolate the runtime, and three further models qualified and rejected. Everything downstream of that score is a single observation: the concurrency figures, the decode rate, and the nine eliminated configuration hypotheses were each measured once. Everything here is about small unified-memory boxes over RoCE. It says nothing about clusters built from cards a vendor validates for the purpose.

- **DO:** Buy the second box if you need a model that does not fit in one. A 156 GiB checkpoint served across two Sparks cleared 22 of 22 certified repairs at a 7 minute median, matching a hosted frontier model that cost $3.14 to run the same pool.
- **DON'T:** Do not budget the vendor’s multi-node procedure as a known quantity. The published DGX Spark playbook prescribes a Ray cluster that deadlocked here at four concurrent requests, and nine separate configuration fixes changed nothing.
- **DO:** Price the runtime as a dependency. The configuration that worked runs on a community-maintained base image, pinned by digest and verified against it, with a thin patch layer on top. That is a supply chain position rather than a vendor-supported one, and somebody has to own it.

**Independence and disclosure:** Self-funded. Both Sparks and the interconnect were bought by the practice at retail, and no vendor saw this before publication or had any say in it. NVIDIA hardware is under test and NVIDIA documentation is criticised here by name. The serving stack that worked is a community-built container image, not a vendor artifact, and that is part of the finding rather than a footnote. Disclosure: Google Cloud is a client of this practice, and a Google model, Gemma 4 31B, is the isolation subject behind the runtime finding. Google did not commission, fund, preview, or have any input into this lab. No other vendor named here is a client.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Certified repair pool, DeepSeek V4 Flash clustered | 22 of 22, twice | Two runs on the pinned configuration, 22 of 22 both times, zero flips. 4.49 h and 4.19 h; 869 and 858 turns; zero timeouts in either. Every verdict from the deterministic gate. |
| Best single-box result on the same pool | 18 of 22, twice | Qwen3-Coder-Next NVFP4, one Spark. Replicated: 18 of 22 on both runs, with six tasks flipping three each way. Wall clock did not replicate. The first run took 11.4 hours on a venue whose serving config was never recorded; the second took 4.8 hours on a recorded one, doing 125 more turns. Compare the clustered figure to the score, not to the hours. |
| Hosted reference on the same pool | 22 of 22 | gpt-5.4-mini, 38.4 minutes, $3.14 metered. |
| Clustered wall clock | 4.49 h total | Median 7.0 minutes per task, max 48.6. Turns: median 20, max 208. |
| Concurrency sustained, DeepSeek V4 Flash | 16 of 16 | 11,200 tokens in 52 seconds, 215.4 tok/s aggregate, 13.5 per stream. |
| Single-stream decode, DeepSeek V4 Flash clustered | 35 tok/s | The same hardware serves Gemma 4 31B, dense, at about 7 tok/s on one box. |
| Vendor-documented path, stock runtime, Gemma 4 31B | 0 of 16 | Deadlock at four concurrent and above. Nine configuration hypotheses eliminated, all failing at the same shared-memory read. |
| Same Gemma 4 31B, working runtime | 16 of 16 | Zero errors, 169.7 tok/s aggregate. Model, hardware, parallelism and fabric settings held fixed. Only the image changed. |
| NCCL all-reduce at DeepSeek V4 Flash activation size | 49.6 us | Raw RDMA write latency is 9.43 us at 64KB. Predicting collective cost from the raw figure understates it five times over. |
| Models qualified for the pair | 1 of 5 | DeepSeek V4 Flash serves. Qwen3-Coder 480B-A35B needs 273 GB before any cache. Qwen3-235B-A22B-2507 fails engine init at width two. Devstral 2 123B clusters at 5.8 tok/s. Nemotron-3 Super 120B runs one box at 74 minutes per repair. |
| Clustering by architecture | dense 2 of 2, MoE 1 of 2 | Clustered here: Gemma 4 31B and Devstral 2 123B, both dense, both work. DeepSeek V4 Flash works on an image built for its architecture; Qwen3-235B-A22B-2507 fails engine init at width two. A community report of Qwen3-VL-30B failing the same way at width two is corroboration, not one of these counts. |
| Metered spend | $0 | Power and roughly a day of setup. The capital was already committed. |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 0 (Compute) | Retained | Compute and fabric are owned outright. The finding is that owning them is not sufficient, because the software path to using them together is not vendor-supported in practice. |
| Layer 2C (Reasoning) | Retained | Reasoning runs on owned hardware with no metered call. The dependency that remains is the container image, which is community-built. |

## Seam map — readiness

| Function | Readiness |
| --- | --- |
| Serve a model larger than one box | ready |
| Concurrent agentic serving | ready |
| Follow the vendor multi-node playbook | needs-cloud |

## What this does not establish

- Replicated, and worth stating precisely. The pool was run twice on the pinned configuration and returned 22 of 22 both times with zero tasks changing hands. The pool does move: Qwen3-Coder-Next scored 18 of 22 on both of its runs with six different tasks flipping, three each way. The variance is real and this model did not show it here. Two runs is still two runs, and neither says anything about a third model, a different pool, or a rebuilt image.
- The build is isolated as the variable, but not explained. The 31B model that failed nine times on the stock image served 16 of 16 concurrent on the working image with the same hardware, parallelism, fabric settings and launch shape, so the runtime is what changed the outcome. Why it changed is not established here. No claim is made about which commit, patch, or flag is responsible.
- Wall clock is not comparable to the other arms in this corpus. The harness ran at concurrency two because the box hosting it also holds half the model, and earlier arms ran at four and six. Solve counts are comparable. Timings are not.
- Nothing here is a statement about clustering generally. This is two small unified-memory boxes over a single RoCE link. It says nothing about clusters built from cards a vendor validates for multi-node work.
- The working path depends on a third-party container image. Its base layer is pinned by digest and was verified byte-for-byte against that digest before the build proceeded. The derived layer, which applies a tokenizer patch, produced a different image ID here than the reference build records, and the build verified it by content and tokenizer-conformance tests instead. That is ordinary Docker behaviour rather than a defect, and it is not evidence about whether the maintainers can reproduce their own build. No claim is made about the project’s provenance or its maintenance.
- No claim that the vendor playbook is broken for every model or every version. What was measured is that its prescribed path deadlocked here, on this hardware, at this version, and that nine plausible fixes did not recover it.
- The dense-clusters, mixture-of-experts-does-not pattern rests on four models actually put across the link: two dense, both of which clustered, and two mixture-of-experts, of which one did. Four is a small sample supporting a mechanism, not a proven law. The community record shows the same split and a third mixture-of-experts failure, but that is somebody else’s measurement and is not counted here. A single counterexample would overturn the pattern, and only one runtime build was tried against the failures.
- Correction, made after publication. This lab originally described the served model as a 671B mixture-of-experts checkpoint at NVFP4. Both figures were wrong. The weights are FP8, e4m3 with ue8m0 block scales in 128 by 128 blocks, read from the checkpoint config; the NVFP4 in the serving stack is the key-value cache format, and the two were conflated. The 671B was never sourced: no parameter count appears in the config, the checkpoint, or the upstream repository, and 671B is DeepSeek V3 and R1’s figure, carried across by name. The checkpoint is 156 GiB, which is what the two-box requirement actually rests on, and no parameter count is claimed here in its place.
- The 235B mixture-of-experts was not disqualified on capability or on capacity. It fits, it carries the context the harness needs, and it never ran a task. It failed engine initialisation on the one image that does multi-node here, which is a statement about that image rather than about the model.
- Every wall-clock figure in this entry is venue-dependent in a way the scores are not. Re-running Qwen3-Coder-Next on a recorded serving config reproduced its 18 of 22 exactly and cut the arm from 11.4 hours to 4.8, while doing more turns, with 21 of 22 tasks cheaper per turn. The original venue was an ad hoc launch whose logs were destroyed, so the difference cannot be attributed. Read the hour counts here as one measurement of one configuration, not as a property of the model.
- That wall-clock instability has a control, and it points at the record-keeping rather than the hardware. DeepSeek ran from a pinned kit with a digest-verified image and a written config, and it reproduced its score and its clock together: 4.49 and 4.19 hours, 869 and 858 turns. Qwen ran from an ad hoc command nobody saved, and it reproduced its score while missing its clock by 2.4x. Same two boxes, same harness, same pool. The difference between the two is whether the venue was written down.

## Open questions

- **Which build change makes the difference, and can it be carried back to the stock engine?** The variable is isolated but the mechanism is not. Running Gemma 4 31B on the working image with an otherwise identical launch gave 16 of 16 concurrent, so the build is the operative difference. What that build does differently is still open: it logs the same shared-memory starvation the stock engine dies on and recovers instead. Whether that is a point-release fix, a patch the image carries, or a build flag is the next thing to read out of the diff.
- **Does 22 of 22 replicate?** Yes, and cleanly. The pool was run a second time on the same pinned configuration and returned 22 of 22 again with zero tasks changing hands, at 4.19 hours against 4.49 and 858 turns against 869. That matters because the pool does move: Qwen3-Coder-Next scored 18 of 22 on both of its runs with six different tasks flipping, three each way. So the plus or minus two is real and this model did not show it. What remains unreplicated is everything downstream of the score. The concurrency figures, the decode rate, and the nine failed configuration hypotheses are all still single observations.
- **How many models actually qualify for this tier, and does the mixture-of-experts wall move?** One of five qualified here, and the exclusions were four different failures rather than one. The generalisation worth testing is that dense models cluster and mixture-of-experts models mostly do not, because only the latter need all-to-all expert dispatch through version-pinned fused kernels. That rests on five attempts and a community record showing the same split, which is suggestive rather than settled. The specific next test is cheap: one more mixture-of-experts model on a newer runtime build, accepting that a newer build reopens the shared-memory deadlock this one was chosen to avoid. If the wall moves with the runtime, the constraint is a software generation and it will age out. If it does not, it is architectural and it belongs in the purchase decision permanently.
- **Is one box enough?** Not by filling it. The obvious move is to run the largest model that fits in 128GB, and that was tested: Nemotron-3 Super 120B-A12B, on corrected serving config, on a box to itself. It took 74 minutes for a repair that Qwen3-Coder-Next, 80B total with about 3B active, completed in 7 on identical hardware, and lost the next task to a two-hour ceiling. The arm was stopped on economics rather than on capability, because a model an order of magnitude slower per verified repair cannot pay for itself whatever it eventually scores. One box favours a smaller, faster model. The bigger model needs the second box for reasons that have nothing to do with whether its weights fit.

## Objection handled

**Two boxes and an interconnect to match something you can rent for $3.14. Where is the return?**

The rental price is per run. The pool took 38 minutes and $3.14 on the hosted model, and it will cost that again next time, and the time after. The clustered pair took 4.5 hours and nothing, on hardware that was already bought.

So the trade is wall clock for marginal cost. Seven times slower, and free after the capital. Where that lands depends on how often you run and whether the work can wait, which is a question about your pipeline rather than about the model.

The stronger objection is the one about the single box. Qwen3-Coder-Next cleared 18 of 22 on one Spark with no interconnect and no clustering, and it did it twice, with different tasks failing each time. If the clustered pair had landed at 18, the second box would have bought nothing. It landed at 22, and it took the one repair no self-hosted model in this corpus had ever solved.

What does not survive scrutiny is the idea that this is turnkey. Getting from two boxes to a working endpoint took a day, and most of that day was spent on a documented path that does not work.

## Summary

Two DGX Sparks were clustered over a 200 Gb/sec RoCE link and measured serving DeepSeek V4 Flash 0731, a 156 GiB mixture-of-experts checkpoint with FP8 weights and an NVFP4 key-value cache, tensor-parallel across both nodes. The pair sustained 16 concurrent requests with zero errors at 215 tok/s aggregate, and cleared 22 of 22 tasks on the certified repair pool with no timeouts and no session errors.

The same hardware, running the vendor-documented configuration on a stock vLLM build, deadlocked at four concurrent requests. Nine configuration hypotheses were tested against an identical load and every one failed at the same point: a shared-memory read timeout inside the inter-worker synchronisation path.

The difference between the two outcomes was the runtime build, not the hardware, the fabric, or the orchestration framework. Both documented multi-node paths failed identically on the stock image. Holding the model fixed and changing only the image settles it: the same 31B model that failed nine times served 16 concurrent requests with zero errors at 169.7 tok/s on the working build.

The failure mechanism survives in both builds. The working image logs the same shared-memory starvation, a 60 second wait for a broadcast block, as an informational line and continues. The stock image raises a timeout from the same call and kills the engine. One build tolerates the condition; the other does not.

Interconnect capability was never the constraint. A NCCL all-reduce at this model’s activation size measures 49.6 microseconds, which puts collective overhead at roughly 8 percent of the token budget at tensor-parallel width two.

Five models were qualified against the pair and one served. DeepSeek V4 Flash 0731 is the one that works. The others failed for four different reasons: Qwen3-Coder 480B-A35B needs 273 GB at four-bit, more than the pair holds before a single token of cache; Qwen3-235B-A22B-2507 fits easily and serves on one node but fails engine initialisation at tensor-parallel width two; Devstral 2 123B fits and clusters and then decodes six times slower than the model that works; and Nemotron-3 Super 120B runs on a single box at ten times the wall clock per repair. Fitting in the combined memory is a necessary condition and not close to a sufficient one.

The pattern across those attempts is that sparsity is what admits a large model to this tier and expert routing is what excludes it. Both dense models clustered here, Gemma 4 31B and Devstral 2 123B, did so without incident. Of the two mixture-of-experts models put across the link, only DeepSeek V4 Flash clustered, and it runs on a container image built specifically for its architecture. Tensor parallelism needs a per-layer all-reduce, which every runtime implements well. Mixture-of-experts additionally needs all-to-all expert dispatch through fused kernels bound tightly to hardware generation and library version, and dense models never enter that path.

## Detail

The pitch for a second small box is straightforward. A model that will not fit in 128GB fits in 256GB, the interconnect is fast enough, and two boxes cost less than one larger machine. Lab 013 closed by naming exactly this gap, and the second Spark was bought to close it.

The vendor publishes a playbook for this. Its multi-node section prescribes a Ray cluster with tensor parallelism across both nodes, and that is where this lab started. On a stock vLLM build serving Gemma 4 31B, dense, that configuration answered a single request at 11.2 tok/s and then deadlocked the moment four requests arrived together. One node sat at 96 percent reported utilisation drawing 20 watts, which is a busy-wait spin rather than compute, while the other sat at zero.

Nine configuration hypotheses were tested against that identical load. The engine watchdog was raised from 300 to 1800 seconds. Step size was capped so no forward pass could approach the timeout. The RoCE GID index, which was unset and free to select a link-local IPv6 entry, was pinned to the IPv4 entry the fabric actually exposes. CUDA graphs were made breakable, then removed entirely. The RDMA memory-lock limit was lifted. Three NCCL settings that the working kit uses were adopted. The orchestration was moved off Ray onto vLLM’s own native multi-node path, which the playbook never mentions. Tensor parallelism was swapped for pipeline parallelism.

Every one of those failed at the same place. The traceback names a shared-memory broadcast read that times out, and POSIX shared memory does not cross a machine boundary. That is not a setting. Ray turned out to be a passenger rather than the cause, because vLLM’s native path routes through the same primitive and fails identically.

What changed the outcome was the build. A community-maintained container image, pinned by digest, running a development build of the serving engine one point release ahead of the stock one, served the same tensor-parallel width across the same two boxes without incident. Sixteen concurrent requests, zero errors, 215 tokens per second aggregate, and the engine still healthy afterwards.

On that image the pair was given DeepSeek V4 Flash 0731, a mixture-of-experts checkpoint of 156 GiB with FP8 weights, which cannot fit either box alone. This is the capacity case the second box exists for, and it is the case Lab 013 could not reach. The model served at 35 tok/s single-stream, which is five times what the same hardware manages with a 31B dense model on one box.

Then it ran the instrument. Twenty-two certified repair tasks, the same pool six earlier local models have run, driven through a headless agentic harness with a deterministic gate deciding every verdict. It cleared all twenty-two. No timeouts, no session errors, three API retries across four and a half hours.

The tasks it took are the interesting part. One repair in that pool had never been solved by any self-hosted model across six arms, and the clustered pair took it in 34 minutes over 57 turns. Another took 208 turns without losing its place in the instrument. That is what instrument literacy looks like when it holds under load, and it is the capability Lab 013 identified as the thing that separates models rather than parameter count.

The comparison that matters commercially is against renting. A hosted frontier model cleared the identical pool in 38 minutes for $3.14. The clustered pair took 4.5 hours and nothing. Seven times the wall clock, zero marginal cost, on capital already spent. Whether that trades well depends on how often the work runs and whether it can wait.

The comparison that matters architecturally is against the single box. The best self-hosted result on this pool was 18 of 22, from a model that fits in 128GB with no interconnect involved. Had the clustered pair matched that, the second box would have justified nothing. It cleared four more, including the one nothing local had ever solved.

So the hardware was never the limit. Two boxes, an in-spec cable, and a correctly configured fabric were sufficient from the first hour, and the fabric measurements said so: collective overhead at tensor-parallel width two is about 8 percent of the token budget. What stood between that hardware and a working endpoint was a documented procedure that does not work and a working configuration that is not documented.

The obvious next question is which other models this tier unlocks, and the answer is narrower than the memory arithmetic suggests. Five candidates were qualified. Qwen3-Coder 480B-A35B, the largest coder model in the same family as the one-box winner, needs 273 GB at four-bit, which exceeds both boxes before any cache. A 123B dense model fits, clusters, and decodes at 5.8 tok/s, six times slower than the sparse model that works, because decode rate tracks active parameters rather than total ones. Nemotron-3 Super 120B-A12B fits a single box and spent 74 minutes on a repair Qwen3-Coder-Next, 80B with about 3B active, finished in 7. And Qwen3-235B-A22B-2507, which fits comfortably and carries the context length the harness needs, fails engine initialisation at tensor-parallel width two with a kernel signature mismatch.

That last failure is the one worth generalising. Both dense models put across the link, Gemma 4 31B and Devstral 2 123B, cluster without incident. Of the two mixture-of-experts models tried, only DeepSeek V4 Flash clusters, and only on a container image built for its architecture; Qwen3-235B-A22B-2507 fails engine initialisation at width two. The mechanism is that tensor parallelism asks for a per-layer all-reduce, a collective every runtime handles, while mixture-of-experts additionally needs all-to-all expert dispatch through fused kernels pinned to a hardware generation and a library version. Dense models never touch that code path. So the sparsity that lets DeepSeek V4 Flash fit inside 256 GB is the same property that makes it hard to split across two machines, and the vendor and community record agrees: multiple reports show Qwen mixture-of-experts models serving at width one and failing at width two, with no working two-Spark demonstration published.

Which turns the purchasing statement into something a spec sheet cannot express. The second box does not buy access to models above 128 GB. It buys access to models above 128 GB that are also either dense, or accompanied by a runtime built for them. Today that is a set of one.

## Method and disclosure

Self-funded. Both DGX Sparks, the QSFP112 cable, and the power were bought by the practice. No vendor was briefed, no vendor reviewed this, and no vendor was given advance sight.

The repair pool, the reproduce-or-drop certification, and the deterministic gate are the same instrument used in Labs 011 through 013, unchanged, so the solve counts sit on the same axis as the earlier arms.

Serving configuration is recorded verbatim in every result file for this lab, including the image, the parallelism width, the sequence cap, and the fabric. Earlier arms in this corpus did not record it, which is a gap this lab does not repeat.

The harness, the gate, and the certification tooling are public. The assessment methodology, the vendor grades this evidence feeds, and the judgment that turns a score into a placement are not.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# Beyond CUDA: the lock was never the silicon

> Lab 017 · Editorial lab · Status: published · Author: Keith Townsend · August 11, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/beyond-cuda · Structured: https://labs.layer2c.com/labs/beyond-cuda.json

**Finding:** Lab 002 rerun off NVIDIA end to end: the workflow is portable and CUDA is a default, not a requirement; what actually gates the path is who will sell you the silicon, and whether your acceptance gate measures what you think it does.

**Question:** Lab 002 ruled that the weights are the asset you keep, and every step of that ruling ran on NVIDIA: the box trained, the box served, and the cloud comparison was NVIDIA-backed. So the verdict carried an untested assumption. Does the workflow that produces and serves owned weights actually require CUDA, or is CUDA just where everyone happens to be standing? This lab reran the Lab 002 fine-tune on a rented AMD MI300X, brought the weights home to Apple silicon, and held everything else constant: same base model, same training set, same frozen thirty questions, same recipe down to the learning rate.

**Load:** The Lab 002 fine-tune and serving workload, rerun with the silicon as the only variable: Low-Rank Adaptation (LoRA) training on rented AMD Instinct MI300X, serving on an owned Mac and on the rented card, quality judged under one rebuilt gate.

**Raw lab detail:** https://labs.layer2c.com/downloads/beyond-cuda-lab-detail.md
**Vendors:** Hot Aisle (cloud), AMD (hw), Apple (hw), NVIDIA (hw), Microsoft (ref)
**Layers:** Layer 0 (Compute), Layer 2B (Runtime), Layer 2C (Reasoning)
**Themes:** model-ownership, ai-factory-economics, serving-economics, validator-authority

## Verdict

**Scope:** The workflow is portable. The same configuration file, the same trainer, the same serving container ran on AMD silicon with zero code changes, trained the same LoRA six times faster than the owned box, and the tuned model held quality parity under every instrument that survived scrutiny. CUDA is a default, not a requirement. What actually gates the non-CUDA path is two things this lab measured: who will sell you the silicon at all, and whether your acceptance gate is measuring what you think it measures.

- **DO:** rent the training run. The 8B fine-tune cost 29 cents and 5.8 minutes on a rented MI300X; the 70B cost $8.89. The weights come home either way. If you own the training set and the recipe, training compute is an errand, not a capital decision.
- **DON'T:** infer capacity from a catalog. AWS rents NVIDIA accelerators fine, and this practice has rented L40S there; what it rents is no AMD datacenter-class part at all. Azure prices the MI300X in 17 regions and offered this account 2, behind a quota of zero. GCP would not provision anything with an accelerator attached. The neocloud published shelf inventory in an API response and delivered a running VM in seconds. For silicon off the CUDA path, access is a product surface, and the hyperscalers do not sell it.
- **DO:** treat the mainstream stack as already portable. PyTorch, the trainer, and vLLM installed clean on ROCm every time. The one real lag was day-zero support for a brand-new architecture, and NVIDIA felt a version of that same lag inside this lab too.
- **DON'T:** let one judge gate pick your model tier. Four instruments scored the same outputs and returned four verdicts. The strict judge pair said the 70B beats the 8B by 2.5x; mechanical quote verification and human reading said parity. If your gate is a model judge, you are buying the judge’s taste.

**Independence and disclosure:** Self-funded editorial with one comped input, disclosed: Hot Aisle’s CEO credited $100 to the account after it was opened, and the lab spent $29.75 of it. The credit funded the compute, not the answer. Hot Aisle did not commission, review, or see this lab before publication, had no say in what ran or shipped, and self-serve payment was available the whole time, so the credit never gated access. Correspondence with Hot Aisle is paraphrased where it appears. Disclosure: Google Cloud is a client of The CTO Advisor LLC, and it appears here twice: it failed the availability probe this lab reports, and it supplied one of the two judges (Gemini 2.5 Pro on Vertex AI, part of about $3 in self-funded API spend alongside AWS). Hot Aisle, AMD, Apple, NVIDIA, AWS, and Microsoft are not clients. The ruling is the author’s alone.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Account-zero to MI300X shell | ~15 active minutes | provision POST to first inference token: ~8 min, image-pull bound |
| 8B LoRA fine-tune | 5.8 min · $0.29 | identical recipe, same 111 steps; Spark baseline was 35 min |
| 70B LoRA fine-tune | 89 min · $8.89 | 2x MI300X, ZeRO-3 with CPU offload; the tier that was relationship-gated out of Lab 002 |
| Gemma 4 31B dense, one MI300X | 55.8 tok/s single-stream | 10 tok/s on 2x clustered Sparks; 1,974 tok/s aggregate at 64 streams ≈ $0.42/M output tokens |
| Tuned 70B, same weights, three venues | 25.7 · 42.3 · 3.2 tok/s | one MI300X merged · two MI300X · two Sparks; rented single card beats the owned cluster 7.2x |
| Quality, one rebuilt gate | AMD-8B 33% vs Spark-8B 27% | training-venue parity; 30/30 valid JSON on both AMD-trained models |
| Quote fidelity (mechanical) | 8B 99% · 70B 100% | the strict gate’s 2.5x claim for the 70B is withdrawn: judge taste, not fabrication |
| Whole-lab cost | $29.75 GPU + ~$3 APIs | staging exceeded compute roughly 2:1 across the campaign |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 0 (Compute) | Delegated | Training and speed-tier serving rented per-minute against published shelf inventory. The Sparks and the Mac retain the floor: the weights run at home on nobody’s quota. |
| Layer 2B (Runtime) | Retained | Same artifact served on three vendors’ silicon; venue chosen per workload, not per platform commitment. Venue fluidity is what retention looks like now. |
| Layer 2C (Reasoning) | Retained | The weights come home, and the layer splits: the acceptance gate is revealed as its own authority surface. Whoever defines the validator defines which model tier wins. |

## What this does not establish

- Mainstream-recipe portability only. The bleeding edge, newest architectures and exotic quantization, was not retested on ROCm beyond the one image-tag lag this lab hit.
- Thirty questions, one task, one domain. Quality parity is directional, not a benchmark.
- The original Lab 002 gate is unreproducible; absolute comparison to the published 70% is broken and replaced by same-instrument relative comparison, disclosed above.
- The 70B’s 83% under the strict gate is reported and withdrawn as a quality claim; it measures judge behavior, not verified grounding.
- One neocloud tested. Hot Aisle’s access model may not generalize across the tier.
- The two-Spark 3.2 tok/s is a floor of this stack version, not a ceiling; the graph-capture workaround may be fixed upstream.

## Open questions

- **Who validates the validator?** Four instruments disagreed about a 2.5x quality gap on the same thirty outputs. Before an acceptance gate picks your model tier, something has to pick the gate, and nothing in the current tooling does that job. This lab leaves that question standing.
- **Does the day-zero support asymmetry actually close?** The newest model family needed a nightly image on ROCm and a special build on CUDA. One image tag of distance, measured once. Whether that gap holds, widens, or closes at the next major model drop is measurable the day it happens.
- **Does the fine-tune survive quantization?** The designated 16GB fair-venue Mac serves this model only quantized. The unquantized arm ran on a larger machine, so the behavior of these weights under 8-bit serving is unmeasured here.

## Objection handled

**But nobody got fired for buying NVIDIA, and the software moat is real.**

It is, at the frontier. Lab 002 hit a four-wall dependency cascade trying to fine-tune a brand-new architecture on CUDA, and this lab hit the same species of wall on ROCm: the stable container did not know the newest model family, and the fix was a nightly image. The moat exists where the weights are newest.

However, for the workload that pays the bills, a Llama-family LoRA on the standard stack, the moat has been abstracted away. The trainer pip-installed clean. The serving container was stock. Nothing in the path knew or cared that CUDA was not underneath it. And the two-Spark cluster needed a workaround this lab’s AMD boxes did not: the graph-capture path hung under runtime LoRA on the NVIDIA stack while the ROCm container ran the identical configuration untouched. Day-N polish is workload-specific. Neither vendor owns it.

## Assessments at the time of the lab

| Vendor | Layer | Grade | As assessed |
| --- | --- | --- | --- |
| AMD Instinct, EPYC & Enterprise AI Suite | Layer 0 · Compute | The Second Source — Silicon Across All Three Sub-Layers | August 10, 2026 |
| AMD Instinct, EPYC & Enterprise AI Suite | Layer 2B · Runtime | Serving Catalogue on a Standard Endpoint, No Distributed Inference Plane | August 10, 2026 |
| AMD Instinct, EPYC & Enterprise AI Suite | Layer 2C · Reasoning | Structurally Out of Reach Without a Data Layer | August 10, 2026 |
| NVIDIA AI Platform | Layer 0 · Compute | NVIDIA Strength — Silicon Authority | July 31, 2026 |
| NVIDIA AI Platform | Layer 2B · Runtime | NVIDIA Authority — Inference + Agent Runtime | July 31, 2026 |

## Summary

Lab 002 proved the weights are the asset, entirely on NVIDIA silicon. This lab moved every stage of that workflow off CUDA and measured what actually changed. Training: the identical LoRA recipe ran six times faster on a rented MI300X than on the owned Spark, for 29 cents. Serving: the tuned weights came home to a Mac and outran the original box. Quality: parity with the Spark-trained original under every instrument that survived scrutiny. The silicon was never the lock.

The two real findings sit either side of the compute. Below it: access. Three hyperscalers could not or would not rent the AMD part this lab needed, while a neocloud published its shelf inventory in an API and delivered in seconds. Above it: validation. The rebuilt judge gate claimed the 70B beat the 8B by 2.5x, and the claim did not survive a human read or a deterministic quote check. Which model tier pays off turned out to be a property of the validator, not the model.

## Detail

The availability ladder answered Lab 002’s own open question first. Does a neocloud actually deliver access? Four venues, one afternoon. GCP: no accelerator provisioning at any tier, including a CPU. AWS: rents NVIDIA accelerators fine, and this practice has rented L40S there, but its only AMD GPU is a 2020 graphics part, and an L40S is not an answer to a 192GB high-bandwidth-memory question. Azure: the only hyperscaler that sells the MI300X, in 2 of its 17 priced regions for this account, family quota zero, spot quota 3 virtual CPUs against a 96-CPU box. The scope of that ladder is AMD Instinct-class silicon, not accelerators generally. The wall is specific to leaving CUDA. Hot Aisle: a GET request returned shelf quantity, a POST returned a running VM, and the first inference token came 8 minutes after the provision call.

Training moved without friction that mattered. Identical recipe, identical data. The MI300X ran Lab 002’s 35-minute fine-tune in 5.8 minutes for 29 cents, converging to the same loss over the same 111 steps. The 70B, the tier that was relationship-gated out of existence in June, trained in 89 minutes for $8.89 across two cards. The frictions worth logging were a stale package index, an offline-cache bug in the image’s Python stack, and the author’s own process mistakes. Nothing ROCm-specific broke, at any point, in the whole campaign.

Serving came home. The merged 8B served unquantized on the Mac at 26 tokens per second: the laptop outran the Spark serving the same weights in Lab 002. The dense 31B that crawls at 10 tokens per second across two clustered Sparks ran 5.6x faster on one rented card, and at 64 concurrent streams the card delivered commodity-token economics on owned weights, about 42 cents per million output tokens. Lab 002 found that custom weights cost $27.50 per million on the managed floor. That floor was never physics. It was packaging.

The dual-card question answered itself. A 70B fits whole on one 192GB card; the second card bought 1.6x single-stream speed for 2x cost, a latency purchase, not an efficiency one. And the owned two-Spark cluster, serving the same tuned 70B, delivered 3.2 tokens per second. The rented single card beat the owned cluster 7.2x. Owned compute at this tier is a sovereignty floor, not a serving tier. The weights run at home on nobody’s quota; when speed matters, you rent the hour.

The gate ate the headline. The original judge script was lost and its judge model retired, so the gate was rebuilt from the frozen contract and calibrated against Lab 002’s archived outputs. The rebuilt gate read grounded more strictly: the archived 70% baseline scored 27% under it. Fine. Re-baseline, compare relatively. The AMD-trained 8B matched the Spark-trained original within noise, so the training venue cost nothing. Then the strict gate said the 70B wins by 2.5x, and that number survived exactly until a human read the outputs. The wording difference is not perceptible. Mechanical quote verification agreed with the human: both models cite real quotes at parity, 99 and 100 percent, one fabricated citation out of 178. The judge pair was not detecting fabrication. It was expressing taste.

Four instruments, four verdicts, one set of outputs. The lenient original gate said the 8B was saturated. The strict rebuilt gate said the 70B wins by 2.5x. Deterministic verification said parity. Human reading said parity. So the question an Enterprise Architect actually asks, does the bigger model pay off, has no answer until you answer a prior question: measured by what? The validator is the control point, and the validator itself requires validation. That is the campaign’s standing thesis arriving from the measurement side, and it is the finding this lab keeps.

## Method and disclosure

Self-funded editorial, no sponsor; the comped Hot Aisle credit is disclosed in the independence line. Every measurement ran under one frozen contract: same training set, same eval set, same retrieval index, same prompts as Lab 002. The only variable was silicon.

Training was Low-Rank Adaptation (LoRA) via the same trainer and recipe as Lab 002, on rented MI300X capacity. Serving was measured on the rented cards, an owned Mac (MLX, fp16), and the owned two-Spark cluster, with the same bench harness throughout. The judge gate was rebuilt from the frozen contract after the original was lost, calibrated against archived outputs, and then itself became a finding when mechanical verification and human reading overruled it.

The raw detail ships at the download link: the availability-ladder data, every bench table, the training configs, the complete friction log including the author’s own mistakes, and the cost ledger with the staging split. The training set, eval set, adapters, and judges stay with the practice.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# You can’t price the task without pricing the fine-tune.

> Lab 018 · Editorial lab · Status: published · Author: Keith Townsend · August 23, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/trained-by-whom · Structured: https://labs.layer2c.com/labs/trained-by-whom.json

**Finding:** The question was cost per task. The finding is that you cannot answer it without first pricing the fine-tune, and the fine-tune is capital that amortizes, not an implementation detail. A week of failing to make an owner’s fine-tune beat a foundation model at code repair produced the thing worth keeping: a decision procedure for when fine-tuning pays, sitting under a compliance override that decides self-hosting first, no matter the capability. Self-hosting has several doors, and fine-tuning is the narrowest: compliance forces it, off-the-shelf open weights can win it with no tune at all, and only then does capability fine-tuning apply, gated by five conditions code repair meets two of. Most enterprise domains that look like fine-tuning problems are retrieval problems wearing the costume.

**Question:** We set out to measure cost per gate-verified task, own hardware against hosted API, and answer whether an expensive owned box earns its price. The measurement kept stalling on a cost we had minimized: the cost of the model itself. You do not get the model for free. You build datasets, run a tune, evaluate it, throw it away, and try again, and the infrastructure the discarded models burned does not vanish because the models did. Pricing a task means pricing that search. This lab ran the search to the end on one domain and came back not with a number but with the procedure that decides whether the number is worth computing at all.

**Load:** One campaign, six days, two owned DGX Sparks, one rented MI300X, and four hosted APIs. The eval instrument: the 22-repair pool and deterministic gate of Labs 011-013, unchanged. New apparatus: an 82-task disjoint training pool mined from a dozen fresh repos, a 59-trajectory teacher corpus, a 1,719-example synthetic precision factory, four trained adapters (three imitation/skill tunes plus one self-consolidation LoRA that ran), a self-generated harvest of 246 gate-passed loop solutions, roughly 150 base and tuned agentic sessions across a dozen arms, a weights-by-harness read, Gemini and DeepSeek baselines, a trajectory taxonomy, two-Spark DDP training, and a mechanical venue preflight that now refuses bad launches. Metered spend roughly $73; owned hardware roughly 90 GPU-hours; two host reboots; every invalidated arm archived, not deleted.

**Raw lab detail:** https://labs.layer2c.com/downloads/trained-by-whom-lab-detail.md
**Vendors:** Google (model), NVIDIA (hw), AMD (hw), Hot Aisle (cloud), Anthropic (harness), OpenAI (model), Mistral AI (model), Alibaba (model), DeepSeek (model)
**Layers:** Layer 0 (Compute), Layer 2B (Runtime), Layer 2C (Reasoning)
**Themes:** fine-tuning-limits, model-ownership, ai-factory-economics, small-model-viability, agentic-repair, validator-authority

## Verdict

**Scope:** Measured on localized repair with an executable test: one 22-task pool, one open-model family as the tuning subject, one week of checkpoints and prices. The decision procedure is the deliverable; the code-repair verdict is the worked example that produced it. The open edges are a domain that satisfies all five conditions, repository-scale work without localization, the file-geometry wall, and a clean re-measurement of harness edit-landing once the tool-call parser was corrected.

- **DO:** settle compliance first, then run the five-condition test. Compliance is an override, not a condition: if the data cannot leave the VPC, or the workload is regulated or air-gapped, you self-host regardless of capability and the conditions below do not apply. Where the hosted frontier is a legal option, the test gates capability fine-tuning: the model has to fail your gate not your budget, the missing capability has to be behavior a corpus cannot inject, a deterministic gate has to exist, the value and volume have to amortize a fixed cost, and you have to price the whole how including the enforcement the tune does not remove. Cost tuning to distill a cheaper model that matches an expensive one is a separate branch that lives in the volume math, not this test.
- **DON'T:** treat the fine-tune as an implementation detail inside a cost-per-task number. It is capital. It amortizes across every task it solves, which makes fine-tuning volume economics: catastrophic over one task, invisible over a million of one shape. Any cost model that expenses the training run and forgets the search that preceded it, and the scrap the search burned, is pricing the wrong thing. This search cost roughly $73 and ninety GPU-hours to buy a two-task gain that a hosted model beats outright.
- **DON'T:** assume owning the weights buys you a corpus’s worth of knowledge. Retrieval injects what a model knows. Fine-tuning changes how it acts. If a corpus plus a frontier model clears your gate, the task was a RAG problem wearing a fine-tune costume, and most enterprise domains that look proprietary are exactly that. Fine-tuning’s defensible territory is behavior a corpus cannot supply, which is a narrow intersection, not a default.
- **DO:** expect the deterministic gate to survive the tune, and budget it. The fine-tune moves behavior most of the way and a hard invariant still leaks the base model’s prior. This practice’s voice work is the proof: even fine-tuned and prompt-guarded, the model reverts to em-dashes, and only deterministic tooling in the loop enforces the pattern. That gate is a recurring cost that never amortizes. Owning the model does not retire the validator. It shifts work onto it.
- **DO:** match the apparatus to the model and get the serving contract right before you read a single score. In the loop the consolidation tune reached 14 of 22 against a fair untimed base of 12, a nudge inside noise. On the harness the story was a serving bug: the wrong tool-call parser silently dropped the model’s native emission, and corrected, the untuned base scored 15 where it had shown six to eleven. Loop and harness are different apparatus, so do not read the loop’s 14 against the harness’s 15. Serving correctness is part of the model contract, and a mechanical preflight, envelope, arm-scale prefill, end-to-end tool smoke, provenance stamp, is now the gate every arm passes first.

**Independence and disclosure:** Self-funded editorial. Disclosure: Google Cloud is a client of The CTO Advisor LLC, and Google models sit at the center of this lab as both tuning subject (Gemma 4) and benched baselines (Gemini 2.5 and 3.6 Flash, Vertex). Anthropic’s Claude Code is the primary harness and is itself implicated in the findings. Hot Aisle, AMD, NVIDIA, OpenAI, Mistral, Alibaba, and DeepSeek are not clients. No vendor commissioned, funded, or previewed any of this.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| The frontier the tune must beat | mini 22/22 · $3.14 · 40 min | gpt-5.4-mini, 10-turn median; the whole pool solved for the price of a coffee, which is the first-condition failure for code repair |
| Owner fine-tunes, gate-verified gain | 3 tunes · 0 · then +2 (noise) | imitation and isolated-skill: zero; self-consolidation LoRA: 14/22 vs a fair untimed loop base of 12/22, a two-task delta inside sampling noise on 22 tasks, and it still loses to the frontier |
| Harness wall was a serving bug | base 6-11 → 15/22 | the wrong tool-call parser silently dropped Gemma’s native emission; the gemma4 parser recovered it; the serving contract is part of the model |
| The dispatcher, per verified repair | loop ~$0 · mini $0.14 · DSV4 $0 | loop clears 17/22 deterministically; mini 22/22; DeepSeek V4 Flash 22/22 twice, 7-min median, on two clustered Sparks |
| Fine-tuning is volume economics | fixed ÷ tasks solved | catastrophic over one task, invisible over a million of one shape; the term that was invisible while we tested a domain the frontier owns |
| The how is layered | tune + non-amortizing gate | fine-tune moves behavior most of the way; a hard invariant leaks the base prior (em-dashes) until a deterministic gate enforces it, and that gate is recurring cost |
| Build vs buy does not zero the term | validation + transfer risk | DeepSeek community tune: free download, real GPU-hours to stand up, and it worked only inside its own training scaffold |
| Cost of the search, the honest unit | time to answer, not dollars per hour | metered compute was ~$73 across ~90 GPU-hours and six days; the money measures electricity, the six days measure when the answer arrived. The accelerator invoice is the only line item with a number; the waiting has none. Slow hardware does not burn attended hours, it occupies the critical path: the run that misses tonight is the experiments that cannot start tomorrow. Cost per solved task was never the constraint; time to solved task was |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 0 (Compute) | Retained | Two altitudes, one placement. The owned boxes carried training, serving, loops, and diagnosis at zero marginal dollars, including a two-Spark DDP run once the fabric proved bandwidth-bound rather than latency-bound. The cluster’s highest use surfaced late: unified memory holds an open frontier-scale model that sweeps this pool. Capacity, not capability, is what ownership buys, and a host reboot twice was the price of pushing memory past its wall. |
| Layer 2B (Runtime) | Retained | The deterministic layer is the part that never regressed and the part the whole finding rests on: gate, loop, crash-safe results, a venue preflight that refuses bad launches, and, this campaign, a tool-call parser correction that recovered a model the wrong parser had buried. Every lesson that lived as code survived; every lesson that lived as agent memory was repaid this week. The runbook is the control plane; agent memory is a cache. |
| Layer 2C (Reasoning) | Retained via selection, and priced | The repair-reasoning seat stays shoppable, and this lab prices it. An owner controls the apparatus and the serving conditions for free and the vendor’s post-training only through model selection. The reachable tune bought two tasks and still lost to the frontier on cost. Retain the seat by choosing tool-trained weights or matching the apparatus to untooled ones, and fine-tune only where the five conditions hold. The determines-done authority never moved. |

## Seam map — readiness (projected, pending bench)

| Function | Readiness |
| --- | --- |

## What this does not establish

- It did not prove owner fine-tuning never pays. It proved that on localized code repair, with one open-model family as subject, three reachable families bought nothing and the best-designed fourth nudged the count by two tasks, a delta inside sampling noise on 22 tasks, measured on one loop configuration without multi-seed runs or confidence intervals. The decision procedure is the transferable claim; the code verdict is one instance of it. The bounded-generation regime where Lab 002 measured a 3x owner-LoRA gain stands untouched, and is itself an instance where the five conditions plausibly hold.
- The parser correction reopens the phase-one harness findings rather than cleanly replacing them. The untuned base scores 15 of 22 with the correct tool parser, but a full re-measurement of edit-landing across all models on the corrected serving was not finished before the card came down. The execution-deficit story is reopened, not overturned.
- The consolidation tune’s two-task gain is measured in the loop against a fair untimed control. Its harness arm was torn down before completion as immaterial to the conclusion, so the tune’s effect on the tool-driven surface is only partially observed.
- The five-condition procedure is validated against one domain that fails it, not against a domain that passes. Its predictive value on a passing domain, the case where fine-tuning is supposed to win, is asserted from mechanism and awaits the next lab. The procedure also gates capability fine-tuning only; the distillation branch (a cheaper model tuned to match an expensive one at volume) is named but not measured here, and the compliance override that precedes all of it is a constraint this lab asserts, not one it tested.
- The DeepSeek V4 Flash sweep is this pool only, on a two-box cluster this practice already owned, and whether it survives an un-localized pool or a different task class is unmeasured.
- Roughly a quarter of the campaign’s spend was avoidable operator error, including two host reboots from pushing unified memory past its wall. The gates built from those errors are asserted to prevent recurrence from mechanism, not yet measured across a subsequent campaign.

## Open questions

- **What domain satisfies all five conditions?** Code repair fails on the first: a hosted model clears the gate for pocket change. The domain where fine-tuning pays needs the foundation model to fail the gate, the missing capability to be behavior a corpus cannot inject, a deterministic gate to exist, and the volume to amortize the build. The Articul8 sponsored work named the Layer 2C reasoning plane but lacks a workload to test at this task-level economics. Finding one, and building its gate before any GPU spend, is the next lab. The gate is the hard part; no gate, no economics.
- **How far does the parser correction reach?** The harness numbers phase one read as a model deficit were measured with a tool-call parser that dropped Gemma’s native emission. Corrected, the untuned base jumps from six to eleven up to 15 of 22. The tool-trained peers were parsed correctly, so the 37%-versus-96% edit-land comparison was partly apples to oranges. A clean re-measurement of edit-landing across every model with the correct parser was not completed before the rented card came down. Until it is, treat the phase-one execution-deficit finding as reopened, not overturned.
- **Does the dispatcher survive losing localization?** Every task here arrived with its files named, which switches off the one advantage agentic scaffolding is priced for. Repository-scale fault localization is the harness’s home game and the loop cannot attempt it. Whether triage-then-escalate holds, and where the local rung’s share falls, when the pool withholds file names is the next pool to mine, and the domain hunt above may find its gate there.
- **Whose harness serves whose weights?** Google’s Antigravity was probed as the bring-your-own-model path for Gemma and closed as unreachable: the desktop app silently declines a local endpoint, and the CLI, inspected at the binary level, lists hosted models only, carries a Google-and-Vertex provider enum with no OpenAI-compatible entry, and gates custom models behind a server-side experiment flag. Local weights under Google’s harness are not routable by any visible surface. The managed floor has moved up into the harness layer, and a routing check that counts requests at the serving log is the mandatory gate for any bring-your-own-model claim.

## Objection handled

**You just tuned badly, and your best tune only gained two tasks. Better data or RL closes it.**

The best-designed tune did run, and it worked: a self-consolidation LoRA trained on the model’s own gate-passed loop solutions scored 14 of 22 in the loop against a fair untimed loop base of 12, a two-task delta that sits inside sampling noise on a pool this size, zero regressions. That is the best the reachable families produced as we reached them, and it is the point. A nudge over base still loses outright to a mini-class hosted model that solves all 22 for $3.14 in forty minutes. The tune succeeding does not move the domain; it confirms the domain belongs to the foundation model even when the tune helps.

The families an owner can reach are imitation, isolated-skill, and consolidation, and all three are measured here. What remains is frontier-scale RL post-training, which an owner cannot buy as a process, only as a product. That product sits in the open-weights catalog: Devstral and Qwen3-Coder land their edits where Gemma’s family does not, and DeepSeek V4 Flash swept this pool twice on owned clustered hardware for nothing but memory and hours. Acquiring the capability by model choice costs nothing. Acquiring it by training cost this lab a week to establish that, for this domain, it should not have been attempted, and the enterprise version runs the same shape at a thousand times the bill.

## Summary

The lab set out to price a task and discovered it could not, because the model is not free and the fine-tune that produces it is capital, not an implementation detail. The deliverable is the decision procedure that comes before the price: five conditions that have to hold together before fine-tuning, and therefore self-hosting, is the right lever. The foundation model has to fail your gate rather than your budget. The missing capability has to be behavior a retrieval corpus cannot inject. A deterministic gate has to exist. The value per task has to be high and the volume large enough to amortize a fixed cost. And you have to price the whole how, including the deterministic enforcement the tune does not remove.

Code repair was the worked example, and it fails the test on the first condition. A mini-class hosted model solves the whole 22-task pool for $3.14. Three owner-side tunes, imitation and isolated-skill, produced no gate-verified improvement, and the imitation tune was actively destructive. The fourth and best-designed tune, a self-consolidation LoRA trained on the model’s own gate-passed loop solutions, did run: 14 of 22 in the loop against a fair untimed loop base of 12, a two-task delta that sits inside sampling noise on a pool this size. A tune that at best nudges the count and still loses outright to the frontier is the cleanest proof that the domain belongs to the foundation model even when ownership works.

Two corrections sharpened the picture. The harness “mechanical wall” that phase one read as a model deficit was substantially a serving bug: the tool-call parser was dropping Gemma’s native emission, and with the correct parser the untuned base scored 15 of 22 instead of six to eleven. The serving contract is part of the model, and getting it wrong reads as incapacity. And the fine-tune never retired the validator: the practice’s own voice work shows a fine-tuned, prompt-guarded model still reverting to em-dashes until deterministic tooling enforces the pattern. The how is layered, and the bottom layer is a gate that never amortizes.

The economics generalize as amortization and branch order. A fine-tune is a fixed cost divided across every task it solves, so fine-tuning is volume economics: catastrophic over one task, invisible over a million of one shape. Build or buy does not change the term, only its shape, and the buy path carries a transfer risk the build path does not, measured here when a community tune worked only inside its own scaffold. Serving cost, the branch the original spec opened on, is the last branch, not the first: above it sit yield, apparatus fit, whether the vendor trained the capability, and whether you needed to train at all or could have retrieved. And self-hosting is not the same decision as fine-tuning: compliance forces it above any capability question, and off-the-shelf open weights can win it with no tune at all, which is why the frontier-owns-code verdict is a verdict about fine-tuning, not about owning hardware.

The campaign priced its own search, which is the exhibit the fine-tuning pitch never includes. Roughly $73 metered, ninety owned GPU-hours, and six days to buy a two-task nudge inside noise, about a quarter of it avoidable operator error now encoded as mechanical gates. The money measures electricity; the six days measure when the answer arrived, and on a dependent question tree the delay is the real bill, because the run that misses tonight is the experiments that cannot start tomorrow. The instrument built to measure cost per solved task showed that cost per solved task was never the constraint. Time to solved task was. The number a self-hosting business case needs is not the training run and not a blended hourly rate. It is time to answer and throughput, the terms the owned box hides and the CFO does not.

## Detail

The lab began as a cost-per-task measurement: run one agentic repair pool against owned hardware and against hosted APIs, and find what a solved task actually costs each way. The measurement kept stalling on a term the original model had minimized. You do not get the model for free. Testing whether an owner’s fine-tune beats a foundation model means building the fine-tune, and building it means datasets, a training run, an evaluation, a disappointing result, a data change, another run, and a pile of discarded checkpoints whose infrastructure cost did not vanish when the checkpoints did. Pricing the task means pricing that search. So the deliverable stopped being a number and became the procedure that decides whether the number is worth computing: five conditions that have to hold together before fine-tuning, and therefore self-hosting, is the right lever.

Code repair was the worked example, and it fails on the first condition. A mini-class hosted model solved the entire 22-task pool for $3.14 in forty minutes, which means the foundation model fails nobody’s gate here, only a budget the frontier already undercuts. Against that floor the owner tunes ran. Three of them, imitation from a teacher and an isolated-skill precision factory, produced no gate-verified improvement, and the imitation tune was worse than nothing: it transferred the teacher’s brevity without the precision that makes brevity work and damaged behaviors the base model already had. The fourth was the one the arithmetic said to build: a self-consolidation LoRA trained on the model’s own gate-passed loop solutions, harvested from a disjoint pool. It ran. Fourteen of 22 in the loop against a fair untimed loop base of 12. On twenty-two tasks a two-task delta sits inside sampling noise, so call it what it is: a small nudge, not a breakthrough, measured against the loop’s own base and not against the harness. A tune that at best nudges the count and still loses outright to a $3.14 hosted run is the cleanest proof that the domain belongs to the foundation model even when ownership is doing its job.

Two corrections kept the read honest. The harness failure that phase one had called a model deficit, a 37% edit-land rate against tool-trained peers near 96%, was substantially a serving bug. The tool-call parser was dropping Gemma’s native emission format, scoring correctly-shaped tool calls as plain text. With the correct parser the untuned base scored 15 of 22 where the wrong one had it at six to eleven. That does not erase the peer gap, and a clean cross-model re-measurement was not finished, but it relocates a chunk of the wall from the model to the serving contract, which is itself part of the model. Two apparatus, two numbers: the loop’s 14 and the harness’s 15 are different instruments and do not compare to each other. The second correction is that the tune never retired the validator, and the parser is the in-domain proof of it: a deterministic layer had to decode what the model would not reliably emit before the harness could score it. The same pattern shows in this practice’s voice work, where a fine-tuned, prompt-guarded model still reaches for the em-dashes a gate has to strip. The how is layered. The fine-tune moves behavior most of the way, and a deterministic layer holds the invariant the base prior keeps violating.

From there the economics generalize, and they generalize as amortization. A fine-tune is a fixed cost divided across every task it solves. Over one task it is catastrophic; over a million tasks of one shape it rounds to zero. Fine-tuning is volume economics, and code repair fails on volume as surely as on the frontier price: even a free-to-run tune has to amortize its build against a domain where the frontier already wins per task. Buying the tune instead of building it does not zero the term, it reshapes it. A community adapter is a free download and a real bill in GPU-hours to stand up, and it carries a transfer risk the build path does not, measured here when a well-regarded community tune worked only inside its own training scaffold and nowhere else. Either way you pay to learn whether the weights clear your gate.

The capability question sits underneath all of it, and it is the one most enterprises get backward. Retrieval injects what a model knows. Fine-tuning changes how it acts. If a corpus plus a frontier model clears the gate, the task was a retrieval problem wearing a fine-tune costume, and the proprietary-knowledge domains that look like fine-tuning candidates are usually exactly that. Fine-tuning earns its cost only where the missing thing is behavior a corpus cannot supply, the gate rejects the frontier, the volume amortizes the build, and the value justifies the search. That intersection is small, which is the finding, not a disappointment: real fine-tuning use cases are rarer than the market implies. And the arithmetic for self-hosting falls out of the same order, once you stop fusing it with fine-tuning. Self-hosting is forced by compliance regardless of capability, no amount of frontier quality overrides a workload that cannot leave the VPC. It can also be won outright by off-the-shelf open weights that need no tune, the way an open model swept this pool self-hosted and untrained. Fine-tuning is only the narrowest door into it. Below all of that, serving cost is the last branch: above it sit yield, apparatus fit, whether the vendor trained the capability you need, and whether you needed to train at all. Own the model when compliance demands it, when open weights already win, or when the five conditions hold and the volume is yours. Rent the frontier otherwise, which is more often than the pitch admits. And price the search in the unit that dominates it, which is not GPU-hours and not attended labor but time to answer: slow hardware occupies the critical path, and the run that misses tonight is the experiments that cannot start tomorrow. That is the term the owned box hides and the CFO does not.

## How it abstracts

The transferable output is a decision procedure, and it starts above the economics with an override. Compliance is not a condition to weigh, it is a veto that decides first: if the data cannot leave the VPC, or the workload is air-gapped, regulated, or contractually bound to stay in-house, you self-host, and no amount of frontier capability changes that. Capability never trumps compliance. Everything below assumes the hosted frontier is a legal option in the first place.

With that settled, separate two decisions the pitch fuses. Self-hosting has several doors, and fine-tuning is the narrowest. Compliance forces it. Off-the-shelf open weights can already beat the frontier on price for your volume with no training at all, the way an open model swept this bench’s pool self-hosted and untuned. And fine-tuning itself splits into two motives that do not share a rule: capability tuning makes a model do what it could not, and cost tuning, distillation, makes a cheaper model do what an expensive one already does and is justified by volume and price delta alone. The five conditions below gate capability tuning. Distillation is a separate calculation that lives entirely in the volume term.

For capability tuning, five conditions have to hold together. One, the foundation model fails your gate, not just your budget; if it produces acceptable output at any price, capability is not what is missing, and you are looking at a distillation or a cost problem instead. Two, the missing capability is behavior, not knowledge; retrieval injects what a model knows, fine-tuning changes how it acts, and if a corpus plus a frontier model clears the gate it was a retrieval problem in a fine-tune costume. Three, a deterministic gate exists; no gate, no cost-per-verified-task, no economics, and this is the condition most domains lack. Four, value per task is high and volume is large, because a fine-tune is a fixed cost amortized across every task it solves. Five, you have priced the whole how, including the deterministic enforcement the tune does not remove.

That fifth condition is the one the market misses. A fine-tune moves behavior most of the way and a hard invariant still leaks the base model’s prior, so a deterministic layer enforces the last mile. This bench saw it twice: a tool-call parser that had to decode the model’s native emission before the harness could score it at all, and, in this practice’s voice work, a gate that strips the em-dashes a fine-tuned model still reaches for. The enforcement is a recurring cost that never amortizes while the fine-tune is capital that does, so more fine-tune converts flow cost into amortizable capital. This is Deterministic Code In The Loop: the model reasons, the deterministic validator enforces, the control point is the gate. Owning the model does not retire the validator; it shifts work onto it.

Then price the search honestly, because the search is the exhibit the pitch omits, and the unit is not dollars per hour. Compute is not the cost, and attended labor is not either: slow hardware does not mostly burn hours at a keyboard, it occupies the critical path. Runs go unattended, but the run that does not finish tonight is the answer that is not there tomorrow, and on a dependent question tree that is the two or three experiments that cannot start. The accelerator invoice is the one line item with a number; the waiting has none, and it surfaces as roadmap slip no one attributes to the cluster. Price four things instead of one: cost per solved task, what the answer cost; time to answer, how long before it exists; research throughput, how many questions close in a month; and capacity liquidity, whether you can get the compute at all. Faster hardware earns its rental not by cutting compute cost but by removing delay, which buys more answers per month. Compute is the cheap, last branch, below yield, apparatus fit, what the vendor trained, and whether you needed to train at all. Own the model when compliance demands it, when off-the-shelf weights already win, or when the five conditions hold and the volume is yours. Rent the frontier otherwise, which is more often than the pitch admits.

## Method and disclosure

Self-funded editorial. The instrument is loopcontrolbench, open under MIT: the reproduce-or-drop gate, the sandboxed runner, both harness drivers, and the additions this campaign contributed, a mechanical venue preflight (envelope, arm-scale prefill, end-to-end tool smoke, provenance stamp), the trajectory taxonomy analyzer, the land-rate gate, the synthetic precision factory, the loop-solution harvester, and a two-Spark DDP training path. The teacher corpus, the trained adapters, and any vendor-assessment consequences stay proprietary. Cross-lab operating discipline accumulated into specs/BENCH-METHOD.md, including the rules this campaign paid for: the tune must earn its place against a clean baseline, serve at the reference envelope with the correct tool-call parser, verify routing by counting requests, cap the loop at two turns, and kill by PID never by pattern.

Every invalidated arm is archived with its defect named, not deleted: a proxy that routed to the wrong API surface, a context cap below the reference envelope, a ceiling miscalibrated for contention, and a tool-call parser that dropped the model’s native emission. Serving configuration travels with every result via the venue note and a per-task backend probe. The comparisons are kept distinct by design: base versus tuned is the measurement, tuned versus teacher is distillation fidelity, and local versus managed serving is the economics, which this campaign resolved one branch above where the spec opened it, and then generalized into the decision procedure that is the lab’s finding.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*


# Local-first is not control-first.

> Lab 019 · Editorial lab · Status: published · Author: Keith Townsend · August 26, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/local-not-control · Structured: https://labs.layer2c.com/labs/local-not-control.json

**Finding:** Checked claim by claim against the shipped surface, the announcement holds on the model and breaks on the controls. It does run a post-trained Qwen 3.8 27B on your box, under a patched inference engine no coverage mentioned. It does not ask before every step, and the reason matters more than the fact. There are three approval surfaces here and they are architecturally different: network egress, which is permission to reach an outside destination; cloud disclosure, which is permission to send context to a remote model or connector; and sandbox escape, which is permission to execute outside the constrained runtime as your account. Consent is granted per class, not per step. Network egress was approved once and never raised again while filesystem access prompted repeatedly in the same session, and the cloud-disclosure surface, the one the classifier was supposed to front, never appeared at all. The classifier that was going to show you exactly what leaves the device never appeared; what appeared was a sandbox-exit request showing a command line. The frontier-advisor path behind the headline benchmark number was never available, because the product was never provisioned for it. Follow the widest of those gaps and it lands in the same place: the operator does hold the decision, and holds it in the one form that cannot be delegated to a rule. Per occurrence, manual, scoped to the account rather than the task, ruled on against a justification the model writes for itself, with no surface at that point on which to encode a standing answer. Retained is the framework call and it is the right one: the human can refuse, and refusal is enforced. It is not the same as the organization being able to encode that refusal, distribute it, enforce it and audit it, and a reader who takes retained to mean enterprise-controlled has drawn the opposite conclusion from the evidence. Lab 5 held that authority you reclaim is authority you run. Local execution moved the data plane. It did not supply a control plane.

**Question:** A launch told the industry what this product does: runs entirely on your own hardware, asks permission before any individual step, shows you exactly what a classifier would let leave the device, and escalates to a frontier advisor for a large benchmark gain. That set of claims is the measuring yard. We installed it on the hardware the launch names, ran it, and checked each claim against the shipped surface. Most of them are partly true in a way that matters, and the widest gap is not in any of the performance numbers. It is in who ends up holding the decision.

**Load:** One install and teardown on a headless DGX Spark, plus two instrumented sessions. Perplexity Portable Computer 26.8.4 running its own patched vLLM nightly and a post-trained Qwen 3.8 27B checkpoint, with a separate 1.2GB PII-masking model and a Rust agent runtime speaking JSON-RPC over a Unix socket. Socket-level egress measurement against a matched idle control. A denied-then-granted sandbox-exit prompt captured verbatim. A canary document held on-box and probed for retrievability from the vendor cloud. Then a second axis: the same five repair tasks from the Lab 11 pool run under the vendor stack and under Claude Code driving the same open weights on the same class of box.

**Vendors:** Perplexity (bench), Alibaba (model), NVIDIA (hw), Anthropic (harness), Google (model)
**Layers:** Layer 0 (Compute), Layer 2A (Orchestration), Layer 2C (Reasoning), Layer 3 (+1) (Applications)
**Themes:** authority-placement, vendor-claim-scrutiny, model-ownership, serving-economics

## Verdict

**Scope:** One product at one version on one class of hardware, over two days. The authority finding is observational and reproducible from the captured prompt. The packaging comparison is n=1 per cell with a confound on our side of the scale, declared below. The advisor path was never enabled here, so every claim about escalation is documentation rather than observation.

- **DO:** hold the vendor to their own architecture, which separates three device-boundary crossings. Network egress is permission to reach an outside destination. Cloud disclosure is permission to send context to a remote model or connector. Sandbox escape is permission to execute outside the runtime as your account. Keep them apart when you evaluate, because they fail differently: the worst governance exposure is the third, the worst residency exposure is the second, and the workflow question is whether the first is scoped and revocable. Then check what the shipped experience actually presents, because here it collapsed. One prompt appeared across the sessions we ran, and it was the broadest of the three. Consent is granted per resource class rather than per step, network egress was approved once and never raised again, and the classifier-led disclosure surface never appeared at all. Read the sandbox-exit prompt as a capability grant, not an action approval. The dialog shows one command line. What it grants, in the vendor’s own words, is a process running directly as your account that can reach your files, network, local services and processes, whose descendants may continue after the agent run ends. Those are different things and only one of them is on the screen. Before any pilot, decide who in your organization is qualified to rule on that, because the person clicking it is being asked to evaluate the model’s own written argument for why it needs out, in the model’s framing, at the moment the model wants out.
- **DON'T:** read local execution as a control plane. The data plane genuinely moves onto your hardware and that is worth something. The control plane does not come with it. The authority at the gate is retained, which sounds like the good outcome and is the expensive one: Lab 3 put the control point on a deterministic rule and explicitly not on the analyst’s self-assessment, and here the model’s self-assessment is the evidence the human rules on, with no surface to encode a ruling so it holds for the next prompt or for a colleague. Budget the operator time, because by Lab 5’s law you are now operating this.
- **DO:** expect the capability to come from the open weights and borrow the vendor’s plumbing for everything around them. The same 27B open model, driven by a generic agent harness on the same class of box, clears the same gates, so the capability is not what you are buying. What you are buying is the assembly, and this corpus has been consistent about what that is worth. Lab 1 said borrow the vendor’s plumbing and keep your own judgment. Lab 12 said buy the harness, not the tier. Lab 5 said authority you reclaim is authority you run. This lab is the same shape seen from the vendor’s side: their plumbing is good, and the judgment they cannot sell you is where the decision belongs.
- **DON'T:** take a residency verdict from this lab. It did not test one. The model does run on your hardware and the masking model is local, and against that, session history inside the app matches the account web history, so session metadata leaves. That is the whole of what was observed about data leaving, and it is not a residency assessment. If residency is the reason you are buying, that is its own lab with payload-level instrumentation this one did not have.

**Independence and disclosure:** Self-funded editorial. No part of this lab was sponsored. No vendor commissioned it, funded it, previewed it, or saw any of it before publication, and the product was installed from the vendor public channel and run without their involvement. Disclosure: Google Cloud is a client of The CTO Advisor LLC, and Google Gemma 4 appears here as the control arm. NVIDIA, Alibaba and Perplexity are not clients. Anthropic Claude Code is the generic agent harness in the comparison and is implicated in the second finding rather than being a neutral instrument.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Coverage: asks permission before sending any individual step | Asked once, then not again | OBSERVED. Consent is per resource class, not per step: web egress was granted once and never re-asked, while filesystem access prompted several times in one run. Neither the coverage framing nor a flat once-per-tool reading survives |
| Coverage: a PII classifier shows exactly what would leave the device | Never saw that flow | OBSERVED absence. The prompt that did appear was a sandbox-exit request showing a command line, not a payload preview and not a PII analysis |
| Coverage: Terminal Bench 2.1, 59.6% to 73.0% with a frontier advisor | Cannot reconcile | The shipped advisor is one question asked over the signed-in account, and the config carries no field naming an advisor model. It was never enabled on this box, so everything known about that path here is documentation |
| Coverage: runs on your own DGX Spark | True, and it assumes a screen | OBSERVED. Sparks are headless appliances. Running the product on the hardware its own launch names took a virtual framebuffer, a window manager and a compositor |
| Absent from launch coverage: ships its own patched inference engine | vLLM nightly, 2.7GB | OBSERVED. A patched build with an lmheadfix, running as its own container. It explains the hard NVIDIA requirement better than "needs an RTX card" does, and nobody reported it |
| Status bar while the run fetched an external page | You are working locally | OBSERVED, on the signed-in session. Session history in the app also matches the account web history, so session metadata does not stay on the box |
| Benchmarked as a coding agent, ships as | docs, pdf, pptx, xlsx, mail | the skill manifest retires an entry named coder while the binary still carries apply_patch and run-code machinery. The tool primitives and the shipped catalog disagree |
| Sandbox-exit grants asked for, per five repair tasks | 7 | auto-approved by the runner; each one a capability grant, not an action approval |
| Same five tasks, vendor stack | 5/5 · 2,331s | 330 tool calls, 16 edits, sequential |
| Same five tasks, hosted frontier | 5/5 · 379s | Opus 4.8 through the same harness; the row that answers the buy-a-box question |
| Output tokens to clear the same five gates | 48,961 vs 84,618 | vendor stack against the generic harness on the same open weights. Against fifteen prior Claude Code arms on these same five tasks, spanning 10,968 to 372,120 output tokens, the 84,618 is the lowest of the three that solved all five, so it is not an unusual run by our own standards. That is a statement about our arms and not about the gap. All fifteen crossed the same translation proxy and the vendor product crossed none, and a confound shared across a distribution leaves the ranking intact while saying nothing about the absolute comparison to a run outside it. The gap is recorded here, not attributed |

## Authority placement (DAPM)

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 0 (Compute) | Retained | The owner’s hardware, the owner’s electricity. The vendor ships its own engine image but it runs on your box, and the whole local stack was reproduced standalone with no vendor application present. Nothing about compute placement is ceded. |
| Layer 2A (Orchestration) | Retained | The sandbox-exit gate. The operator decides, every time, and deny is enforced by the sandbox rather than by the prompt, so by this framework’s own use of the word the authority is retained. What the framework does not track is the shape of the retention, and that is where the cost sits: it is manual, per-occurrence, scoped to the account rather than the task, of a duration that outlives the run by the vendor’s own statement, and ruled on against a justification the model writes for itself. The controls that do exist are org-level and sit around this point rather than on it: disable the product, govern connectors, collect audit logs, set retention. None of them lets an operator or an organization rule once that this agent never runs unsandboxed and have that ruling hold. Retained here means the human can refuse and the refusal holds. It does not mean the organization can encode that refusal, distribute it, enforce it, or audit it against a policy, and those are different properties that this framework does not separately track. Lab 5 named the consequence before this lab met it. Authority you reclaim is authority you run. |
| Layer 2C (Reasoning) | Delegated | Reasoning runs on owned weights the owner can serve independently, which is the strongest ownership position in the product. It is delegated rather than retained because the orchestration, tool scoping and depth limits are the vendor’s and there is no outbound interface to drive them. |
| Layer 3 (+1) (Applications) | Ceded | No public API, no CLI that drives the agent, nothing to embed. The application is always the caller and never the callee, so the workflow cannot be composed into anything the enterprise already runs. |

## Seam map — readiness (projected, pending bench)

| Function | Readiness |
| --- | --- |

## What this does not establish

- It did not measure data residency. Byte counters showed task-correlated traffic to object storage far above idle, replicated across three working windows, with contents encrypted and unidentified. Volume is not content and the lab claims neither direction from it. An earlier at-rest figure asserted here was withdrawn: it rested on a single window that two other captures contradict by two orders of magnitude, which means the control window was not controlled. The byte counts sit in the notes as the seed of a residency lab, not as a result of this one.
- It did not check the headline benchmark claim. The published Terminal Bench figure attributes a large gain to a frontier advisor. That path was never provisioned on this box, so the number is neither confirmed nor contradicted here. What can be said is narrower: the shipped advisor is one question asked over the signed-in account, and the configuration carries no field naming an advisor model, so the mechanism behind the published figure and the mechanism in the shipped product could not be reconciled from the surface.
- It did not prove data left the box. The canary document was not retrievable from the vendor’s search index, which establishes non-retrievability and nothing more. A web search cannot see an internal store, same-day ingestion would not be indexed, and telemetry is invisible to this probe. Claim it as not retrievable, never as did not leave.
- It did not prove the vendor harness is better engineered than the generic one. It measured that the vendor stack reached the same verdicts on fewer tokens and less wall clock, with a translation proxy on our side of the scale that we added and they do not have.
- It did not measure the advisor, the escalation path the launch coverage leads with. That path was never enabled on this box.
- It did not test the product at the job it ships for. The skill catalog is documents, spreadsheets, presentations and mail. This lab ran a code-repair pool because that is where the instrument and the corpus are. The one knowledge-work observation, a structured assessment over two local files, was competent and correctly grounded.

## Open questions

- **Is a granted sandbox exit scoped or general?** The agent self-reported that after a grant, a plain unrelated command ran without further approval. That self-report was later shown accurate on a different point, so it is credible, but it is not verified. One command to an unrelated host inside an existing session settles it, and it decides whether the grant is a door or a doorway.
- **What does the advisor actually do?** Never invoked here, across fifteen recorded sessions and 6,142 audited events, every one of them at depth zero with no subagent ever spawned. The tools that did run were shell 577 times, read 129, edit 23, write 2, and load_skill 4. An earlier version of this entry explained that by saying the configuration file was never provisioned. That was wrong: the file was written partway through the observation window, so three sessions ran without it and twelve ran with it, and the advisor was available for those twelve and reached for in none of them. The published Terminal Bench figure attributes a large gain to that path, which makes the corrected statement narrower and more useful than unavailability. Offered the escalation, the product did not take it on this workload.
- **How much of the wall-clock gap is context length rather than anything either vendor did?** Open, and it is an apparatus question rather than a result about either product. A venue note recorded in the lab notes: on one idle engine the same weights decoded far slower at agentic context sizes than at benchmark ones. Until that is measured properly across contexts and replicated, this lab claims wall clock end to end and does not decompose it into a decode term. The token-volume difference, 48,961 against 84,618 output tokens for the same gates, is independent of context and concurrency and is the half that stands.
- **Does the translation proxy change token volume, not just rate?** A faithful proxy should not change how many tokens a model emits. If reasoning blocks or tool results are reshaped such that context is lost between turns, the model re-derives and turn counts grow. Two observations sit near this and neither settles it.

## Objection handled

**You are describing a permission dialog. Every agent has one, and this one is more honest than most.**

It is more honest than most, and the lab says so. Perplexity states the blast radius in the prompt, in plain language, including that descendants outlive the run. Deny is enforced by the sandbox rather than by the prompt, and denied, the agent degraded gracefully and did not fabricate. Those are real engineering choices and they deserve credit before the disagreement starts.

The objection is not to the dialog. It is to the absence of anything behind it. A permission prompt is an event. Governance is a rule that survives the event. A governing rule needs a subject, an allowed or prohibited capability, a scope, a condition, a duration, an approver, and an audit record. This surface presents an event. It does not expose the rule. There is no policy surface, no standing prohibition, no allowlist of permitted commands, and no way to encode a decision once so it binds the next session or a colleague. That is what makes the fatigue structural: the operator is not ruling once and being done, they are ruling every time, against an argument the model composes fresh each occurrence. An enterprise cannot delegate authority to a control it cannot configure.

## Summary

The launch made a specific set of claims and this lab checked them one at a time. Two hold. It runs a post-trained Qwen 3.8 27B on the owner hardware, and the personally identifiable information masking really is a separate model running locally. Two do not. It does not ask before every individual step, because consent is granted per resource class and web egress was approved once and never re-asked, while filesystem access prompted repeatedly in the same run. And the classifier that was going to show exactly what would leave the device never appeared at all; the prompt that did appear was a request to run a command outside the sandbox. One claim could not be checked: the frontier advisor behind the headline benchmark figure was never available on this box, so everything this lab knows about that path is documentation rather than observation. One thing nobody reported at all is that the product ships its own patched inference engine, which explains the hardware requirement better than the coverage did.

The authority half is retained, and that is the expensive outcome rather than the reassuring one. When the agent needs out of the sandbox it presents a dialog showing one command line, and what the operator grants is a process running as their account with access to files, network, local services and processes, whose descendants may continue after the run ends. The vendor states this plainly, which is to their credit. The problem is what happens next: nothing. There is no policy surface, no standing rule, no allowlist. The same decision is re-presented on every occurrence, and the evidence the human weighs is the model’s own written justification for why it needs out. Lab 3 put the control point on a deterministic rule and explicitly not on the analyst’s self-assessment. This inverts that, and Lab 5 supplies the price: authority you reclaim is authority you run, so an unencodable gate is an operator you are now staffing.

On capability, the answer to the question everyone asked is that it is the model. The same open weights under a generic agent harness on the same class of box clear the same gates. What the vendor adds is the assembly, and it is worth real time. The serve configuration was extracted and reproduced standalone in an afternoon, so that part is copyable. The rest is not, and the honest evidence for that is our own week: a proxy pointed at a dead port for a day, four arms voided by venue misconfiguration, one stalled at three turns. Lab 1 said borrow the plumbing and keep the judgment. This is what the plumbing is worth when you price it by trying to lay it yourself.

The comparison carries a confound that belongs to us and points against the result. Claude Code speaks one API and the local engine speaks another, so every request in our arms crossed a translation proxy that exists because of the lab design. Portable Computer has no such layer. What it costs is unresolved. A small probe put it in single digits, which describes small probes and not the arm; an attempt to measure it at the context an agentic session actually carries returned the wrong sign, meaning the probe could not resolve the difference rather than that the difference is small. The honest form of the claim is about the combination, not about the harness.

For an architect the split is clean and it is not the split the launch described. The model is real and it runs where they said it does. The controls are thinner than announced, and the one that matters most is not missing so much as unrepeatable: you hold the decision, every time, with nowhere to write it down. Do not buy it for time either, because a hosted frontier model finished the same five tasks in 379 seconds against 2,331 on owned hardware. And do not read a residency verdict into any of this. This lab did not test residency; it observed that session history inside the app matches the account web history, so session metadata leaves, and it measured task-correlated traffic to object storage far above idle across three windows without identifying any of it. Volume is not content, and neither observation is a residency assessment. Running the model locally relocates where the tokens are computed. It does not, by itself, put you back in the chair.

## Detail

Perplexity shipped a desktop agent that runs a large language model on hardware you own, and the coverage around it made a specific set of promises. Runs locally. Asks before every step. Shows you exactly what a classifier would let leave the device. Escalates to a frontier advisor for a large benchmark gain. That list is a measuring yard. So which parts survive contact with the shipped product? I installed it on the hardware its own launch names and went through the list. The first thing I learned wasn’t on the list. A DGX Spark is a headless appliance, and this is a desktop application that assumes a screen and a person sitting at one. Getting it running took a virtual framebuffer, a window manager, and a compositor. That isn’t a criticism of the product. It’s the first sign that the thing announced and the thing shipped were described by different people.

Two of the claims held up. It really does run a post-trained Qwen 3.8 27B on hardware you own. The personally identifiable information masking really is a separate model, running locally. The coverage underplayed that one rather than overselling it. Two of them did not survive the check. It doesn’t ask before every individual step, whatever the coverage said. Consent turns out to be granted per resource class, so web access was approved once and never raised again, while file access prompted repeatedly in the same session. And the classifier that was going to show me exactly what would leave the device never appeared at all. What appeared was a request to run a command outside the sandbox. One claim I couldn’t check, because the frontier advisor behind the headline benchmark number was never provisioned here, so everything I know about that path is documentation. Then there’s the thing nobody reported: the product ships its own patched inference engine as a container. That explains the hard hardware requirement better than any sentence in the launch coverage did.

Follow the widest of those gaps and it lands somewhere more useful than a feature checklist. The permission dialog is honest, and it deserves credit before I disagree with it. It states the blast radius in plain language, including that processes it starts may outlive the run. Deny is enforced by the sandbox rather than by the prompt. Denied, the model said it was blocked and didn’t invent an answer. So what’s actually being approved? The screen shows one command line. What’s granted is a process running as your account, reaching your files, your network, your local services. And the justification you rule on was written by the model, in the model’s framing, at the moment the model wants out. Lab 3 put the control point on a deterministic rule and explicitly not on the analyst’s self-assessment. This inverts it.

So the authority is retained, which sounds like the good answer. It is the expensive one. There’s no policy surface at the decision point, no standing rule, no allowlist. The Enterprise tier governs around that point rather than on it: disable the product, control connectors, collect audit logs. This is a single operator surface and not a platform for building on, so judge it as one. However, where do you write down a decision? Nothing lets you rule once that this agent never runs unsandboxed and have that hold tomorrow, or hold for a colleague. Lab 5 named the price before I met it here. Authority you reclaim is authority you run. You run this one every prompt, which turns consent fatigue from a user failing into an operating cost you have to staff.

They benchmark it as a coding agent and ship it as a knowledge worker’s assistant. The skill catalog is documents, spreadsheets, presentations, and mail. The manifest retires an entry named coder, while the binary still carries patch-application and code-running machinery. Which product is being reviewed, then? I ran a code-repair pool because that is where my instrument and my corpus are. It is a fair test of the benchmark they published, and not a test of the job the catalog describes. The single best thing I saw all week was on the other side of that line. Given two local files and a method document, the local model produced a structured assessment. It was correctly grounded, applied the rule it was handed, and held the register. It also echoed the end marker, which told me it had read the whole instrument. That is the use case. It is not the one on the leaderboard.

The capability question people kept asking has a dull answer. Is the model that good, or is the packaging better? It’s the model. The same open weights under a generic agent harness on the same class of box cleared the same gates. What the vendor adds is assembly, and assembly isn’t free, which is where I have to be careful about my own week. Our comparison ran through a translation proxy that exists only because our harness and their engine speak different application programming interfaces. Their product has no such layer. That proxy sits on our side of the scale, and I couldn’t measure what it costs at the context an agentic session actually carries. So I record the gap and don’t attribute it. What I’ll stand behind is narrower. A hosted frontier model finished the same five repairs in 379 seconds against 2,331 on owned hardware, and every arm solved five of five. Whatever the case for running this locally is, it isn’t speed. It’s the data plane. Just don’t read a data plane as a control plane, because this lab is the difference between the two.

## How it abstracts

The transferable question is not where the model runs. It is whether a decision can be written down, and that is testable rather than rhetorical. A product has programmable governance only if an authorized administrator can express a rule carrying all seven of: a subject, user, group, agent identity or workload; a capability, command execution, destination reachability, connector invocation, file scope or cloud escalation; a scope, task, session, device, repository, destination or tenant; a condition, data classification, environment, risk state or business process; a duration, one-time, bounded, standing or revocable; an authority, the designated approver or delegated owner; and a record, the policy identity, the matching conditions, the decision, any override and the execution outcome. Run that against Portable Computer at the sandbox-exit gate and none of the seven can be represented. It has honest consent. It does not yet have programmable governance, and those are different products to buy. Apply the same seven to any local-first agent before a pilot, because the first two rows can be true while an operator still rules alone, every time, against an argument the model wrote. Any agent that can be granted a capability needs three things before it belongs in an enterprise: a scope narrower than the account, a duration shorter than forever, and a surface on which a ruling persists. Portable Computer fails all three. The scope is the whole account, the duration outlives the agent run by the vendor’s own statement, and there is no surface on which a ruling persists. It describes the first two failures honestly and in plain language, which is worth more than most disclosures and is not the same as meeting the requirement. Evaluate any local-first agent on the third, because the first two can be true and still leave every operator ruling alone, repeatedly, against an argument the model wrote.

Local-first and control-first are separate axes and vendors sell them as one. Moving the data plane onto owned hardware is a real and defensible reason to buy. It answers residency, it answers some regulatory questions, and it is the half this product delivers. It says nothing about who holds reasoning-plane authority, and on the evidence here the two can move in opposite directions at the same time.

When a vendor packages open weights, separate the capability from the assembly before pricing either. The capability travelled: the same weights under a different harness cleared the same gates. The assembly did not, and this lab is poor evidence that it is cheap. Our own generic arm needed a translation proxy, and that proxy spent a day pointed at a dead port; four arms were voided by venue misconfiguration; one stalled at three turns. That is the build path, priced honestly, over one week, by people who do this for a living. The corpus has said the same thing from three other directions: borrow the plumbing and keep the judgment (Lab 1), buy the harness rather than the tier (Lab 12), and expect that authority you reclaim is authority you run (Lab 5). Lab 19 does not revise that line. It confirms it from the vendor side.

## Method and disclosure

Self-funded editorial. The instrument is loopcontrolbench, open under MIT: the reproduce-or-drop gate, the sandboxed runner, and the Claude Code harness driver. Five tasks from the Lab 11 pool, scored by each project’s own tests, with no model judging anything.

Product observation ran on a headless DGX Spark under a virtual framebuffer, because the product is a desktop application and the hardware its launch names has no display. That apparatus is written up separately. Egress was measured by polling socket byte counters owned by the application’s own processes, against a matched idle control of the same duration; TLS means destinations and volumes only, never payloads.

Declared confound: Claude Code speaks the Anthropic Messages API and the local engine speaks OpenAI, so every request in the comparison arms crossed a litellm translation proxy. That proxy exists because of the lab design and the vendor product has no equivalent. It sits on our side of the scale and inflates any lead measured for the vendor stack. A small probe put it in single digits and that figure describes small probes only. The attempt to measure it at arm-scale context returned the shimmed path as faster than direct, which is the wrong sign, so the probe cannot resolve it and no arm-scale figure is claimed.

Two arms were stopped before completion on the operator’s call once it was clear the venue was producing times far outside the reference, and both are recorded rather than deleted. Every arm and its results are archived off the venue; nothing cited here exists only on the box that produced it.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*