# Lab 014 raw lab detail: The second box works. The playbook doesn't.

Two NVIDIA DGX Sparks, GB10, 128GB of unified memory each, joined by a single 200 Gb/sec RDMA
over Converged Ethernet (RoCE) link. Everything below was measured on that pair or on one of its
two nodes. Nothing here was rented.

## The one question

Buy a second small artificial intelligence (AI) box, cluster it, and get a bigger tier for less
than a bigger box costs. That is the pitch every vendor makes about small-node clustering.

The question is not whether two boxes hold more memory than one. They obviously do. The question
is whether the documented path from two boxes to a served model works, and whether the models
that path unlocks are worth the interconnect.

## The instrument

Twenty-two Python repair tasks, each reconstructed from a real maintainer-merged pull request,
pinned to a repository and a base commit. Each task carries held-out tests the model never sees.
A deterministic gate applies the model's edit, runs those tests in a network-isolated container,
and rules. The model never scores itself.

The pool is the certified miss set: tasks that models have failed before. It is the hard end of a
70-task bench, not a random sample, and every score below is against that hard end.

The harness is a headless agentic session with file read, edit, and shell. The model orients,
reads, edits, tests, and declares done. The gate decides whether it was.

## The measured spine

| What | Value | Note |
| --- | --- | --- |
| Certified pool, clustered DeepSeek V4 Flash | 22 of 22, twice | Zero flips between runs |
| Run 1 / run 2 wall clock | 4.49 h / 4.19 h | 869 and 858 turns, zero timeouts either run |
| Concurrency sustained | 16 of 16 | 11,200 tokens in 52 s, 215.4 tok/s aggregate |
| Vendor path, stock runtime | 0 of 16 | Deadlock at four concurrent and above |
| Same 31B, working runtime | 16 of 16 | 169.7 tok/s aggregate, only the image changed |
| NCCL all-reduce at activation size | 49.6 us | Raw RDMA write is 9.43 us at 64KB |
| Models qualified for the pair | 1 of 5 | Fitting in 256 GB is necessary, not sufficient |
| Clustering by architecture | dense 2 of 2, mixture-of-experts 1 of 2 | Tested here, not counting community reports |
| Metered spend | $0 | Power and about a day of setup |

The replication is the load-bearing part. The pool moves: Qwen3-Coder-Next scored 18 of 22 on
both of its runs with six different tasks flipping, three each way. DeepSeek V4 Flash scored 22 of
22 twice and moved nothing. So the variance is real and this model did not show it.

## Serving configuration, and why the runtime is the finding

Stock `vllm/vllm-openai:v0.25.1` deadlocks on multi-node here in both documented paths, the Ray
cluster and the native multiprocessing executor, always at the same shared-memory broadcast. Nine
configuration hypotheses were tested and eliminated. None recovered it.

The configuration that works runs a community-built image, `dsv4-0731-production:anemll-0.1.1`,
derived from a base pinned by digest and verified against that digest before the build proceeded.
That is a supply chain position, not a vendor-supported one, and somebody has to own it.

The isolation that makes this a finding rather than an anecdote: Gemma 4 31B, same model, same
hardware, same parallelism, same fabric settings, same launch shape. Zero of sixteen concurrent
on the stock image. Sixteen of sixteen on the working one. Only the image changed.

Both runs used a pinned kit with the serving configuration written down: model path, image
digest, checkpoint manifest hash, context length, sequence cap, memory utilisation, and prefix
caching state. That record is why run 2 could be a replication rather than a new measurement.

## What the pair actually unlocks, and what it does not

Five models were qualified. One served.

- **DeepSeek V4 Flash 0731.** A 156 GiB checkpoint, FP8 weights, NVFP4 key-value cache. Serves,
  clusters, 22 of 22 twice.
- **Qwen3-Coder 480B-A35B.** Needs 273 GB at four-bit. Exceeds both boxes before a single token
  of cache.
- **Qwen3-235B-A22B-2507.** Fits easily, carries the context the harness needs, serves on one
  node, and fails engine initialisation at tensor-parallel width two on a fused mixture-of-experts
  kernel signature mismatch. It never ran a task.
- **Devstral 2 123B.** Fits, clusters, and decodes at 5.8 tok/s. Too slow to use.
- **Nemotron-3 Super 120B-A12B.** Runs on one box at 74 minutes per repair against 7 for
  Qwen3-Coder-Next. Stopped on economics, not capability.

The pattern across those attempts: both dense models put across the link clustered without
incident. Of the two mixture-of-experts models tried, only the one with a container image built
for its architecture clustered at all.

The mechanism is that tensor parallelism needs a per-layer all-reduce, which every runtime
implements well. Mixture-of-experts additionally needs all-to-all expert dispatch through fused
kernels pinned to a hardware generation and a library version. Dense models never touch that path.

So the sparsity that lets a checkpoint this size run at all on two small boxes is the same
property that makes it hard to split across them.

## Findings that died in measurement, kept deliberately

**The gate tool rescued nothing.** The single-box model's failures were mostly it declaring
success without editing the file the fix belonged in, with an hour still on the clock. The obvious
fix was to give the session a tool that runs the same gate that scores it, returning one line,
authority unchanged. It was built and run. Three tasks flipped to solved, three flipped the other
way, and not one rescue came after the gate had told the model it was wrong. The model called the
tool once, at turn 26, after it had already decided it was done. Making feedback available changes
nothing if the model will not consult it while there is still time to act.

**The wall clock is not a property of the model.** Qwen3-Coder-Next reproduced its 18 of 22
exactly across two runs while its arm went from 11.4 hours to 4.8, doing 125 more turns, with 21
of 22 tasks cheaper per turn. The first run's serving configuration was an ad hoc command whose
logs were destroyed, so the difference cannot be attributed. DeepSeek, run from the pinned kit,
reproduced score and clock together. The variable that separates those two outcomes is whether the
venue was written down.

## Operational record, mistakes included

**Three instruments were measuring and nothing was listening.** Every serve script opened with
`docker rm -f`, which destroyed the previous run's container logs, so the prefix-cache hit rate
vLLM prints every ten seconds was discarded for fourteen arms. The gate already runs the whole
test file and asserts only on the target tests, so every regression it executes is parsed and then
dropped. And no arm in this campaign retained the patch it produced, which means the diffs behind
22 verified repairs are unrecoverable. Log capture and patch retention are now fixed. The
regression assertion is not: the gate still scores only the tests a task names, so a repair that
works and breaks a neighbour in the same file still reads as solved.

**Three configuration errors produced results that looked like model failures.** A wrong
tool-call parser scored 0 of 12 with no error anywhere, because the model's calls were swallowed
and the client saw an empty response. A context length set to the harness minimum rather than the
model's native maximum made 19 of 22 sessions die on `ContextWindowExceeded`. A worker timeout set
below the time a full reasoning budget needs killed the single hardest task outright. All three
were mine, all three were in a config line, and all three would have entered the corpus as
capability numbers without a session-log check.

**Prefix caching was never off.** It was suspected, then measured. On Qwen3-Coder-Next, a
two-turn shared prefix reused 46,784 of 46,853 tokens, and the instrumented arm held a 92.5%
median hit rate under four-way concurrency. DeepSeek V4 Flash ran at a 93.4% median across run 2.
The re-prefill tax this campaign worried about did not exist on either model.

## Spend

Zero metered. Both Sparks and the interconnect were bought at retail by the practice. The
comparison point is a hosted frontier model that cleared the same pool in 38.4 minutes for $3.14,
and will cost that again next time, and the time after.

## What this hands the next labs

The pair is not only a clustering partner. It is also a memory-domain control and a candidate
key-value cache tier, and only the first has been tested.

Three numbers want a home. Key-value cache per token spans 8.5 KB on this model, which pairs multi-head latent attention
with an NVFP4 cache, against 160 KB on a 24B dense one, which is a sizing table worth building. A rented discrete-GPU arm ran 84%
GPU-idle under this harness, which says a fast accelerator starves harder and bin-packing is where
the money is. And the constrained-loop apparatus reaches 21 of 22 with this model in a four-turn
budget, against 22 of 22 for the harness at roughly twenty times the prompts, which turns the
apparatus choice into a procurement question rather than an instrument preference.