# The second box works. The playbook doesn’t.

> Lab 014 · Editorial lab · Status: published  
> Published by The Advisor Bench LLC · Layer2C Labs

**Question:** Buy a second small AI box and cluster it, and you get a bigger, faster tier for less than a bigger box costs. That is the pitch. Following the vendor’s documented procedure produces a server that answers one request and dies on four.

**Load:** Twenty-two certified Python repair tasks driven through a headless agentic harness, plus a concurrency sweep against the serving endpoint.

## Executive Summary

Two DGX Sparks were clustered over a 200 Gb/sec RoCE link and measured serving DeepSeek V4 Flash 0731, a 156 GiB mixture-of-experts checkpoint with FP8 weights and an NVFP4 key-value cache, tensor-parallel across both nodes. The pair sustained 16 concurrent requests with zero errors at 215 tok/s aggregate, and cleared 22 of 22 tasks on the certified repair pool with no timeouts and no session errors.

The same hardware, running the vendor-documented configuration on a stock vLLM build, deadlocked at four concurrent requests. Nine configuration hypotheses were tested against an identical load and every one failed at the same point: a shared-memory read timeout inside the inter-worker synchronisation path.

The difference between the two outcomes was the runtime build, not the hardware, the fabric, or the orchestration framework. Both documented multi-node paths failed identically on the stock image. Holding the model fixed and changing only the image settles it: the same 31B model that failed nine times served 16 concurrent requests with zero errors at 169.7 tok/s on the working build.

The failure mechanism survives in both builds. The working image logs the same shared-memory starvation, a 60 second wait for a broadcast block, as an informational line and continues. The stock image raises a timeout from the same call and kills the engine. One build tolerates the condition; the other does not.

Interconnect capability was never the constraint. A NCCL all-reduce at this model’s activation size measures 49.6 microseconds, which puts collective overhead at roughly 8 percent of the token budget at tensor-parallel width two.

Five models were qualified against the pair and one served. DeepSeek V4 Flash 0731 is the one that works. The others failed for four different reasons: Qwen3-Coder 480B-A35B needs 273 GB at four-bit, more than the pair holds before a single token of cache; Qwen3-235B-A22B-2507 fits easily and serves on one node but fails engine initialisation at tensor-parallel width two; Devstral 2 123B fits and clusters and then decodes six times slower than the model that works; and Nemotron-3 Super 120B runs on a single box at ten times the wall clock per repair. Fitting in the combined memory is a necessary condition and not close to a sufficient one.

The pattern across those attempts is that sparsity is what admits a large model to this tier and expert routing is what excludes it. Both dense models clustered here, Gemma 4 31B and Devstral 2 123B, did so without incident. Of the two mixture-of-experts models put across the link, only DeepSeek V4 Flash clustered, and it runs on a container image built specifically for its architecture. Tensor parallelism needs a per-layer all-reduce, which every runtime implements well. Mixture-of-experts additionally needs all-to-all expert dispatch through fused kernels bound tightly to hardware generation and library version, and dense models never enter that path.

## DAPM Table — Authority Verdict

| Layer | Placement |
| --- | --- |
| layer0 | Retained |
| layer2c | Retained |

## Seam Map — Readiness

| Function | Readiness |
| --- | --- |
| Serve a model larger than one box | ready |
| Concurrent agentic serving | ready |
| Follow the vendor multi-node playbook | needs-cloud |

## Detailed Writeup

The pitch for a second small box is straightforward. A model that will not fit in 128GB fits in 256GB, the interconnect is fast enough, and two boxes cost less than one larger machine. Lab 013 closed by naming exactly this gap, and the second Spark was bought to close it.

The vendor publishes a playbook for this. Its multi-node section prescribes a Ray cluster with tensor parallelism across both nodes, and that is where this lab started. On a stock vLLM build serving Gemma 4 31B, dense, that configuration answered a single request at 11.2 tok/s and then deadlocked the moment four requests arrived together. One node sat at 96 percent reported utilisation drawing 20 watts, which is a busy-wait spin rather than compute, while the other sat at zero.

Nine configuration hypotheses were tested against that identical load. The engine watchdog was raised from 300 to 1800 seconds. Step size was capped so no forward pass could approach the timeout. The RoCE GID index, which was unset and free to select a link-local IPv6 entry, was pinned to the IPv4 entry the fabric actually exposes. CUDA graphs were made breakable, then removed entirely. The RDMA memory-lock limit was lifted. Three NCCL settings that the working kit uses were adopted. The orchestration was moved off Ray onto vLLM’s own native multi-node path, which the playbook never mentions. Tensor parallelism was swapped for pipeline parallelism.

Every one of those failed at the same place. The traceback names a shared-memory broadcast read that times out, and POSIX shared memory does not cross a machine boundary. That is not a setting. Ray turned out to be a passenger rather than the cause, because vLLM’s native path routes through the same primitive and fails identically.

What changed the outcome was the build. A community-maintained container image, pinned by digest, running a development build of the serving engine one point release ahead of the stock one, served the same tensor-parallel width across the same two boxes without incident. Sixteen concurrent requests, zero errors, 215 tokens per second aggregate, and the engine still healthy afterwards.

On that image the pair was given DeepSeek V4 Flash 0731, a mixture-of-experts checkpoint of 156 GiB with FP8 weights, which cannot fit either box alone. This is the capacity case the second box exists for, and it is the case Lab 013 could not reach. The model served at 35 tok/s single-stream, which is five times what the same hardware manages with a 31B dense model on one box.

Then it ran the instrument. Twenty-two certified repair tasks, the same pool six earlier local models have run, driven through a headless agentic harness with a deterministic gate deciding every verdict. It cleared all twenty-two. No timeouts, no session errors, three API retries across four and a half hours.

The tasks it took are the interesting part. One repair in that pool had never been solved by any self-hosted model across six arms, and the clustered pair took it in 34 minutes over 57 turns. Another took 208 turns without losing its place in the instrument. That is what instrument literacy looks like when it holds under load, and it is the capability Lab 013 identified as the thing that separates models rather than parameter count.

The comparison that matters commercially is against renting. A hosted frontier model cleared the identical pool in 38 minutes for $3.14. The clustered pair took 4.5 hours and nothing. Seven times the wall clock, zero marginal cost, on capital already spent. Whether that trades well depends on how often the work runs and whether it can wait.

The comparison that matters architecturally is against the single box. The best self-hosted result on this pool was 18 of 22, from a model that fits in 128GB with no interconnect involved. Had the clustered pair matched that, the second box would have justified nothing. It cleared four more, including the one nothing local had ever solved.

So the hardware was never the limit. Two boxes, an in-spec cable, and a correctly configured fabric were sufficient from the first hour, and the fabric measurements said so: collective overhead at tensor-parallel width two is about 8 percent of the token budget. What stood between that hardware and a working endpoint was a documented procedure that does not work and a working configuration that is not documented.

The obvious next question is which other models this tier unlocks, and the answer is narrower than the memory arithmetic suggests. Five candidates were qualified. Qwen3-Coder 480B-A35B, the largest coder model in the same family as the one-box winner, needs 273 GB at four-bit, which exceeds both boxes before any cache. A 123B dense model fits, clusters, and decodes at 5.8 tok/s, six times slower than the sparse model that works, because decode rate tracks active parameters rather than total ones. Nemotron-3 Super 120B-A12B fits a single box and spent 74 minutes on a repair Qwen3-Coder-Next, 80B with about 3B active, finished in 7. And Qwen3-235B-A22B-2507, which fits comfortably and carries the context length the harness needs, fails engine initialisation at tensor-parallel width two with a kernel signature mismatch.

That last failure is the one worth generalising. Both dense models put across the link, Gemma 4 31B and Devstral 2 123B, cluster without incident. Of the two mixture-of-experts models tried, only DeepSeek V4 Flash clusters, and only on a container image built for its architecture; Qwen3-235B-A22B-2507 fails engine initialisation at width two. The mechanism is that tensor parallelism asks for a per-layer all-reduce, a collective every runtime handles, while mixture-of-experts additionally needs all-to-all expert dispatch through fused kernels pinned to a hardware generation and a library version. Dense models never touch that code path. So the sparsity that lets DeepSeek V4 Flash fit inside 256 GB is the same property that makes it hard to split across two machines, and the vendor and community record agrees: multiple reports show Qwen mixture-of-experts models serving at width one and failing at width two, with no working two-Spark demonstration published.

Which turns the purchasing statement into something a spec sheet cannot express. The second box does not buy access to models above 128 GB. It buys access to models above 128 GB that are also either dense, or accompanied by a runtime built for them. Today that is a set of one.

## Method and Disclosure

Self-funded. Both DGX Sparks, the QSFP112 cable, and the power were bought by the practice. No vendor was briefed, no vendor reviewed this, and no vendor was given advance sight.

The repair pool, the reproduce-or-drop certification, and the deterministic gate are the same instrument used in Labs 011 through 013, unchanged, so the solve counts sit on the same axis as the earlier arms.

Serving configuration is recorded verbatim in every result file for this lab, including the image, the parallelism width, the sequence cap, and the fabric. Earlier arms in this corpus did not record it, which is a gap this lab does not repeat.

The harness, the gate, and the certification tooling are public. The assessment methodology, the vendor grades this evidence feeds, and the judgment that turns a score into a placement are not.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*