The second box works. The playbook doesn’t.

Two NVIDIA DGX Sparks, GB10, 128GB unified memory each, joined by one 200 Gb/sec RoCE link. One mixture-of-experts checkpoint of 156 GiB, FP8 weights with an NVFP4 key-value cache, run twice on the certified pool with zero flips, one 31B dense model used to isolate the runtime, and three further models qualified and rejected. Everything downstream of that score is a single observation: the concurrency figures, the decode rate, and the nine eliminated configuration hypotheses were each measured once. Everything here is about small unified-memory boxes over RoCE. It says nothing about clusters built from cards a vendor validates for the purpose.

Self-funded. Both Sparks and the interconnect were bought by the practice at retail, and no vendor saw this before publication or had any say in it. NVIDIA hardware is under test and NVIDIA documentation is criticised here by name. The serving stack that worked is a community-built container image, not a vendor artifact, and that is part of the finding rather than a footnote. Disclosure: Google Cloud is a client of this practice, and a Google model, Gemma 4 31B, is the isolation subject behind the runtime finding. Google did not commission, fund, preview, or have any input into this lab. No other vendor named here is a client.

The pitch for a second small AI box is clean arithmetic. A model that won't fit in 128GB fits in 256GB, the interconnect is fast enough, and two boxes cost less than one larger machine. Lab 013 closed by naming exactly that gap, so I bought a second NVIDIA DGX Spark, joined the pair over a 200 Gb/sec Remote Direct Memory Access over Converged Ethernet (RoCE) link, and followed the vendor's documented multi-node procedure. What that procedure produces is a server that answers one request and dies on four. So the question this lab ended up answering isn't whether the second box works. It's what actually stands between the hardware and the tier it was bought for.

Nine fixes, one traceback

The published DGX Spark playbook prescribes a Ray cluster with tensor parallelism across both nodes. On a stock vLLM build serving Gemma 4 31B, a dense model, that configuration answered a single request at 11.2 tokens per second and deadlocked the moment four requests arrived together. One node sat at 96 percent reported utilization drawing 20 watts, a busy-wait spin rather than compute, while the other sat at zero. The power meter told the truth the utilization counter didn't.

Nine configuration hypotheses went against that identical load. The engine watchdog was raised from 300 to 1800 seconds. Step size was capped so no forward pass could approach the timeout. The RoCE Global Identifier (GID) index, which was genuinely misconfigured and free to select a link-local IPv6 entry, was pinned to the IPv4 entry the fabric actually exposes. CUDA graphs were made breakable, then removed entirely. The RDMA memory-lock limit was lifted. Three NCCL (NVIDIA Collective Communications Library) settings from a working kit were adopted. The orchestration was moved off Ray onto vLLM's own native multi-node path, which the playbook never mentions. Tensor parallelism was swapped for pipeline parallelism. Every one failed at the same place: a shared-memory broadcast read that times out, and POSIX shared memory doesn't cross a machine boundary. Ray was a passenger, not the cause. Several of those fixes repaired real problems. None of them was the problem, and if I'd stopped after any one of them I would have published a confident finding about the wrong cause.

Only the image changed

What changed the outcome was the runtime build. A community-maintained container image, pinned by digest and carrying a development build of the serving engine one point release ahead of the stock one, served the same tensor-parallel width across the same two boxes without incident. The isolation is clean because the model was held fixed: the same Gemma 4 31B that failed nine times served 16 of 16 concurrent requests at 169.7 tokens per second aggregate with zero errors, with the hardware, parallelism, and fabric settings untouched. The failure mechanism even survives in both builds. The working image logs the same shared-memory starvation, a 60 second wait for a broadcast block, as an informational line and continues. The stock image raises a timeout from the same call and kills the engine. Why one build tolerates the condition is not established here, and I make no claim about which commit, patch, or flag is responsible.

What saved the day was a working reference. Somebody had already built the thing and put the scripts in public, and reading their launcher answered in ten minutes what a day of configuration had not. I'd had that repository cloned on disk for hours before I opened the file that mattered. Price what that means, though. The configuration that works runs on a community-maintained image with a thin patch layer on top. That's a supply chain position, not a vendor-supported one, and somebody in your organization has to own it.

Twenty-two, twice

On that image the pair got the model the second box exists for: DeepSeek V4 Flash 0731, a 156 GiB mixture-of-experts checkpoint with FP8 weights and an NVFP4 key-value cache, which cannot fit either box alone. It served at 35 tokens per second single-stream, roughly five times what the same hardware manages with the dense 31B on one box, and sustained 16 concurrent requests at 215.4 tok/s aggregate with zero errors. Then it ran the instrument: the 22 certified repair tasks six earlier local models have run, driven through a headless agentic harness with a deterministic gate deciding every verdict. It cleared all 22. Run again on the pinned configuration, it cleared all 22 again with zero tasks changing hands, at 4.49 hours then 4.19, 869 turns then 858. That replication matters because this pool moves. The best single-box model, Qwen3-Coder-Next, scored 18 of 22 on both of its runs with six different tasks flipping, three each way. The clustered pair showed none of that variance, and it took the one repair no self-hosted model in this corpus had ever solved, in 34 minutes over 57 turns. Another repair ran 208 turns without the session losing its place. That's what instrument literacy looks like under load.

The commercial comparison is blunt. A hosted frontier model, gpt-5.4-mini, cleared the identical pool in 38.4 minutes for $3.14, and it will cost that again every time it runs. The pair took 4.5 hours and nothing, on capital already spent. Seven times the wall clock, zero marginal cost, and where that trades depends on how often the work runs and whether it can wait. The architectural comparison is the one that justifies the purchase. Had the pair matched the single box's 18, the second box would have bought nothing. It cleared four more.

A tier of one

Then I asked which other models this tier admits, and the answer is narrower than the memory arithmetic suggests. Five candidates were qualified and one serves. Qwen3-Coder 480B-A35B needs 273 GB at four-bit, more than the pair holds before a single token of cache. Qwen3-235B-A22B-2507 fits comfortably, serves on one node, and fails engine initialization at tensor-parallel width two with a kernel signature mismatch. Devstral 2 123B, dense, fits and clusters and then decodes at 5.8 tok/s, six times slower than the sparse model that works, because decode rate tracks active parameters rather than total ones. And Nemotron-3 Super 120B fits a single box and spent 74 minutes on a repair Qwen3-Coder-Next finished in 7, which is an economics problem no eventual score can pay back.

The pattern worth generalizing sits in those failures. Both dense models put across the link clustered without incident. Of the two mixture-of-experts models tried, only DeepSeek V4 Flash clustered, and only on an image built for its architecture. The mechanism: tensor parallelism needs a per-layer all-reduce, a collective every runtime handles, while mixture-of-experts additionally needs all-to-all expert dispatch through fused kernels pinned to a hardware generation and a library version. Dense models never enter that path. So the sparsity that lets DeepSeek V4 Flash fit inside 256 GB is the same property that makes it hard to split across two machines. Four models is a small sample supporting a mechanism, not a law, and a single counterexample overturns it. The cheap next test is one more mixture-of-experts model on a newer runtime build. If the wall moves with the runtime, the constraint is a software generation and it ages out. If it doesn't, it's architectural and belongs in the purchase decision permanently.

The interconnect was never the constraint. An NCCL all-reduce at this model's activation size measures 49.6 microseconds, which puts collective overhead near 8 percent of the token budget, and predicting that cost from the raw 9.43 microsecond RDMA write latency would understate it five times over. The hardware was sufficient from the first hour. What stood between it and a working endpoint was a documented procedure that doesn't work and a working configuration that isn't documented. I've said before that the validator decides, not the model, and this lab is the hardware version of that: the measurement that mattered was a power meter reading 20 watts next to a utilization counter reading 96 percent. So the purchasing statement lands somewhere a spec sheet can't express. The second box doesn't buy access to models above 128 GB. It buys access to models above 128 GB that are also either dense or accompanied by a runtime built for them. Today that's a set of one, and whether that wall is a software generation or an architecture is the question this lab leaves standing.

The numbers

Where each layer belongs

Method and disclosure

Self-funded. Both DGX Sparks, the QSFP112 cable, and the power were bought by the practice. No vendor was briefed, no vendor reviewed this, and no vendor was given advance sight.

The repair pool, the reproduce-or-drop certification, and the deterministic gate are the same instrument used in Labs 011 through 013, unchanged, so the solve counts sit on the same axis as the earlier arms.

Serving configuration is recorded verbatim in every result file for this lab, including the image, the parallelism width, the sequence cap, and the fabric. Earlier arms in this corpus did not record it, which is a gap this lab does not repeat.

The harness, the gate, and the certification tooling are public. The assessment methodology, the vendor grades this evidence feeds, and the judgment that turns a score into a placement are not.

The essay on this page was rendered from the lab’s frozen structured record by a model writing under this practice’s voice specification, with a deterministic gate checking every figure against the record before publication. The record is the canonical surface: where prose and record disagree, the record rules.