{ "slug": "second-box", "lab_number": 14, "title": "The second box works. The playbook doesn’t.", "type": "editorial", "status": "published", "sponsor": null, "author": "Keith Townsend", "date": "August 8, 2026", "date_iso": "2026-08-08", "layers": [ "layer0", "layer2c" ], "axes": [ { "instrument": "4plus1", "layers": [ "layer0", "layer2c" ], "has_dapm_table": true } ], "supersedes": [], "superseded_by": null, "corrects": [], "corrected_by": [], "vendors": [ { "key": "nvidia", "role": "hw" }, { "key": "deepseek", "role": "model" }, { "key": "google", "role": "model" }, { "key": "alibaba", "role": "model" }, { "key": "mistral", "role": "model" } ], "themes": [ "serving-economics", "vendor-claim-scrutiny", "model-ownership" ], "finding": "Two clustered DGX Sparks ran a 156 GiB mixture-of-experts checkpoint to 22 of 22 on the certified repair pool, matching a hosted frontier model at zero metered cost, after the vendor’s own multi-node playbook deadlocked at four concurrent requests.", "question": "Buy a second small AI box and cluster it, and you get a bigger, faster tier for less than a bigger box costs. That is the pitch. Following the vendor’s documented procedure produces a server that answers one request and dies on four.", "load": "Twenty-two certified Python repair tasks driven through a headless agentic harness, plus a concurrency sweep against the serving endpoint.", "verdict": { "scope": "Two NVIDIA DGX Sparks, GB10, 128GB unified memory each, joined by one 200 Gb/sec RoCE link. One mixture-of-experts checkpoint of 156 GiB, FP8 weights with an NVFP4 key-value cache, run twice on the certified pool with zero flips, one 31B dense model used to isolate the runtime, and three further models qualified and rejected. Everything downstream of that score is a single observation: the concurrency figures, the decode rate, and the nine eliminated configuration hypotheses were each measured once. Everything here is about small unified-memory boxes over RoCE. It says nothing about clusters built from cards a vendor validates for the purpose.", "independence": "Self-funded. Both Sparks and the interconnect were bought by the practice at retail, and no vendor saw this before publication or had any say in it. NVIDIA hardware is under test and NVIDIA documentation is criticised here by name. The serving stack that worked is a community-built container image, not a vendor artifact, and that is part of the finding rather than a footnote. Disclosure: Google Cloud is a client of this practice, and a Google model, Gemma 4 31B, is the isolation subject behind the runtime finding. Google did not commission, fund, preview, or have any input into this lab. No other vendor named here is a client.", "calls": [ { "kind": "do", "text": "Buy the second box if you need a model that does not fit in one. A 156 GiB checkpoint served across two Sparks cleared 22 of 22 certified repairs at a 7 minute median, matching a hosted frontier model that cost $3.14 to run the same pool." }, { "kind": "dont", "text": "Do not budget the vendor’s multi-node procedure as a known quantity. The published DGX Spark playbook prescribes a Ray cluster that deadlocked here at four concurrent requests, and nine separate configuration fixes changed nothing." }, { "kind": "do", "text": "Price the runtime as a dependency. The configuration that worked runs on a community-maintained base image, pinned by digest and verified against it, with a thin patch layer on top. That is a supply chain position rather than a vendor-supported one, and somebody has to own it." } ] }, "objection": { "q": "Two boxes and an interconnect to match something you can rent for $3.14. Where is the return?", "a": [ "The rental price is per run. The pool took 38 minutes and $3.14 on the hosted model, and it will cost that again next time, and the time after. The clustered pair took 4.5 hours and nothing, on hardware that was already bought.", "So the trade is wall clock for marginal cost. Seven times slower, and free after the capital. Where that lands depends on how often you run and whether the work can wait, which is a question about your pipeline rather than about the model.", "The stronger objection is the one about the single box. Qwen3-Coder-Next cleared 18 of 22 on one Spark with no interconnect and no clustering, and it did it twice, with different tasks failing each time. If the clustered pair had landed at 18, the second box would have bought nothing. It landed at 22, and it took the one repair no self-hosted model in this corpus had ever solved.", "What does not survive scrutiny is the idea that this is turnkey. Getting from two boxes to a working endpoint took a day, and most of that day was spent on a documented path that does not work." ] }, "numbers": [ { "label": "Certified repair pool, DeepSeek V4 Flash clustered", "value": "22 of 22, twice", "note": "Two runs on the pinned configuration, 22 of 22 both times, zero flips. 4.49 h and 4.19 h; 869 and 858 turns; zero timeouts in either. Every verdict from the deterministic gate." }, { "label": "Best single-box result on the same pool", "value": "18 of 22, twice", "note": "Qwen3-Coder-Next NVFP4, one Spark. Replicated: 18 of 22 on both runs, with six tasks flipping three each way. Wall clock did not replicate. The first run took 11.4 hours on a venue whose serving config was never recorded; the second took 4.8 hours on a recorded one, doing 125 more turns. Compare the clustered figure to the score, not to the hours." }, { "label": "Hosted reference on the same pool", "value": "22 of 22", "note": "gpt-5.4-mini, 38.4 minutes, $3.14 metered." }, { "label": "Clustered wall clock", "value": "4.49 h total", "note": "Median 7.0 minutes per task, max 48.6. Turns: median 20, max 208." }, { "label": "Concurrency sustained, DeepSeek V4 Flash", "value": "16 of 16", "note": "11,200 tokens in 52 seconds, 215.4 tok/s aggregate, 13.5 per stream." }, { "label": "Single-stream decode, DeepSeek V4 Flash clustered", "value": "35 tok/s", "note": "The same hardware serves Gemma 4 31B, dense, at about 7 tok/s on one box." }, { "label": "Vendor-documented path, stock runtime, Gemma 4 31B", "value": "0 of 16", "note": "Deadlock at four concurrent and above. Nine configuration hypotheses eliminated, all failing at the same shared-memory read." }, { "label": "Same Gemma 4 31B, working runtime", "value": "16 of 16", "note": "Zero errors, 169.7 tok/s aggregate. Model, hardware, parallelism and fabric settings held fixed. Only the image changed." }, { "label": "NCCL all-reduce at DeepSeek V4 Flash activation size", "value": "49.6 us", "note": "Raw RDMA write latency is 9.43 us at 64KB. Predicting collective cost from the raw figure understates it five times over." }, { "label": "Models qualified for the pair", "value": "1 of 5", "note": "DeepSeek V4 Flash serves. Qwen3-Coder 480B-A35B needs 273 GB before any cache. Qwen3-235B-A22B-2507 fails engine init at width two. Devstral 2 123B clusters at 5.8 tok/s. Nemotron-3 Super 120B runs one box at 74 minutes per repair." }, { "label": "Clustering by architecture", "value": "dense 2 of 2, MoE 1 of 2", "note": "Clustered here: Gemma 4 31B and Devstral 2 123B, both dense, both work. DeepSeek V4 Flash works on an image built for its architecture; Qwen3-235B-A22B-2507 fails engine init at width two. A community report of Qwen3-VL-30B failing the same way at width two is corroboration, not one of these counts." }, { "label": "Metered spend", "value": "$0", "note": "Power and roughly a day of setup. The capital was already committed." } ], "opened": [ { "q": "Which build change makes the difference, and can it be carried back to the stock engine?", "a": "The variable is isolated but the mechanism is not. Running Gemma 4 31B on the working image with an otherwise identical launch gave 16 of 16 concurrent, so the build is the operative difference. What that build does differently is still open: it logs the same shared-memory starvation the stock engine dies on and recovers instead. Whether that is a point-release fix, a patch the image carries, or a build flag is the next thing to read out of the diff." }, { "q": "Does 22 of 22 replicate?", "a": "Yes, and cleanly. The pool was run a second time on the same pinned configuration and returned 22 of 22 again with zero tasks changing hands, at 4.19 hours against 4.49 and 858 turns against 869. That matters because the pool does move: Qwen3-Coder-Next scored 18 of 22 on both of its runs with six different tasks flipping, three each way. So the plus or minus two is real and this model did not show it. What remains unreplicated is everything downstream of the score. The concurrency figures, the decode rate, and the nine failed configuration hypotheses are all still single observations." }, { "q": "How many models actually qualify for this tier, and does the mixture-of-experts wall move?", "a": "One of five qualified here, and the exclusions were four different failures rather than one. The generalisation worth testing is that dense models cluster and mixture-of-experts models mostly do not, because only the latter need all-to-all expert dispatch through version-pinned fused kernels. That rests on five attempts and a community record showing the same split, which is suggestive rather than settled. The specific next test is cheap: one more mixture-of-experts model on a newer runtime build, accepting that a newer build reopens the shared-memory deadlock this one was chosen to avoid. If the wall moves with the runtime, the constraint is a software generation and it will age out. If it does not, it is architectural and it belongs in the purchase decision permanently." }, { "q": "Is one box enough?", "a": "Not by filling it. The obvious move is to run the largest model that fits in 128GB, and that was tested: Nemotron-3 Super 120B-A12B, on corrected serving config, on a box to itself. It took 74 minutes for a repair that Qwen3-Coder-Next, 80B total with about 3B active, completed in 7 on identical hardware, and lost the next task to a two-hour ceiling. The arm was stopped on economics rather than on capability, because a model an order of magnitude slower per verified repair cannot pay for itself whatever it eventually scores. One box favours a smaller, faster model. The bigger model needs the second box for reasons that have nothing to do with whether its weights fit." } ], "not_proved": [ "Replicated, and worth stating precisely. The pool was run twice on the pinned configuration and returned 22 of 22 both times with zero tasks changing hands. The pool does move: Qwen3-Coder-Next scored 18 of 22 on both of its runs with six different tasks flipping, three each way. The variance is real and this model did not show it here. Two runs is still two runs, and neither says anything about a third model, a different pool, or a rebuilt image.", "The build is isolated as the variable, but not explained. The 31B model that failed nine times on the stock image served 16 of 16 concurrent on the working image with the same hardware, parallelism, fabric settings and launch shape, so the runtime is what changed the outcome. Why it changed is not established here. No claim is made about which commit, patch, or flag is responsible.", "Wall clock is not comparable to the other arms in this corpus. The harness ran at concurrency two because the box hosting it also holds half the model, and earlier arms ran at four and six. Solve counts are comparable. Timings are not.", "Nothing here is a statement about clustering generally. This is two small unified-memory boxes over a single RoCE link. It says nothing about clusters built from cards a vendor validates for multi-node work.", "The working path depends on a third-party container image. Its base layer is pinned by digest and was verified byte-for-byte against that digest before the build proceeded. The derived layer, which applies a tokenizer patch, produced a different image ID here than the reference build records, and the build verified it by content and tokenizer-conformance tests instead. That is ordinary Docker behaviour rather than a defect, and it is not evidence about whether the maintainers can reproduce their own build. No claim is made about the project’s provenance or its maintenance.", "No claim that the vendor playbook is broken for every model or every version. What was measured is that its prescribed path deadlocked here, on this hardware, at this version, and that nine plausible fixes did not recover it.", "The dense-clusters, mixture-of-experts-does-not pattern rests on four models actually put across the link: two dense, both of which clustered, and two mixture-of-experts, of which one did. Four is a small sample supporting a mechanism, not a proven law. The community record shows the same split and a third mixture-of-experts failure, but that is somebody else’s measurement and is not counted here. A single counterexample would overturn the pattern, and only one runtime build was tried against the failures.", "Correction, made after publication. This lab originally described the served model as a 671B mixture-of-experts checkpoint at NVFP4. Both figures were wrong. The weights are FP8, e4m3 with ue8m0 block scales in 128 by 128 blocks, read from the checkpoint config; the NVFP4 in the serving stack is the key-value cache format, and the two were conflated. The 671B was never sourced: no parameter count appears in the config, the checkpoint, or the upstream repository, and 671B is DeepSeek V3 and R1’s figure, carried across by name. The checkpoint is 156 GiB, which is what the two-box requirement actually rests on, and no parameter count is claimed here in its place.", "The 235B mixture-of-experts was not disqualified on capability or on capacity. It fits, it carries the context the harness needs, and it never ran a task. It failed engine initialisation on the one image that does multi-node here, which is a statement about that image rather than about the model.", "Every wall-clock figure in this entry is venue-dependent in a way the scores are not. Re-running Qwen3-Coder-Next on a recorded serving config reproduced its 18 of 22 exactly and cut the arm from 11.4 hours to 4.8, while doing more turns, with 21 of 22 tasks cheaper per turn. The original venue was an ad hoc launch whose logs were destroyed, so the difference cannot be attributed. Read the hour counts here as one measurement of one configuration, not as a property of the model.", "That wall-clock instability has a control, and it points at the record-keeping rather than the hardware. DeepSeek ran from a pinned kit with a digest-verified image and a written config, and it reproduced its score and its clock together: 4.49 and 4.19 hours, 869 and 858 turns. Qwen ran from an ad hoc command nobody saved, and it reproduced its score while missing its clock by 2.4x. Same two boxes, same harness, same pool. The difference between the two is whether the venue was written down." ], "executive_summary": [ "Two DGX Sparks were clustered over a 200 Gb/sec RoCE link and measured serving DeepSeek V4 Flash 0731, a 156 GiB mixture-of-experts checkpoint with FP8 weights and an NVFP4 key-value cache, tensor-parallel across both nodes. The pair sustained 16 concurrent requests with zero errors at 215 tok/s aggregate, and cleared 22 of 22 tasks on the certified repair pool with no timeouts and no session errors.", "The same hardware, running the vendor-documented configuration on a stock vLLM build, deadlocked at four concurrent requests. Nine configuration hypotheses were tested against an identical load and every one failed at the same point: a shared-memory read timeout inside the inter-worker synchronisation path.", "The difference between the two outcomes was the runtime build, not the hardware, the fabric, or the orchestration framework. Both documented multi-node paths failed identically on the stock image. Holding the model fixed and changing only the image settles it: the same 31B model that failed nine times served 16 concurrent requests with zero errors at 169.7 tok/s on the working build.", "The failure mechanism survives in both builds. The working image logs the same shared-memory starvation, a 60 second wait for a broadcast block, as an informational line and continues. The stock image raises a timeout from the same call and kills the engine. One build tolerates the condition; the other does not.", "Interconnect capability was never the constraint. A NCCL all-reduce at this model’s activation size measures 49.6 microseconds, which puts collective overhead at roughly 8 percent of the token budget at tensor-parallel width two.", "Five models were qualified against the pair and one served. DeepSeek V4 Flash 0731 is the one that works. The others failed for four different reasons: Qwen3-Coder 480B-A35B needs 273 GB at four-bit, more than the pair holds before a single token of cache; Qwen3-235B-A22B-2507 fits easily and serves on one node but fails engine initialisation at tensor-parallel width two; Devstral 2 123B fits and clusters and then decodes six times slower than the model that works; and Nemotron-3 Super 120B runs on a single box at ten times the wall clock per repair. Fitting in the combined memory is a necessary condition and not close to a sufficient one.", "The pattern across those attempts is that sparsity is what admits a large model to this tier and expert routing is what excludes it. Both dense models clustered here, Gemma 4 31B and Devstral 2 123B, did so without incident. Of the two mixture-of-experts models put across the link, only DeepSeek V4 Flash clustered, and it runs on a container image built specifically for its architecture. Tensor parallelism needs a per-layer all-reduce, which every runtime implements well. Mixture-of-experts additionally needs all-to-all expert dispatch through fused kernels bound tightly to hardware generation and library version, and dense models never enter that path." ], "dapm": [ { "instrument": "4plus1", "layer": "layer0", "placement": "Retained", "note": "Compute and fabric are owned outright. The finding is that owning them is not sufficient, because the software path to using them together is not vendor-supported in practice." }, { "instrument": "4plus1", "layer": "layer2c", "placement": "Retained", "note": "Reasoning runs on owned hardware with no metered call. The dependency that remains is the container image, which is community-built." } ], "seam_map": [ { "function": "Serve a model larger than one box", "p": "ready" }, { "function": "Concurrent agentic serving", "p": "ready" }, { "function": "Follow the vendor multi-node playbook", "p": "needs-cloud" } ], "writeup": [ "The pitch for a second small box is straightforward. A model that will not fit in 128GB fits in 256GB, the interconnect is fast enough, and two boxes cost less than one larger machine. Lab 013 closed by naming exactly this gap, and the second Spark was bought to close it.", "The vendor publishes a playbook for this. Its multi-node section prescribes a Ray cluster with tensor parallelism across both nodes, and that is where this lab started. On a stock vLLM build serving Gemma 4 31B, dense, that configuration answered a single request at 11.2 tok/s and then deadlocked the moment four requests arrived together. One node sat at 96 percent reported utilisation drawing 20 watts, which is a busy-wait spin rather than compute, while the other sat at zero.", "Nine configuration hypotheses were tested against that identical load. The engine watchdog was raised from 300 to 1800 seconds. Step size was capped so no forward pass could approach the timeout. The RoCE GID index, which was unset and free to select a link-local IPv6 entry, was pinned to the IPv4 entry the fabric actually exposes. CUDA graphs were made breakable, then removed entirely. The RDMA memory-lock limit was lifted. Three NCCL settings that the working kit uses were adopted. The orchestration was moved off Ray onto vLLM’s own native multi-node path, which the playbook never mentions. Tensor parallelism was swapped for pipeline parallelism.", "Every one of those failed at the same place. The traceback names a shared-memory broadcast read that times out, and POSIX shared memory does not cross a machine boundary. That is not a setting. Ray turned out to be a passenger rather than the cause, because vLLM’s native path routes through the same primitive and fails identically.", "What changed the outcome was the build. A community-maintained container image, pinned by digest, running a development build of the serving engine one point release ahead of the stock one, served the same tensor-parallel width across the same two boxes without incident. Sixteen concurrent requests, zero errors, 215 tokens per second aggregate, and the engine still healthy afterwards.", "On that image the pair was given DeepSeek V4 Flash 0731, a mixture-of-experts checkpoint of 156 GiB with FP8 weights, which cannot fit either box alone. This is the capacity case the second box exists for, and it is the case Lab 013 could not reach. The model served at 35 tok/s single-stream, which is five times what the same hardware manages with a 31B dense model on one box.", "Then it ran the instrument. Twenty-two certified repair tasks, the same pool six earlier local models have run, driven through a headless agentic harness with a deterministic gate deciding every verdict. It cleared all twenty-two. No timeouts, no session errors, three API retries across four and a half hours.", "The tasks it took are the interesting part. One repair in that pool had never been solved by any self-hosted model across six arms, and the clustered pair took it in 34 minutes over 57 turns. Another took 208 turns without losing its place in the instrument. That is what instrument literacy looks like when it holds under load, and it is the capability Lab 013 identified as the thing that separates models rather than parameter count.", "The comparison that matters commercially is against renting. A hosted frontier model cleared the identical pool in 38 minutes for $3.14. The clustered pair took 4.5 hours and nothing. Seven times the wall clock, zero marginal cost, on capital already spent. Whether that trades well depends on how often the work runs and whether it can wait.", "The comparison that matters architecturally is against the single box. The best self-hosted result on this pool was 18 of 22, from a model that fits in 128GB with no interconnect involved. Had the clustered pair matched that, the second box would have justified nothing. It cleared four more, including the one nothing local had ever solved.", "So the hardware was never the limit. Two boxes, an in-spec cable, and a correctly configured fabric were sufficient from the first hour, and the fabric measurements said so: collective overhead at tensor-parallel width two is about 8 percent of the token budget. What stood between that hardware and a working endpoint was a documented procedure that does not work and a working configuration that is not documented.", "The obvious next question is which other models this tier unlocks, and the answer is narrower than the memory arithmetic suggests. Five candidates were qualified. Qwen3-Coder 480B-A35B, the largest coder model in the same family as the one-box winner, needs 273 GB at four-bit, which exceeds both boxes before any cache. A 123B dense model fits, clusters, and decodes at 5.8 tok/s, six times slower than the sparse model that works, because decode rate tracks active parameters rather than total ones. Nemotron-3 Super 120B-A12B fits a single box and spent 74 minutes on a repair Qwen3-Coder-Next, 80B with about 3B active, finished in 7. And Qwen3-235B-A22B-2507, which fits comfortably and carries the context length the harness needs, fails engine initialisation at tensor-parallel width two with a kernel signature mismatch.", "That last failure is the one worth generalising. Both dense models put across the link, Gemma 4 31B and Devstral 2 123B, cluster without incident. Of the two mixture-of-experts models tried, only DeepSeek V4 Flash clusters, and only on a container image built for its architecture; Qwen3-235B-A22B-2507 fails engine initialisation at width two. The mechanism is that tensor parallelism asks for a per-layer all-reduce, a collective every runtime handles, while mixture-of-experts additionally needs all-to-all expert dispatch through fused kernels pinned to a hardware generation and a library version. Dense models never touch that code path. So the sparsity that lets DeepSeek V4 Flash fit inside 256 GB is the same property that makes it hard to split across two machines, and the vendor and community record agrees: multiple reports show Qwen mixture-of-experts models serving at width one and failing at width two, with no working two-Spark demonstration published.", "Which turns the purchasing statement into something a spec sheet cannot express. The second box does not buy access to models above 128 GB. It buys access to models above 128 GB that are also either dense, or accompanied by a runtime built for them. Today that is a set of one." ], "abstraction": [], "method": [ "Self-funded. Both DGX Sparks, the QSFP112 cable, and the power were bought by the practice. No vendor was briefed, no vendor reviewed this, and no vendor was given advance sight.", "The repair pool, the reproduce-or-drop certification, and the deterministic gate are the same instrument used in Labs 011 through 013, unchanged, so the solve counts sit on the same axis as the earlier arms.", "Serving configuration is recorded verbatim in every result file for this lab, including the image, the parallelism width, the sequence cap, and the fabric. Earlier arms in this corpus did not record it, which is a gap this lab does not repeat.", "The harness, the gate, and the certification tooling are public. The assessment methodology, the vendor grades this evidence feeds, and the judgment that turns a score into a placement are not." ], "assessments": [], "publisher": "The Advisor Bench LLC", "program": "Layer2C Labs", "url": "https://labs.layer2c.com/labs/second-box", "markdown_url": "https://labs.layer2c.com/labs/second-box.md", "json_url": "https://labs.layer2c.com/labs/second-box.json" }