# Lab 020 raw lab detail: For a lab, the software lag is the cost.

One Apple Mac Studio, M5 Ultra (36-core CPU, 80-core GPU), 256 GB of unified memory, 4 TB, macOS
27.0.1, bought by the practice. Compared with the two NVIDIA DGX Spark venue from Lab 014 (GB10,
128 GB each, one 200 Gb/sec RoCE link). Everything on the Studio was measured on October 17 and 18,
2026. The Spark figures come from Lab 014's runs and the qualification record of its serving kit.
Nothing was rented. Metered spend: $0.

## The one question

Can one 256 GB Mac Studio replace two clustered DGX Sparks for local agentic work on a model that
does not fit in one Spark? The test model is DeepSeek V4 Flash 0731, a 156 GiB mixture-of-experts
checkpoint with FP8 weights, the model the second Spark was bought for.

## The instrument

The same instrument as Labs 011 to 014, unchanged. Twenty-two Python repair tasks reconstructed
from maintainer-merged pull requests, each with held-out tests the model never sees. A
deterministic gate applies the model's edit, runs the tests in a network-isolated container, and
rules. The harness is a headless Claude Code session (v1 harness, Claude Code 2.1.218) with file
read, edit and shell, reaching the model through a litellm proxy (1.93.0, `openai/` provider).
Two sessions at a time, 7,200-second ceiling, hardest task first.

The harness ran on a DGX Spark, not the Studio, because bench code is not run on a machine on the
author's personal network. The Spark reached the Studio's server through an SSH relay.

## The measured spine

| What | Studio | Spark pair | Note |
| --- | --- | --- | --- |
| Certified pool | 22 of 22 | 22 of 22, twice | Studio: one run, zero timeouts, zero session errors |
| Seconds per agentic turn, two sessions | 35.5 | 17.6 to 18.6 | Summed task time / turns |
| Effective output per stream | 17.0 tok/s | 31.1 to 33.6 tok/s | Output tokens / summed task time |
| Output per turn | 602 | 578 / 591 | Nearly identical |
| Single-stream decode, speculation off | 40.6 tok/s | 27.1 tok/s | Different prompts and settings, directional |
| Single-stream decode, production config | 40.6 tok/s | 52.0 tok/s | Pair drafts three tokens; settings differ (temperature, output length), directional |
| Aggregate output | 58.6 @2, 84.2 @4 | 132.2 @6, 211.4 @16 | No matched point above one stream |
| Cold prefill | 1,263 @22K, 1,354 @66K | 1,929 @131K | Unique prompts, cache could not serve them |
| Prefix cache share of prompt tokens | 90.9% | about 98% | Studio counter includes preflight requests |
| Resident memory | 149.8 GB | split across two nodes | 157.7 GB with the speculative head loaded |
| Price, US list, October 2026 | $12,299 | $13,900 | Pair before the interconnect cable |
| Elapsed, clean runs, two sessions | 2 h 27 min | 2 h 10 min (run 2) | Not a venue comparison: the pair took 77% more turns (reasoning section) |
| Tool errors | 10 | 26 (run 2) | |

## Per task

Hardest first. Minutes and turns. Neither turns nor minutes compare cleanly across venues,
because reasoning reached the agent on the Studio and not on the pair (reasoning section below).
The table is the raw record.

| Task | Studio min / turns | Pair r1 min / turns | Pair r2 min / turns |
| --- | --- | --- | --- |
| prettytable-pr299 | 28.2 / 25 | 34.4 / 57 | 31.3 / 74 |
| prettytable-pr332 | 31.8 / 28 | 15.8 / 20 | 20.3 / 20 |
| prettytable-pr289 | 29.4 / 29 | 15.0 / 50 | 34.5 / 153 |
| prettytable-pr296 | 14.8 / 28 | 37.0 / 208 | 17.7 / 48 |
| task-r029 | 9.5 / 17 | 12.7 / 24 | 10.4 / 24 |
| click-pr3678 | 16.4 / 30 | 9.5 / 36 | 6.5 / 21 |
| click-pr3482 | 26.9 / 48 | 48.6 / 97 | 38.7 / 97 |
| dateutil-pr700 | 6.3 / 11 | 16.4 / 28 | 12.0 / 28 |
| more-itertools-pr1117 | 7.4 / 15 | 7.4 / 17 | 3.4 / 13 |
| requests-pr7502 | 15.6 / 43 | 5.2 / 28 | 5.7 / 35 |
| prettytable-pr344 | 7.6 / 26 | 21.6 / 132 | 10.7 / 35 |
| prettytable-pr345 | 9.7 / 17 | 6.5 / 17 | 4.4 / 28 |
| requests-pr7308 | 12.0 / 25 | 4.2 / 21 | 3.0 / 13 |
| requests-pr7433 | 11.9 / 28 | 7.5 / 34 | 11.3 / 35 |
| more-itertools-pr944 | 6.6 / 10 | 6.5 / 14 | 5.2 / 12 |
| click-pr3126 | 11.9 / 19 | 3.2 / 10 | 13.8 / 128 |
| task-r007 | 7.3 / 11 | 2.4 / 7 | 3.8 / 10 |
| more-itertools-pr1119 | 11.3 / 11 | 4.6 / 10 | 10.7 / 37 |
| click-pr3504 | 10.0 / 17 | 3.9 / 20 | 3.9 / 16 |
| requests-pr7309 | 4.2 / 13 | 3.1 / 13 | 0.9 / 7 |
| prettytable-pr445 | 4.7 / 19 | 2.2 / 15 | 1.7 / 14 |
| requests-pr7205 | 2.9 / 15 | 1.8 / 11 | 1.7 / 10 |

Every task solved on all three runs. Where the turn counts are close, the pair is usually two to
three times faster per task, consistent with the per-turn finding. On the long tasks where the
pair took many more turns, the Studio sometimes finishes first.

Pair run 1 was resumed after an interruption, skipping five tasks already done, as Lab 014
records. Its per-task times are valid, its elapsed time is not.

## The lab verdict is a requirements judgment

There is no authoritative measurement of what makes a better lab platform. The verdict here,
that the Spark pair is the better lab venue for this author, weighs the measured results against
one set of requirements:

- **Testing models as they arrive.** A lab exists to test new models and their serving features
  when they ship. On this model the Spark stack had working speculative decoding at qualification;
  the Mac stack still did not eleven weeks later. This author weighs that most heavily.
- **Wall time per turn under concurrent load.** Lab runs are batch work at two or more sessions,
  where the Studio took about twice as long per turn.
- **Existing investment.** The author already owns two Sparks, so the Studio would be a second
  venue, not a replacement.

Another architect can read the same results and decide differently. Factors that would favor the
Studio, all measured or bounded here: one system instead of two, a lower list price, a desk
machine in the same box, faster single-stream decode with speculation off, faster bring-up, and
work that runs one or two sessions at a time. The measurements are the instrument's. Which of them
decides is the reader's call.

## Why the speed gap is about 2x, and what it is made of

With speculative decoding off on both, the Studio decodes one stream about 1.5x faster. With the
pair's production speculation on, the pair leads one stream by 1.28x. The rest of the gap opens
under load. At two sessions on short prompts the Studio fell from 40.6 to 29.4 tok/s per stream.
Over the pool, where average contexts ran 35K to 49K tokens, its server averaged 20.0 tok/s. From
the cache counters, the Studio re-prefilled an estimated 3,150 uncached tokens per turn against
about 820 on the pair, worth about 2 seconds per turn.

So the doubling is speculation plus the Studio losing more speed to concurrency and long context.
(An earlier draft cited the single gatetest task, run alone, as faster per turn on the Studio. Its
turns carried less than half the output tokens of the pair's, and in output per second the pair
was still faster, 38.4 to 38.8 tok/s against 29.6. Withdrawn.)

A dense 31B model measured separately on the same Studio barely slowed at two sessions, 44.9 to
43.1 tok/s per stream on 1,024-token runs. That does not separate architecture from size and
precision (18.4 GB 4-bit against 149.8 GB FP8), and that model's own pool run has not completed. How much each contributes, and whether
the Studio's concurrency loss is its GPU or oMLX's batching, was not separated.

## Serving envelope, diffed against the Spark arm

Diffed against the run-time configuration file on the head Spark (`~/ds4-spark/.env`, dated August
6, 2026), not the published record, which does not state speculation or sampling.

| Setting | Spark arm | Studio | Status |
| --- | --- | --- | --- |
| Checkpoint | 0731 rev 9e165c30, FP8 | same files, SHA-verified | match |
| Serving engine | vLLM 0.25.2.dev0, community image pinned by digest | oMLX v0.7.1rc1 (7a5a3d7), MLX 0.32.4 | diverges |
| Speculative decoding | three-token, probabilistic | off | diverges, deliberate |
| Thinking / effort | on, maximum | on, maximum (default; client can override) | match, verified by prompt length |
| Sampling | temperature 1.0, top_p 0.95 | temperature 1.0, top_p 0.95 | match |
| Context | 1,048,576 | 1,048,576 | match |
| Prefix caching | on | on (memory plus SSD tier) | match in kind |
| Key-value cache | NVFP4 | MLX native | diverges |

Speculation stayed off because oMLX's implementation on this model has an open bug
(jundot/omlx#4344) in which a failed draft handoff strands about 63 GB of GPU memory during long
tool-calling turns. The Studio raised the matching error ("MTP handoff failed... rms_norm") on its
first request.

Stock MLX (mlx-lm 0.32.2) cannot load this architecture. Support is an open upstream pull request
(ml-explore/mlx-lm#1797). oMLX injects an earlier version of that support (mlx-lm PR 1192) as a
patch. Its loader maps the FP8 weights and their block scales onto MLX's native mxfp8 format at
load time. Whether that mapping is bit-exact was not verified.

## Gates before the pool

| Gate | Result |
| --- | --- |
| Long generation | 24,000 tokens in one request, no Metal resource error, 35.9 tok/s. A known MLX buffer leak (ml-explore/mlx-lm#1662) killed this model at about 11,500 |
| Tool call | Parsed `read_file` call, correct answer from the tool result on the second turn |
| Harness preflight | Envelope check, file-write smoke through the real CLI, large-prefill probe, provenance stamp |
| Decode, 1 / 2 / 4 streams | 40.6 / 58.6 / 84.2 tok/s aggregate |

## What the agent saw: the reasoning difference

| | Spark arm (run 2) | Studio |
| --- | --- | --- |
| Turns | 858 | 485 |
| Tool calls | 836 (shell 681, read 98, edit 57) | 463 (shell 378, read 45, edit 40) |
| Tool calls per turn | 0.97 | 0.95 |
| Messages with more than one tool call | 0 | 0 |
| Reasoning blocks | 321 | 286 |
| Characters inside them | 0 | 756,398 |
| Tool errors | 26 | 10 |

Every reasoning block that reached the harness on the Spark arm was empty. On the Studio the
reasoning came through. Same harness, same proxy version and configuration. The proxy called
oMLX's Responses API; which endpoint it called on vLLM was not confirmed. The Studio sessions took
45 percent fewer actions at the same one tool call per turn. Whether visible reasoning caused that
was not tested. The test is one venue with reasoning stripped from what the agent receives.

Consequence for this lab: turn counts and elapsed time compare two apparatus states, not two
venues. Per-turn figures are used instead, with the caveat that the Spark's extra turns were small,
high-cache-hit steps.

Consequence for the corpus: every earlier arm that ran a reasoning model through this proxy to
vLLM may have had the same silent loss. Cross-arm turn counts need an audit before they are read.

## Setup effort

| Local time, Oct 17 | Step |
| --- | --- |
| 12:24 | Studio boots after migrating from the author's main Mac |
| 12:44 | First authenticated remote login (remote login enabled, one SSH key installed) |
| 12:46 | Stock mlx-lm installed: cannot load the model |
| 12:48 to 13:12 | A 154 GiB 4-bit conversion downloaded. Not needed: the FP8 checkpoint was already on disk |
| 13:12 | oMLX installed from source |
| 13:14 | FP8 checkpoint loaded in 18.8 s at 157.7 GB with the speculative head, first token |
| 13:21 to 13:24 | Xcode installed, Metal Toolchain downloaded, custom kernels built in 26 s |
| 13:30 | First valid server (two earlier restarts had silently failed) |
| 13:44 | Gates pass |
| 13:53 | Envelope matched to the Spark arm |
| 14:03 | Harness preflight passes, first pool task starts: 1 h 19 min after first login |

Lab 014 took about a day to reach a working endpoint after the vendor multi-node playbook
deadlocked, plus a separate interconnect project. The Studio clock starts after the checkpoint had
migrated onto it, reuses the Spark's harness and proxy, and benefits from knowing the model.

## Defaults that were wrong without an error

1. Speculative decoding on by default, with the open memory bug above.
2. Reasoning effort defaulting to low. Visible only as prompt length: 9 tokens at low, 101 at
   maximum.
3. The server process renames itself (`omlx-server`), so stopping it by its launch command matches
   nothing. A restart then fails to bind while the old process keeps serving. Two measurement
   rounds were silently voided this way and retracted. Stop it by process ID.
4. Custom Metal kernels need full Xcode and its Metal Toolchain component, not just the command
   line tools. Without them the server warns that long-context prefill runs several times slower.
   Not cleanly measured: a stale pre-kernel process read 1,198 tok/s at 15K, and with kernels the
   Studio read 1,263 at 22K. Decode was unchanged.
5. The loader can rewrite safetensors headers in place on some checkpoints of this model. It did
   not touch these files: their modification times are unchanged since the August checksum
   verification.

## Price, US list, October 2026

| Item | Price | Source |
| --- | --- | --- |
| Mac Studio M5 Ultra, 80-core GPU, 256 GB, 4 TB (as tested) | $12,299 | Third-party configurator captures and a published review unit |
| Same, 1 TB | $10,799 | Third-party configurator capture, August 26, 2026 |
| DGX Spark, 128 GB, each | $6,950 | Confirmed by the author, October 2026. Earlier list: $3,999 at launch, $4,699 from February 2026 |
| DGX Spark, 64 GB | $4,999 | Confirmed by the author. Two hold 128 GB, too little for this model |

Two Sparks carry 8 TB of storage between them against the Studio's 4 TB, and need an interconnect
cable that is not priced here. A Studio buyer still needs a separate host or VM for agent code.

## Operational record, mistakes included

- The FP8 checkpoint was already on the Studio's disk from the migration. The first inventory
  checked only the model cache and missed it, so an unneeded 154 GiB download ran.
- The restart trap above voided two rounds of decode numbers. They were published to the author as
  57 to 63 tok/s single-stream and "no batching gain at two streams", then retracted.
- Speculation was first turned off "for parity" on the belief that the Spark arm did not
  speculate. It did. The decision to keep it off was then made deliberately, on the open bug.
- The harness preflight's large-prefill probe passed in 1 second on the pool launch because the
  prefix cache served it. On a caching venue that probe proves nothing about cold prefill. It
  needs a unique prompt each run.
- The bench tally printed "22 / 23" because it counts the preflight stamp file as a result row.
  Solve counts here are read from the per-task files.
- During the pool, another lab's harness sessions ran on the same Spark, and a file copy wrote to
  the Studio's disk (capped at 60 MB/s, about four minutes in two intervals). No other model ran on
  the Studio. The GPU wired-memory limit
  stayed at Apple's default.

## Public runbooks consulted

- Elite Test Engineering, "DeepSeek V4 Flash: DGX Cluster vs. M5 Ultra": a 4-bit MXFP4 conversion
  of the same model on a 256 GB M5 Ultra at 44.8 tok/s at about 1K context, 38.7 at 32K, with
  speculation off and two sequences.
- Elite Test Engineering, "DeepSeek-V4-Flash on a Mac Studio: What Worked, What Didn't".
- A two-Mac-Studio campaign on the same model (GitHub gist, aidiffuser): MLX pitfalls including
  decode decay from compile-cache growth and a long-context memory blow-up.
- DeepSeek's model card: temperature 1.0 and top_p 0.95 for agent tasks.

## Spend

$0 metered. The Studio and both Sparks were bought by the practice. Power only.

## What this hands the next labs

- Rerun the pool on the Studio when oMLX's speculative decoding works on this model. That is the
  test of whether one box can match the pair on time.
- Audit every earlier arm for empty reasoning blocks before comparing turn counts across arms.
- Hold the Studio fixed and change the serving software to separate GPU limits from batching
  limits.
- Measure the Studio as a desk machine while it serves: a four-way batch or a long cold prefill
  during ordinary work. Lab 021 observed ordinary desk use during lighter pools (15 GB and 105 GB
  models) without instrumenting it. A footprint like this lab's 166 GB wired is untested.