# Pick the model first. It picks the training hardware, and it can rule out the managed service.

> Lab 027 · Editorial lab · Status: published · Author: Keith Townsend · October 21, 2026  
> Published by The Advisor Bench LLC · Layer2C Labs  
> Canonical: https://labs.layer2c.com/labs/finetune-model-first · Structured: https://labs.layer2c.com/labs/finetune-model-first.json

**Finding:** Lab 18's dense Gemma 4 31B coding-agent tune, rerun on owned hardware: a Mac Studio (M5 Ultra) trained it in 2.7 hours against 10.2 on a DGX Spark, and the adapters from both scored 11 to 14 of 22 on Lab 18's own task bar on either box, inside Lab 18's noise. The Studio needed rewritten attention to train it at all. CoreWeave Forge's self-serve list doesn't include the model.

**Question:** In Lab 26 I fine-tuned Gemma 4 26B, a mixture-of-experts (MoE) model, on an M5 Ultra Mac Studio and an NVIDIA DGX Spark. The Studio was six times faster. In Lab 24, a similar job on CoreWeave Forge took 31 minutes end to end. So do those results carry through to a dense 31B model? That's the least interesting part of the question. The answer drives an architectural buying decision: you have to know what model you need to train before you evaluate training performance. Here the model was Lab 18's coding-agent tune of the dense Gemma 4 31B. Forge's self-serve menu doesn't list it. The Spark trained it only on an older, pinned software release. The Studio trained it 3.8 times faster, after I rewrote part of its software. Both adapters landed in the same range on Lab 18's coding tasks, 11 to 14 of 22.

**Load:** One base model, Gemma 4 31B, dense, bf16. Lab 18's loop-worker corpus: 171 training examples (9 over its 32K-token cap dropped), mean 8,663 tokens, longest 30,553, 1,481,399 tokens, loss on every assistant turn; 21 held-out validation examples from 6 tasks; Lab 18's 22-task coding loop as the outcome bar. LoRA rank 32, alpha 64, dropout 0.05 on attention q, k, v and o, 2 epochs, 86 steps of 4, learning rate 1e-4 cosine. One DGX Spark and one Mac Studio (M5 Ultra, 256 GB), owned, October 2026.

**Raw lab detail:** https://labs.layer2c.com/downloads/finetune-model-first-lab-detail.md
**Vendors:** Apple (hw), NVIDIA (hw), Google (model), CoreWeave (ref), Anthropic (bench)
**Layers:** Layer 0A (Compute), Layer 2C (Reasoning)
**Themes:** model-ownership, managed-abstraction-cost, operating-cost-of-ownership

## Verdict

**Scope:** One dense model (Gemma 4 31B, bf16), one long-sequence LoRA job and Lab 18's 22-task coding loop as the outcome bar, one DGX Spark and one Mac Studio (M5 Ultra, 256 GB), one training run and one loop run per arm. The venues are compared as services at the contract layer: whatever runtime each needed to take the job is part of the venue, and what it took is its cost. The Spark ran Lab 18's own PyTorch stack and vLLM; the Studio ran mlx-lm with attention I rewrote to fit, and mlx-lm's server with a cache cap. Power is each GPU's rail as its system reports it. CoreWeave Forge was checked against its published self-serve list, not run. A faster Spark software stack is the open question.

- **DO:** Pick the model your use case needs, then check which venues can train it at all. CoreWeave Forge's published self-serve list for serverless fine-tuning (read October 21, 2026) has five base models, with the LoRA rank fixed per model. The dense Gemma 4 31B isn't on it. I didn't ask CoreWeave's support whether it can be enabled; on the self-serve path, the managed route ended before speed or effort came up.
- **DO:** Prove the job fits at its longest example, on the exact stack you'll run, before you compare speed. The Spark trained the 30,553-token example only on Lab 18's pinned transformers release, with as little as 5.7 GB of its 121 GB free; the current release took 2.0 times a 26B model's weights in memory while loading, against 1.74 times for the pinned one. On the Studio neither framework's stock training path fit: mlx-lm ran out of memory at a 224 GB footprint, and PyTorch's Apple GPU backend stopped at its indexing limit, about 11,600 tokens.
- **DO:** Count the Mac Studio as a training and serving venue for this model, and budget the engineering to get it there. With rewritten attention it trained the job in 2.69 hours against the Spark's 10.25. On Lab 18's 22-task bar, the adapters from both boxes scored 11 to 14 on either box, inside Lab 18's own noise (its tune 14, its base 12). Serving the loop, the Studio took 1 hour 46 minutes to about 2.5 hours per adapter against the Spark's 5 hours 17 to 5 hours 41.
- **DON'T:** Don't treat the OpenAI-compatible API as the portability layer. The loop spoke the same API to both servers. vLLM ran Lab 18's recorded settings unchanged; mlx-lm's server ran out of memory with 51 GB of old conversations in its prompt cache and needed a 32 GB cap. The runtime's operating settings move with the workload: self-serve them and they're yours, take a managed endpoint and they're the provider's.
- **DON'T:** Don't buy the Spark expecting its compute to carry this job. In a matrix-multiply benchmark the Studio's GPU ran about 1.2 times the Spark's at bf16, with 4.6 times the memory bandwidth. The 3.8x training gap sits between those, nearer bandwidth. Whether a faster Spark software stack closes it is untested.

**Independence and disclosure:** Self-funded editorial. The DGX Sparks and the Mac Studio were bought by the practice. Disclosure: Google Cloud is a client of this practice, and the model under test, Gemma 4 31B, is a Google model. Google did not commission, fund, preview or have any say in this lab. Apple and NVIDIA hardware are both under test; CoreWeave's Forge appears through its published model list and Lab 24's record. Neither Apple, NVIDIA, CoreWeave nor Anthropic is a client. No vendor paid for this answer.

## Measured

| Measurement | Result | Note |
| --- | --- | --- |
| Training time, 86 steps | 9,670 s against 36,885 s | Studio (mlx-lm, blocked attention) against Spark (Lab 18's recipe and software). Both are the trainer's own time. Lab 18's original run took about 10 hours by its checkpoint timestamps. |
| Training throughput | 306 against 80 tokens/s | 2,962,798 tokens over 2 epochs. Median step 109.6 s against 377.5 s; the per-step ratio ran from 2.5 to 5.4 with the step’s lengths. |
| Lab 18's 22-task loop, solved | 11 to 14 of 22, every arm | Spark-trained adapter: 11 served on the Spark (vLLM), 14 on the Studio (mlx-lm). Studio-trained adapter: 13 on the Spark, 12 on the Studio. Lab 18's tune: 14; its base: 12, a gap Lab 18 called noise. One sampled run per arm; the same adapter moved by three tasks between venues. |
| Loop time, 22 tasks | 1 h 46 min to 2.5 h against 5 h 17 min to 5 h 41 min | Studio (mlx-lm) against Spark (vLLM), two tasks in flight, as in Lab 18. The Studio’s 2.5 hours is the valid loop time for one adapter after two apparatus faults were fixed and the arm resumed. |
| Held-out validation loss | 0.151 against 0.154 (Studio runtime); 0.151 against 0.151 (Spark) | Studio-trained against Spark-trained adapter, 21 examples, 11,458 labeled tokens, Lab 18's loss. Base model 1.048 and 1.068. Lab 18's original adapter 0.153 on both. |
| Next-token agreement with the Spark-trained adapter | 98.3% | Studio-trained adapter on held-out labels, in either runtime. The base model agrees 93.6% of the time, and Lab 18’s original adapter 98.4%, so the two new adapters differ about as much as two Spark-trained ones. |
| Step 1, same examples and starting adapter | 1.0182 against 1.0080128908157349 | Studio against Spark. The Spark's value is Lab 18's logged step-1 loss to every digit. |
| Memory, training | 144.0 GB against 102.4 GB allocator peak | MLX peak against PyTorch's. Studio process footprint 205 GB at the 30,553-token step, 231 GB over the run, of 256; Spark lowest free memory 5.7 GB of 121 over the run. |
| Stock training paths at 30,553 tokens | out of memory (Studio) · fit (Spark) | mlx-lm's attention: Metal out-of-memory at a 224 GB footprint. PyTorch on Apple's GPU (MPS): stopped at its 32-bit indexing limit, about 11,600 tokens. Lab 18's PyTorch stack on the Spark: fit. In MLX 0.32.4 the attention backward is linear in memory with a causal mask and quadratic with Gemma's sliding-window masks (12.9 GB at 8,192 tokens, 50.3 GB at 16,384). |
| Model load on the Spark | 6 GB free at the low point | transformers 5.15.1 (Lab 18’s release). On a 26B model, 5.20.2 took free memory from 113.5 to 17.3 GB (2.0 times the weights) against 30.0 GB for 5.15.1 (1.74 times); I stopped the 31B load under 5.20.2 at 39% with 20 GB free. |
| bf16 matrix multiply, job shapes | about 95 to 122 against 77 to 98 TFLOPS | Studio against Spark, forward, median of three, in each box’s training stack. Typically about 1.2x. One outlier: the Studio’s transposed MLP-down product ran 66.7 against the Spark’s 82.7. Steady over a 10-minute sustained run on both. |
| Memory bandwidth | 1,083 against 233 GB/s | c = a + b over a 4 GiB bf16 buffer. 4.6x. |
| Rough share of matmul ceiling while training | about 47% against 15% | A 6 x parameters x tokens estimate (30.7B, checkpoint recompute included, attention excluded) divided by training time, against the benchmark above. A heuristic, not measured utilization. |
| GPU-rail power and energy, training | 135 W, 367 Wh against 66 W, 656 Wh | Studio IOReport GPU channel against Spark nvidia-smi power.draw, whole run. Neither is wall-plug; not comparable system energy. |
| Forge | not on the self-serve list | CoreWeave's published list for serverless training, read October 21, 2026: five base models, the only Gemma the 26B-A4B at rank 1, rank fixed per model. Support path not tried. |
| Metered spend | $0 | Owned hardware throughout. Nothing was sent to Forge. |

## Authority placement (DAPM)

**Reading:** Portability: could the enterprise take this layer elsewhere without rebuilding?

| Layer | Placement | Basis |
| --- | --- | --- |
| Layer 0A (Compute) | Retained | Mac Studio training: MLX and mlx-lm with a rewritten attention function in plain MLX. Owned, open-source trainer; the same job and adapters ran on NVIDIA hardware too, so nothing ties the workload to Apple silicon. Follows Labs 21, 24 and 26. |
| Layer 0A (Compute) | Retained | DGX Spark training: PyTorch, transformers 5.15.1 and PEFT in NVIDIA's NGC container. Commodity NVIDIA hardware is Retained by the instrument's rule. |
| Layer 0A (Compute) | Retained | Serving runtime under the OpenAI-compatible API, self-served: vLLM 0.25.1 on the Spark and mlx-lm's server on the Studio. The runtime's operating settings (cache budget, concurrency, memory) are the operator's. Keith's 2026-10-21 rule: the underlay is Retained when self-served and Ceded on a managed route. |
| Layer 0A (Compute) | Ceded | A managed fine-tune-and-serve route such as Forge (not run here): the provider sets the model menu, the LoRA rank and placement, and the serving runtime and its settings, behind the same API. Lab 24 measured Forge's own paths. |
| Layer 2C (Reasoning) | Retained | The trained adapters from either box. Open weights, standard PEFT and mlx-lm files, translated by an exact transpose, served on both runtimes with the same outcomes. |

## What this does not establish

- The Studio's training numbers are for attention I rewrote, not mlx-lm as shipped. It matched stock forward outputs on captured layer inputs and its gradients were no worse than stock against an exact fp32 reference on two examples at the start of training; it wasn't checked later in training.
- One training run per box and one loop run per arm, sampled at temperature 1.0. The same adapter moved by three tasks between venues, so differences of a few tasks aren't findings. Lab 18 itself called 14 against 12 noise.
- Density and sequence length changed together relative to Lab 26, so this lab can't say which one moved the training gap from about 6x to 3.8x.
- The Spark's software was Lab 18's stock stack; no faster Spark path was tried. The share-of-ceiling figures are a rough heuristic that leaves out attention.
- Forge was checked against its published self-serve list, not run. Lab 24's notes say its key listed 13 trainable models, which I couldn't reproduce; I didn't ask CoreWeave whether the 31B can be enabled. Other managed services, such as Vertex AI, weren't checked.
- Power is each GPU's rail as its system reports it, not wall-plug, and the two counters aren't comparable system energy.
- The Studio's arms ran the loop and its test sandbox on a separate Mac through an SSH tunnel; the Spark's ran the loop on the box itself. The Studio's first loop arm was voided by my driver fault and resumed after a second fault; the valid results are kept and the faults are in the lab detail.
- Two owned desktop boxes. Nothing here is a claim about a training cluster or a data center, where scale, cooling and power change the economics.

## Open questions

- **Does a faster Spark software stack close the 3.8x?** The Spark trained on stock Hugging Face code, well below its measured compute ceiling. A faster path (fused kernels such as Liger, or Unsloth) on the same job would show how much of the gap is its software and how much its 233 GB/s memory. Untested.
- **Will stock MLX train this model once sliding-window attention lands?** MLX 0.32.4's attention backward is already linear in memory for causal masks. Gemma 4's sliding-window layers still take the quadratic path. MLX PR #4734 adds a sliding-window option to the forward kernels; I've asked on the PR whether the backward will follow. If it does, the Studio's attention rewrite goes away.

## Objection handled

**Isn't a managed fine-tuning service the easy path? Why do the heavy lifting?**

When your model is on the menu, it may be. In Lab 24, Forge trained a Gemma 4 26B adapter in 31 minutes end to end. That's a real convenience, and I'm not dismissing it.

Here the menu ruled it out. Forge's published self-serve list for serverless training has five base models: Gemma 4 26B-A4B at a fixed rank 1, Llama 3.1 8B, Nemotron 3.5 Lightning 30B-A3B, Qwen3.8 27B and Qwen3.6 35B-A3B. Rank and alpha are set per model and can't be changed. Lab 18's job needs the dense 31B at rank 32. Forge routes unlisted models to its support team, which I didn't contact, so this is the self-serve path, not a rejection.

And the menu isn't the only thing the provider decides. In Lab 24, the adapter Forge's own training produced came back with a rank, alpha and placement I didn't choose, and Forge's shared endpoint silently served the base model for it. The heavy lifting on owned hardware is real: re-pinning the Spark to Lab 18's transformers release, a load that bottoms out at 6 GB free, rewritten attention and a cache cap on the Studio. That's the cost of the owned venues, and it bought the model the work called for, trained and served on both.

## Sources

- [CoreWeave: Serverless Training available models](https://docs.coreweave.com/training/serverless-rl/available-models)
- [MLX PR #4563: VJP for scaled dot product attention](https://github.com/ml-explore/mlx/pull/4563)
- [MLX PR #4734: sliding window attention (with this lab's backward-memory comment)](https://github.com/ml-explore/mlx/pull/4734#issuecomment-6325994922)
- [MLX issue #3860: fp32 matmul defaults to TF32-class precision](https://github.com/ml-explore/mlx/issues/3860)

## Summary

The job is Lab 18's coding-agent tune: Gemma 4 31B, a dense model with every parameter active on every token, trained with a rank-32 LoRA on agent transcripts that average 8,663 tokens and run to 30,553. Lab 18 trained it on one DGX Spark in about ten hours, and in its 22-task coding loop the tune solved 14 against the base model's 12, a gap it called noise. This lab reran it on the Spark and on a Mac Studio, scored both adapters on Lab 18's loop on both boxes, and checked whether CoreWeave's Forge could take it.

Both boxes trained from the same data, example order, starting adapter and hyperparameters, in different frameworks. The Spark's first step reproduced Lab 18's logged loss to every digit, 1.0080128908157349, so the Spark run is Lab 18's job on Lab 18's software.

Fitting it took work on both. Lab 18's launcher installed whatever transformers release was current; the current one used about two copies of the weights while loading, and I stopped the 31B load at 39% with 20 GB free. Lab 18's own error traceback identified 5.15.1, and pinned to it the load bottomed out at 6 GB free. On the Studio, stock mlx-lm's attention ran out of memory on the longest example, and so did PyTorch on Apple's GPU, at its indexing limit. I rewrote attention to compute in blocks.

The Spark trained in 10.25 hours (80 tokens per second) and the Studio in 2.69 (306): 3.8 times faster, against about 6 times on Lab 26's small job. On Lab 18's 22-task loop, the adapters from both boxes, each served on both boxes, solved 11 to 14 of 22. Lab 18's own tune solved 14 and its base 12. Nothing separates. On 21 held-out examples both adapters cut the base model's loss from about 1.05 to 0.15.

Serving the loop, the Studio ran each adapter's 22 tasks in 1 hour 46 minutes to about 2.5 hours, against 5 hours 17 to 5 hours 41 on the Spark. It took two fixes to get there, one in my loop driver and one cache cap in mlx-lm's server. vLLM on the Spark ran Lab 18's recorded settings unchanged.

A matrix-multiply benchmark put the Studio's GPU at about 1.2 times the Spark's at bf16 and its memory bandwidth at 4.6 times. Forge's published self-serve list doesn't include the dense 31B, so it wasn't run.

## Detail

Lab 26 ended on a question: the Studio beat the Spark on a small job where each step reads a lot of weights and does little math, so does a heavier job flip it? The heaviest fine-tune in the corpus is Lab 18's: a dense 31B model trained on agent transcripts eight thousand tokens long, scored in a 22-task coding loop. So I ran that on both boxes from the same data, order and starting adapter, scored both adapters in Lab 18's loop on both boxes, and checked whether a managed service could take it.

Forge's self-serve list couldn't. It has five base models, and the dense 31B isn't one of them. That's the first finding, and it isn't about hardware at all. The model you need decides which venues exist for you.

The owned path took work, and I'm counting it as the venues' cost. Lab 18's launcher installed whatever transformers release was current, and today's release used about two copies of the weights while loading; on a 121 GB Spark I stopped it at 39% with 20 GB free. Lab 18's own error log pinned its release to 5.15.1, and on that release the longest example trains with as little as 5.7 GB free. The Studio had the memory and still couldn't run the stock path. mlx-lm's attention ran out at 224 GB on the 30,553-token example, and PyTorch on Apple's GPU stopped at an indexing limit near 11,600 tokens. I rewrote attention to compute a block of queries at a time against only the keys it can see.

With that, the Studio trained the job in 2.69 hours against the Spark's 10.25. Did it train as well? On Lab 18's 22-task loop, the Spark-trained adapter solved 11 on the Spark and 14 on the Studio; the Studio-trained adapter solved 13 and 12. Lab 18's own tune solved 14 and its base 12. Every arm sits in the band Lab 18 already called noise, so the honest reading is the same outcome, not a better or worse one. On 21 held-out examples the two adapters predicted the same next token 98.3% of the time, about as often as two Spark-trained adapters agree.

Serving told the same story about speed and a different one about effort. The Studio ran each adapter's 22 tasks in one and a half to two and a half hours, against more than five on the Spark. But mlx-lm's server kept old conversations' caches until 51 GB of them took the box out of memory, and I had to cap it. vLLM ran Lab 18's recorded settings unchanged. Both spoke the same OpenAI-compatible API, and the workflow still didn't move without changing settings the API never shows. That's a portability finding: the runtime's operating settings travel with the workload.

Did the heavier job close the gap? Partly. Training went from about 6 times to 3.8. A matrix-multiply benchmark puts the Studio's GPU at about 1.2 times the Spark's and its memory bandwidth at 4.6 times, so 3.8 sits nearer bandwidth than compute. The Spark also ran far below its own measured ceiling on stock Hugging Face code. Whether that's its memory or its software, I haven't separated; a faster Spark stack is the next test.

## Method and disclosure

Self-funded. The DGX Spark and the Mac Studio were bought by the practice. No vendor was briefed, reviewed this, or was given advance sight. Google Cloud is a client of this practice, and the model under test is a Google model; Google had no part in this lab.

Inputs: Lab 18's corpus run once through Lab 18's own build function under its pinned release (171 kept, 9 dropped, ids sha256 21c9ee21720614a2, byte-identical under transformers 5.15.1 and 5.20.2). The 2-epoch order drawn by Lab 18's trainer with its seeded sampler, checked by replaying the trainer. One starting adapter, seed 42, shared by both boxes.

Spark training: Lab 18's train_lora_traj.py patched only to read the shared ids, order and starting adapter, in NVIDIA's NGC PyTorch 25.11 container with transformers 5.15.1, accelerate 1.14.0 and PEFT 0.18.0 baked in. Lab 18's release was identified from its own error traceback, line for line. Gradient checkpointing and a chunked loss that skips Gemma's final-logit softcap (a cap of 30 on the output scores), as Lab 18 ran. A watchdog stopped the container below 3 GB free; it never fired.

Studio training: MLX 0.32.4 and mlx-lm 0.32.2, base converted to MLX bf16 with stock mlx-lm, which drops Gemma 4's vision tower (about 1 GB; text-only training never runs it, and the Spark loads it unused). A trainer written to Lab 18's recipe: the same chunked uncapped loss, per-micro-batch averaging over 4, clipping at 1.0, AdamW with torch's defaults, cosine without warmup, per-layer gradient checkpointing. Attention computed in blocks of 2,048 queries with mx.checkpoint per block: full-attention layers through the fused kernel's causal mode, sliding-window layers through small block masks.

Loop eval: Lab 18's p7 loop (stateful.py, its 22-task list, two turns, thinking on, 49,152-token budget, two tasks in flight), unchanged on the Spark apart from the results folder. Spark serving: Lab 18's tuned-serve container settings, recovered from the stopped container (vLLM 0.25.1, runtime LoRA, 131,072-token context, 0.85 memory utilization), with only the adapter mount changed. Studio serving: stock mlx-lm server bound to localhost with Gemma's sampling defaults, a 32 GB prompt-cache cap, and a ten-line wrapper so the loop's model name resolves to the adapter; the loop and its Docker test sandbox ran on a separate Mac through an SSH tunnel, not on the Studio. The loop's job control was replaced there by a two-worker driver, after macOS's bash ran all 22 tasks at once.

Validation: both adapters and Lab 18's original translated by an exact transpose and scored with Lab 18's loss on the 21 held-out examples, in both runtimes.

Benchmark: bf16 matrix multiply at square and job shapes, a 4 GiB bandwidth test and a 10-minute sustained run, in each box's training stack. MLX's fp32 matmul defaults to TF32-class precision (MLX issue #3860), so exact fp32 references were computed on the Studio's CPU.

Power: the Studio's GPU channel from IOReport, read without sudo; the Spark's nvidia-smi power.draw every 5 seconds.

Forge: CoreWeave's available-models page for serverless training, read October 21, 2026. Nothing was sent to Forge.

Peer check: the draft was attacked by two outside models (ChatGPT and Google Antigravity) before revision; their findings and my adjudication are kept with the working files.

The training scripts, the attention rewrite, the benchmark and the checks are the instrument and can be published. The verdict and the trained adapters stay proprietary.

Claude Code drove the bench, so the drafting model's vendor also appears on the bench. A comment posted to MLX PR #4734 from the author's account was generated with Claude and says so.

---
*Layer2C Labs · The Advisor Bench LLC · labs.layer2c.com*