# Lab 026 raw lab detail: On this fine-tune, the Mac Studio trained six times faster than a DGX Spark, and both adapters passed.

One NVIDIA DGX Spark (GB10, 128 GB) and one Apple Mac Studio (M5 Ultra, 256 GB), both bought by
the practice. Runs were made October 20, 2026. Metered spend: $0. This is a follow-up to Lab 024
(finetune-interpreter): the same training split, frozen corpus, control set and scorer.

This download carries the raw material: both training recipes and every declared difference
between them, the canaries, the memory-mode and memory-cap runs, timings, memory, power, losses,
validation and arm results, and the parity checks. It's the working record as kept, dated
section by section, so later sections revise earlier readings. The method for assessing a vendor's
placement is not published. File paths refer to the lab's working tree.

Follow-up measurement for the Lab 024 notes, not a change to the published lab
(`src/data/labs/finetune-interpreter.js` is untouched). Lab 024 showed the Forge adapter serves on
the Studio with the same outcomes as on a Spark. This asks the training-side question: what
changes when the same LoRA job trains on the Apple M5 Ultra under mlx-lm instead of an NVIDIA DGX
Spark under PyTorch and PEFT? Keith rules on sufficiency and on whether any of it becomes a lab
entry.

## 2026-10-20: the job, matched as closely as mlx-lm allows

**Venue.** M5 Ultra Mac Studio (owned, 256 GB unified memory, the Studio), idle at start: no
mlx-lm or oMLX servers, 97% memory free, 966 GB disk free. Base is the MLX bf16 conversion made
during Lab 024 (`~/models/gemma-4-26B-A4B-it-mlx-bf16`, 47 GB), no new download. MLX 0.32.4,
mlx-lm 0.32.2 (`~/venvs/mlx`). The Studio trained and served. The harness ran from the main Mac
through an SSH tunnel to the Studio's localhost-bound server, as in Lab 024. $0, owned hardware.

**Data.** The published split, byte-identical on both venues: `sft_train_msgs.jsonl` (2,872
examples, sha256 `cf092844ecda`) as `train.jsonl`, `sft_val_msgs.jsonl` (320, `9d731f3125fd`) as
`valid.jsonl`.

**Recipe.** `finetune-interpreter-tools/mac_train_r16.yaml` and `mac_train_r1.yaml`, run by
`mac_train.sh` (mlx_lm.lora plus a power sampler, `/usr/bin/time -l` for peak memory).
Attention projections q, k, v, o on all 30 decoder layers; rank 16 scale 2.0 (alpha 32 / r 16)
and rank 1 scale 2.0 (alpha 2 / r 1); dropout 0.05; 2,154 steps (3 epochs of 718 batches of 4);
lr 1e-4, cosine to 0, no warmup; AdamW with weight decay 0, betas 0.9/0.999, eps 1e-8, bias
correction on (torch's defaults); `mask_prompt`; bf16 base; seed 1. Validation loss on the whole
320 at steps 1, 718, 1436 and 2154.

**Pre-flight** (`mac_train_check.py`, on the real split, before any training):

- Placement matches exactly on the language model. mlx-lm wraps 115 modules (q, k, o on 30
  layers, v on 25) with 11,489,280 trainable parameters at rank 16. The Spark adapter's
  language-model tensors are the same 115 modules and the same 11,489,280 parameters. Gemma 4's
  five full-attention layers have no `v_proj` in either implementation (the k_eq_v alias from
  Lab 024). PEFT's reported 15,470,592 includes 3,981,312 vision-tower attention LoRA
  parameters; text-only training never runs that tower, so they stay at their zero init.
- Masking differs by three tokens. The Spark trainer scores the whole assistant chunk,
  `<|turn>model\n{"answer": ...}<turn|>\n`; mlx-lm scores only `{"answer": ...}<turn|>\n`.
- Neither venue trains on the prompt the server sends. With `enable_thinking` false, Gemma's
  template ends the generation prompt with an empty thought channel
  (`<|channel>thought\n<channel|>`); both trainers render the history form without it. Same on
  both venues, so it isn't a venue difference, but it's a train/serve mismatch the lab has
  carried all along.

**Canary.** 30 steps: model load plus training in 25.7 s, peak memory footprint 68.2 GB, GPU rail
about 88 W. No memory pressure, so the full runs went ahead unchanged.

**Spark power and memory probe.** The notes had Spark step time but no training power or memory.
Node 1 ran the published rank 16 recipe (`ft_train.sh`, unchanged) for 15 minutes into a
throwaway output dir, sampled every 5 s (`spark_power_probe.sh`, log
`spark_power_probe_r16.txt`), then the container was stopped. GPU busy: median 42.9 W, max
45.3 W (nvidia-smi `power.draw`). Host memory used peaked at 107 GB of 121 GB, against 7.7 GB at
idle. The probe logged 89 optimizer steps in about 620 s of GPU-busy time, slower than the
published run's 3.74 s per step; I didn't diagnose that, and the table uses the published
full-run timing. Node 1 back to 5.0 W, no containers.

## Results

| | Spark (DGX, PyTorch + PEFT) | Studio (M5 Ultra, mlx-lm) |
|---|---|---|
| r16 wall time, 2,154 steps | 8,056 s (2.2 h) | **993 s (16.6 min)**, incl. load and 4 full val passes |
| r1 wall time | 7,933 s | **976 s** |
| r16 steps per second | 0.267 | 2.17 (wall), about 2.3 training only |
| Peak memory | 107 GB host used of 121 GB (probe) | 69.9 GB process footprint of 256 GB |
| Power while training | about 43 W GPU rail (nvidia-smi, 15-min probe) | about 94 W GPU rail (IOReport) |
| Energy per r16 run, GPU rail | about 95 Wh (43 W x 8,056 s, extrapolated) | 25.8 Wh (measured over the run) |
| Train loss, r16 / r1 | 0.026 / 0.092 | 0.045 / 0.134 (not comparable, see confounds) |
| Validation loss, r16 / r1 | not measured | 0.005 / 0.007 |
| Validation exact match, r16, same Mac runtime | 281/320 strict, **308** fence-stripped | **300/320** (no fences) |
| Validation exact match, r1, same Mac runtime | 241/320 strict, **295** fence-stripped | **243/320** (no fences) |

Arms (Mac-trained served on the Studio by mlx-lm, greedy; Spark-trained from the published runs,
served by vLLM 0.25.1 on a Spark, temperature 1.0 seed 1). 0 crashes in 2,498 sessions per
adapter. Paired against the fair few-shot (`ft_fewshot2_*`), exact McNemar, fixed / new / p:

| | Few-shot (fair) | Spark r16 | **Mac r16** | Spark r1 | **Mac r1** |
|---|---|---|---|---|---|
| Frozen: E non-contestable leaks | 13/496 | 1/496 (13/1, p 0.002) | **0/496** (13/0, p 0.0002) | 0/496 (13/0, p 0.0002) | **56/496** (10/53, p 3e-8) |
| Frozen: F costed hand-offs | 16/955 | 4/955 (12/0, p 0.0005) | **7/955** (9/0, p 0.004) | 9/955 (7/0, p 0.016) | **68/955** (0/52, p 4e-16) |
| Frozen: F benign completion | 275/309 | 273/309 | 273/309 | 273/309 | 271/309 |
| Slice: E leaks | 8/166 | 0/166 | **0/166** (8/0, p 0.008) | 3/166 | **18/166** (5/15, p 0.04) |
| Slice: F costed hand-offs | 6/294 | 3/294 | **3/294** (3/0, p 0.25) | 5/294 | **14/294** (1/9, p 0.02) |
| F tokens per session | 1,468 | 285 | 286 | 284 | 276 |

Session parity, Mac-trained against Spark-trained (`mac_parity.py`, which reproduces the published
Forge Mac-vs-Spark parity of 1,910/1,910 sessions and 2,159/2,251 answers exactly):

| | Same session outcome | Same model answer |
|---|---|---|
| r16, frozen | 1,901/1,910 (99.5%) | 2,072/2,269 (91.3%) |
| r16, slice | 584/588 (99.3%) | 641/691 (92.8%) |
| r1, frozen | 1,669/1,910 (87.4%) | 1,625/2,245 (72.4%) |
| r1, slice | 528/588 (89.8%) | 523/685 (76.4%) |

For scale: the Forge adapter served on both venues (same trained weights) gave 100% and 95.9%.

## What it shows

**Rank 16 transfers; rank 1 doesn't.** The Mac-trained rank 16 clears Keith's bar on both legs
against the fair few-shot, on the frozen corpus and the slice, and matches the Spark-trained rank
16 within a few sessions (F 7 against 4 frozen; identical on the slice). The Mac-trained rank 1
fails: it leaks more than few-shot on E (56 against 13) and quadruples F's costed hand-offs. Its
validation misses are concentrated: confirm 14/45, with 31 of the 32 errors being "unclear"
answered "yes", the injection follow-up line where the Spark rank 1 also showed its surface-form
seam on the slice. On the Mac that seam opens on the frozen corpus too.

**The rank 1 gap is training, not serving.** Served by the same mlx-lm runtime on the same Studio,
the Spark-trained rank 1 scores 295/320 (fence-stripped) and the Mac-trained rank 1 scores 243.
At rank 16 the same comparison is 308 against 300. So the two trainers produce different
adapters from the same data, and the difference only becomes material when the adapter has one
direction per layer to work with. Which declared difference does it (clipping, loss
normalization, length-sorted batches, the three header tokens, init, seed) isn't tested here.

**A fence habit shows up only in the Spark-trained adapters on the Mac.** 28 of the Spark r16's
and 54 of the Spark r1's validation answers come back wrapped in a json code fence on the Mac
runtime, content correct. Mac-trained adapters never fence. Lab 024's first Spark step had loss
7.8, mostly from base Gemma's fence habit, so the habit is the base prior. The Spark adapters
weren't validated strictly on vLLM in Lab 024 (smoke tests only), so I can't say whether they
fence there too. The session harness parses through the fence, so arms aren't affected.

**Speed.** On this job the Studio trained 8.1 times faster in wall time than the Spark and used
about a quarter of the GPU-rail energy, at roughly twice the GPU-rail power. That is a stack
comparison as much as a chip comparison: see confounds 1 and 2. The Spark run was also memory
tight (107 of 121 GB); the Studio used 70 of 256 GB.

## Declared confounds

1. **Software stack, not only silicon.** Spark: NGC PyTorch 25.11, transformers 5, PEFT 0.18,
   Lab 018's `train_lora_traj.py`. Studio: MLX 0.32.4, mlx-lm 0.32.2 trainer. The MoE forward
   path differs (transformers' Gemma 4 expert implementation against mlx-lm's fused SwitchGLU
   gather kernels). The 8x can't be attributed to the M5 Ultra over GB10 alone.
2. **Gradient checkpointing and chunked loss on the Spark, neither on the Studio.** The Spark
   recipe recomputes activations (it was built for 32K-token trajectories) and runs lm_head plus
   cross-entropy in checkpointed 2,048-token chunks. The Studio had the memory room to skip both.
3. **Batching.** Spark: micro-batch 1 times 4 accumulation steps, random order, loss averaged
   per example. mlx-lm: one batch of 4, padded, examples sorted by length into batches and the
   batch order shuffled, loss averaged per token.
4. **Gradient clipping.** transformers clips at norm 1.0 by default; mlx-lm doesn't clip.
5. **Masking.** The Spark scores the three header tokens `<|turn>model\n` as well as the answer;
   mlx-lm scores the answer and end-of-turn only. This also makes the two train-loss numbers
   incomparable: the header tokens are near-free after the first steps and pull the Spark's
   mean down.
6. **LoRA init.** PEFT initializes A Kaiming-uniform; mlx-lm initializes A uniform within
   1/sqrt(in). B is zero in both. Scale (alpha/r) and dropout (0.05) match.
7. **Seed.** Spark: transformers default (42). Studio: 1. One seed per venue; no repeat runs, so
   run-to-run variance on either venue is unmeasured. That matters most for the rank 1 result.
8. **Optimizer implementation.** torch's AdamW against MLX's AdamW, with matched hyperparameters
   (mlx-lm's defaults of weight decay 0.01 and no bias correction were overridden).
9. **Evaluation sampling.** The Mac samples greedily (mlx-lm default); the published Spark arms
   used Gemma's generation config, temperature 1.0, seed 1.
10. **Serving runtime in the arms.** Mac-trained adapters were served by mlx-lm, Spark-trained by
    vLLM. Parity mixes training venue with serving venue. The same-runtime validation rows
    separate them for the 320-example check only. For scale, Lab 024's same-weights parity across
    the two runtimes was 100% of sessions.
11. **Power is a GPU rail on each side, not wall-plug.** Studio: IOReport "GPU Energy", read
    without sudo (`mac_power_sample.py`, the source macmon uses). The CPU, DRAM and ANE channels
    read zero without privileges, so CPU and memory power aren't in the Studio number. Spark:
    nvidia-smi `power.draw` over a 15-minute probe, extrapolated to the run; I haven't verified
    what that counter covers on GB10. No wall meter on either. Treat the energy row as
    directional.
12. **Validation loss** was measured only on the Studio. The Spark trainer doesn't evaluate.
13. **Train/serve prompt mismatch** (the empty thought channel), shared by both venues.

## Files

- Recipe and tools (new): `finetune-interpreter-tools/mac_train_r16.yaml`, `mac_train_r1.yaml`,
  `mac_train.sh`, `mac_train_check.py`, `mac_power_sample.py`, `mac_peft_to_mlx.py` (exact
  PEFT-to-mlx translation for the attention-only adapters), `mac_val_match.py`,
  `mac_parity.py`, `spark_power_probe.sh`, `spark_power_probe_r16.txt`.
- Training records: `finetune-interpreter-tools/mac_training/` (train logs, times, power
  samples, serve logs).
- Adapters: `finetune-interpreter-tools/adapters/mac-trained-r16/` (sha256 `0896993fcf3b9d57`,
  46 MB) and `mac-trained-r1/` (`6579268ab72109c7`).
- Results: `results/ft_mac_trained_r16_{test,slice,val}`, `results/ft_mac_trained_r1_{test,slice,val}`,
  `results/ft_spark_trained_r16_on_mac_val`, `results/ft_spark_trained_r1_on_mac_val`, tallies
  `results/tally_{test,slice}_mac_trained_vs_fewshot.json` and
  `results/tally_{test,slice}_mac_trained_r1_vs_fewshot.json`. Arm logs `mac_trained_arms.log`,
  `mac_trained_arms_r1.log`.

**State.** Studio idle after (no mlx-lm or python processes, 97% memory free); tunnel closed;
both Sparks idle (5.0 W and 4.7 W, no containers). The Studio keeps `~/ft-mac-train/` (data,
both adapters, the translated Spark adapters).

## Open, for Keith

- Whether rank 16 parity is enough to call the Studio a training venue for this job, given the
  rank 1 failure.
- The rank 1 split is one seed per venue. A second Mac seed (about 16 minutes, $0) would say
  whether it's the trainer or the draw. Clipping at 1.0 and per-example loss averaging are the
  two confounds mlx-lm could close with a small trainer change, if the second seed holds the gap.
- A strict validation of the Spark adapters on vLLM would settle whether the fence habit is a
  property of the adapter or of the runtime.
- A wall meter on both boxes would replace the GPU-rail energy row.

## 2026-10-20 (later): what memory mode bought, the 2x2 and the capped Studio runs

Keith approved two single-box runs ("Let's take the two single box runs") to see how much of the
8x the memory-saving mode explains. Then, after "Let's run that cap limit," two Studio runs with
an MLX memory limit.

**Spark, savings off.** `mac_training/spark_train_lora_traj_nosave.py`, a copy of Lab 018's
trainer with two changes and nothing else: `gradient_checkpointing=False`, and the stock
`Trainer` loss (the model's forward with labels, full logits) in place of `ChunkedCETrainer`.
Launcher `spark_ft_train_nosave.sh` (the published launcher plus a version print: torch
2.10.0a0 nv25.11, transformers 5.20.2, PEFT 0.18.0). One-step canary on node 1 first, on the
longest example: peak 95.5 GB used, 29 GB available, above the 10 GB line. The full run had a
watchdog that would stop the container below 6 GB available. It never fired: peak 92.5 GB used,
32 GB minimum available.

**Studio, savings on.** `mac_train_r16_gc.yaml`: the r16 config plus `grad_checkpoint: true`.
mlx-lm has no chunked loss, so only half the Spark's memory-saving mode is mirrored.

**Studio, capped.** `mac_train_capped.py` sets `mx.set_memory_limit` and `mx.set_cache_limit`
before calling mlx-lm's trainer, savings off. At 121 GiB (the Spark's usable pool) and 64 GiB.
MLX's limit is a guideline, not a hard cap: past it, the allocator frees cache and waits, and only
fails when physical RAM runs out.

| Rank 16, 2,154 steps | Wall time | s/step | Peak memory | GPU-rail power | Energy, GPU rail | Train loss | Val match (Mac runtime, fence-stripped) |
|---|---|---|---|---|---|---|---|
| Spark, savings on (published) | 8,056 s | 3.74 | 107 GB used of 121 | ~43 W (probe) | ~95 Wh (extrapolated) | 0.026 | 308/320 |
| Spark, savings off | 5,713 s | 2.65 | 92.5 GB used of 121 | 43.7 W | 69.6 Wh (measured, incl. load) | 0.023 | 309/320 |
| Studio, savings on | 1,282 s* | ~0.57 | 58.6 GB footprint | 98.7 W | 35.0 Wh | 0.045 | 306/320 |
| Studio, savings off | 993 s* | ~0.43 | 69.9 GB footprint | 94.2 W | 25.8 Wh | 0.045 | 300/320 |
| Studio, off, cap 121 GiB | 994 s* | ~0.43 | 69.9 GB | 94.2 W | 25.8 Wh | 0.047 | not served |
| Studio, off, cap 64 GiB | 987 s* | ~0.43 | 69.7 GB | 94.6 W | 25.9 Wh | 0.045 | not served |

\* Studio wall time includes model load and four full validation passes (about 62 s). Spark times
are the trainer's `train_runtime`, which excludes load. The Studio's s/step is wall time less
those passes, so it's approximate. Spark memory is host "used" from `free`, which includes about
7.7 GB of OS at idle. Studio memory is the process's peak footprint from `/usr/bin/time -l`.
Neither is the same counter.

**What memory mode explains.** It's worth about 1.4x on the Spark (8,056 to 5,713 s) and 1.3x on
the Studio (1,282 to 993 s). The original 8.1x compared the Spark with savings on against the
Studio with savings off. Like for like, the Studio is 6.3x faster with savings on and 5.8x faster
with savings off. So memory mode explains about 1.3 to 1.4x of the 8.1x. In log terms that's 12
to 16% of the gap. The remaining 5.8 to 6.3x is stack and silicon together, and nothing here
splits them.

**The cap doesn't touch it.** Capped at the Spark's pool (121 GiB), the Studio ran in the same
994 s, because the job only needs about 70 GB. At 64 GiB (68.7 GB) it also ran in 987 s, and the
process peaked at 69.7 GB, about 1 GB over. That's plausibly Python's own non-MLX memory, but the
cap is soft and I can't separate the two. No swap was used. On this job, the Studio's 256 GB isn't
what makes it fast. It fits in roughly the same memory as the Spark's working set, minus the
Spark's OS and the PyTorch overhead.

**Quality didn't move.** Every new adapter validates within the spread of the earlier run on its
own venue: Spark 308 to 309, Studio 300 to 306. So the frozen-corpus and slice arms were skipped.
The two Studio runs (checkpointing on and off, same seed and data, mathematically the same
gradients) agree on 302 of 320 answers. That's a rough reading of run-to-run numerical drift on
one venue, and it's the scale to read the earlier rank 1 split against, though rank 1's gap (243
against 295) is far larger. The fence habit shows up again in Spark-trained adapters (33 fenced
for savings off) and now in one Studio adapter (10 fenced, checkpointing on). So it isn't purely a
Spark-trainer property.

**New confounds this round.**

14. **The published Spark loss skipped Gemma's logit softcap.** Gemma 4 sets
    `final_logit_softcapping` 30. `ChunkedCETrainer` calls `lm_head` directly, so the published
    Spark adapters trained on uncapped logits. The savings-off run uses the model's own forward,
    which applies the cap, as mlx-lm does. Turning savings off on the Spark therefore changed the
    objective as well as the memory mode. This bears on the earlier rank 1 split: the Mac-trained
    rank 1 trained on capped logits, the Spark rank 1 on uncapped. Untested whether it's the cause.
15. **Loss normalization changed with it.** The stock loss in transformers 5.20 averages over all
    tokens in the four accumulated micro-batches; the chunked loss averaged per micro-batch. The
    savings-off loss isn't comparable to the published one for that reason too.
16. **transformers isn't pinned in the Spark launcher.** Each run `pip install`s the current
    transformers. The savings-off run got 5.20.2; the published runs' version wasn't recorded.
17. **Partial mirror on the Studio.** mlx-lm checkpointing is on, but there's no chunked loss to
    mirror. At 354 tokens, the logits are about 0.4 GB, so the missing half is probably small.

**Why the Studio might be this far ahead (a hypothesis, not tested).** This job is short sequences
(about 230 tokens), batch 4, on a mixture-of-experts model with about 4B active parameters per
token. Each step reads a lot of weights and does little math, so memory bandwidth probably matters
more than CUDA's compute edge. The Studio drew about 95 W, well under what its cooling handles;
Keith noted the fan never came up. On the Spark side, transformers' Gemma 4 expert path isn't
CUDA's best case. I haven't measured either. A long-sequence run (where compute dominates) on both
boxes would test the bandwidth reading.

**State.** Studio idle (no mlx-lm or python processes, 97% memory free), tunnel closed. Node 1
idle (5.2 W, no containers). Node 2 is running a vLLM container, `pb-gemma26b` (vllm-openai
v0.25.1, up 37 minutes at check), that this session didn't start and left alone.

**Files added this round.** `mac_train_r16_gc.yaml`, `mac_train_capped.py`, `mac_train_capped.sh`;
`mac_training/spark_*` (the no-savings trainer copy, launcher, canary, run scripts, logs, power);
`mac_training/*r16gc*`, `*r16cap121*`, `*r16cap64*`; adapters `adapters/mac-trained-r16-gc/`
(`993df2f274aaba95`) and `adapters/spark-nosave-r16/` (`3b1108b2094601b8`); results
`results/ft_mac_trained_r16_gc_val`, `results/ft_spark_nosave_r16_on_mac_val`.

**Open, for Keith.** Whether the softcap difference (14) is worth a Spark rank 1 rerun with the
cap, to see if it closes the rank 1 split. Whether to run the long-sequence test of the bandwidth
reading.

## 2026-10-20 (evening): second Mac seed for rank 1

Keith: "It doesn't look like a 2nd seed costs much in time or money. Let's run it."
`mac_train_r1_s2.yaml` is `mac_train_r1.yaml` with seed 2 and its own output dir, nothing else.
It ran alone on the Studio: 978 s wall, 69.7 GB peak, 94.9 W GPU rail, 25.7 Wh, mean train loss
0.141, validation loss 0.008 (seed 1: 0.007). Adapter `adapters/mac-trained-r1-s2/`
(`1a1712f285ec3d91`). Scored exactly like seed 1, 0 crashes in 2,498 sessions.

| Rank 1, vs fair few-shot | Few-shot | Spark-trained | Mac seed 1 | Mac seed 2 |
|---|---|---|---|---|
| Validation, same Mac runtime (fence-stripped) | | 295/320 | 243/320 | 258/320 |
| Frozen: E non-contestable leaks | 13/496 | 0/496 | 56/496 (10/53) | 42/496 (13/42, p 0.0001) |
| Frozen: F costed hand-offs | 16/955 | 9/955 | 68/955 (0/52) | 64/955 (0/48, p 7e-15) |
| Frozen: F benign completion | 275/309 | 273/309 | 271/309 | 269/309 (guard fails) |
| Slice: E leaks | 8/166 | 3/166 | 18/166 | 15/166 |
| Slice: F costed hand-offs | 6/294 | 5/294 | 14/294 | 15/294 (1/10, p 0.01) |

Parity on the frozen corpus: seed 2 against seed 1, 94.6% of sessions and 89.6% of answers.
Seed 2 against the Spark-trained r1, 91.2% and 78.9% (seed 1 against Spark was 87.4% and
72.4%). Slice, seed 2 against Spark: 93.9% and 83.6%.

**It's the trainer, not the draw.** Both Mac seeds fail Keith's bar on F, both are worse than
few-shot on E, and they fail the same way. Their validation errors share the same two patterns:
self-pay "no" read as "unclear" (23 in each seed) and the injection follow-up "unclear" answered
"yes" (31 in seed 1, 13 in seed 2). The two seeds sit far closer to each other than either does
to the Spark-trained adapter: E 56 and 42 against 0, F 68 and 64 against 9. One caveat: the
Spark side is still one seed. A Spark draw this bad would have to be very unlikely to flip the
reading, but I haven't measured Spark variance.

What in the trainer does it is still untested. The strongest candidate from this file is the
logit softcap (confound 14): the published Spark rank 1 trained on uncapped logits, and both Mac
seeds on capped ones. Next are clipping (4), per-example against per-token loss (3, 15), and the
three header tokens (5). At rank 16 the same trainer difference didn't matter (Mac 300/308, the
arms passed). So the gap shows up only when the adapter is narrow. That fits the earlier reading
that one direction per layer is fragile on this job.

## 2026-10-20 (evening): Forge's recipe, trained on the Studio

Keith approved training Forge's exact recipe on the Mac, so the Forge-vs-Mac training time
compares the same job. `mac_train_forge.yaml`: rank 1, scale 32 (Forge's alpha 32 / r 1),
dropout 0.0 (Forge's `adapter_config.json` says 0.0), keys q, k, v, o, the dense MLP's gate, up
and down, and the MoE experts' gate, up and down (`experts.switch_glu.*`) on all 30 layers.
Same split, 3 epochs, 2,154 batches of 4, lr 1e-4 cosine to 0 with no warmup, `mask_prompt`,
seed 1. mlx-lm can put LoRA on the fused SwitchGLU experts (`LoRASwitchLinear`, the same path
Lab 024's translation used), so no target was substituted.

**One structural difference, declared.** Forge's expert gate and up projections are one fused
`gate_up` layer, so each expert's rank-1 A vector is shared between gate and up. mlx-lm splits
gate and up into separate layers, each with its own A. That's about 41.7M trainable parameters on
the Mac against 30,917,760 in Forge's file. The 10.8M difference is the extra expert A matrices
(128 experts by 2,816 by 30 layers). Forge stores the adapter in bf16; mlx-lm trains fp32 LoRA
weights. Forge's other hyperparameters (optimizer, clipping, loss normalization, init) come
through ART and aren't visible in the file.

| Forge recipe | Forge (CoreWeave service) | Studio (M5 Ultra, mlx-lm) |
|---|---|---|
| Wall time | 1,852 s (service wall clock: includes provisioning, checkpointing, upload; GPU type and count unknown) | 1,231 s including load and 4 full validation passes |
| Training only | not reported separately | about 1,140 s (wall less the four ~20 s validation passes and ~10 s load; estimated) |
| Peak memory | not visible | 74.8 GB footprint |
| Energy, GPU rail | not visible | 27.7 Wh at 81.8 W median |
| Final loss | 2.7e-6 (last single batch) | 0.004 (last 10-step window); mean 0.030; validation loss 0.005 |
| Validation exact match, same Mac runtime | 309/320 (Forge's adapter, Lab 024) | 300/320 strict, 304 fence-stripped (4 fenced) |
| Frozen: E leaks / F costed | 0/496, 4/955 (Forge's adapter on the Studio) | **0/496, 6/955** (E 13/0, p 0.0002; F 10/0, p 0.002 vs few-shot) |
| Frozen: F benign completion | 273/309 | 273/309 |
| Slice: E leaks / F costed | 0/166, 2/294 | 0/166, 3/294 |

Session parity, Mac-trained Forge recipe against Forge's own adapter (both served on the Studio
by mlx-lm, both greedy, so serving isn't a confound here): frozen corpus 1,906 of 1,910 sessions
(99.8%), 2,171 of 2,267 answers (95.8%); slice 584 of 588 (99.3%), 656 of 691 (94.9%). That's
the same scale as Forge's adapter across two serving venues (100% and 95.9%).

**What it says.** Rank 1 trains fine on the Studio when it's placed the way Forge places it. The
Forge recipe, trained on owned Apple hardware in about 20 minutes, reproduces Forge's result on
both legs and lands within 1 to 2 sessions of Forge's adapter everywhere. So the rank 1 failure
above isn't "rank 1 on a Mac." It's rank 1 on attention only with this trainer, which the wider
placement (and alpha 32) makes robust to. On time, the Studio's 1,231 s against Forge's 1,852 s
isn't a hardware comparison. Forge's number is a service wall clock on unknown GPUs and includes
provisioning and upload. What it does show is that for this job, the owned box finished faster
than the hosted service's end-to-end time, at $0 marginal cost against Forge's metered charge.

**State.** Studio idle (no mlx-lm, python or queue processes, 97% memory free), tunnel closed.
Spark node 1 idle (5.0 W). Node 2 still runs another session's `pb-gemma26b` vLLM container
(36 W), untouched by me.

**Files added this round.** `mac_train_r1_s2.yaml`, `mac_train_forge.yaml`,
`mac_queue_after.sh`; `mac_training/*r1s2*`, `*forge*`; adapters
`adapters/mac-trained-r1-s2/` and `adapters/mac-forge-recipe/` (`d425df46b4723936`, 167 MB
fp32); results `results/ft_mac_trained_r1_s2_{val,test,slice}`,
`results/ft_mac_forge_recipe_{val,test,slice}`, tallies
`results/tally_{test,slice}_mac_trained_r1_s2_vs_fewshot.json` and
`results/tally_{test,slice}_mac_forge_recipe_vs_fewshot.json`; arm logs
`mac_trained_arms_r1s2.log`, `mac_forge_recipe_arms.log`.

**Open, for Keith.** Whether to test the softcap as the rank 1 cause (a Mac r1 run with the cap
removed, or a Spark r1 with it applied, about 16 minutes or 2.2 hours respectively, both $0).

## 2026-10-20: ruling

Keith: "I think we have enough to publish because the next question is a larger job." The
measurement is sufficient as it stands. Two follow-ups are open and stay out of this record:
the rank 1 softcap test, and a bf16 compute microbenchmark (Keith doubts the Spark out-computes
the M5 Ultra GPU; I'd claimed it might without evidence, and dropped that). The next question is
a larger job, where compute rather than memory bandwidth may set the pace. What publish means
for this record, and in what form, goes through the lab process (layer2c-lab, peer-check,
publish-gate), not this notes file.