# Lab 027 raw lab detail: Pick the model first. It picks the training hardware, and it can rule out the managed service.

One NVIDIA DGX Spark (GB10, 128 GB) and one Apple Mac Studio (M5 Ultra, 256 GB), both bought by
the practice. Runs were made October 20 and 21, 2026, including Lab 018's 22-task loop on both boxes. Metered spend: $0. This follows Lab 026
(finetune-venue) and reruns Lab 018's (trained-by-whom) loop-worker tune: the same corpus, recipe
and software, from shared inputs on both boxes. CoreWeave Forge was checked against its published
model list and not run.

This download carries the raw material: the pinning and load findings, the shared inputs, the
microbenchmark, the attention rewrite and its checks, the gradient investigation, per-run timing,
memory, power, losses and held-out validation. It's the working record as kept, dated section by
section, so later sections revise earlier readings (the gradient section in particular walks back
several of my own early conclusions). The method for assessing a vendor's placement is not
published. File paths refer to the lab's working tree.
## 2026-10-20: staging and P1

**Weights.** google/gemma-4-31B-it snapshot `3548789` (Lab 018's refs/main; its chat template and
tokenizer match the copies saved in Lab 018's adapter dir byte for byte). Relayed node 1 to Studio
through the main Mac, 22:01 to 22:15 UTC, sha256 equal on both shards (`stage_31b_studio.log`).
Converted to MLX bf16 at `~/models/gemma-4-31B-it-mlx-bf16` (57 GB, vision tower dropped).

**Config facts read at pre-flight.** 60 layers (50 sliding, window 1,024, head_dim 256, 16 KV heads;
10 full, head_dim 512, 4 KV heads, k_eq_v), hidden 5,376, MLP 21,504, vocab 262,144, tied
embeddings, `final_logit_softcapping` 30 (Lab 018's chunked loss skipped it).

**Lab 018's run, from its own checkpoint** (`~/tbw-train/out-loop-single/checkpoint-86`):
max_grad_norm 1.0, weight decay 0, AdamW fused, betas 0.9/0.999, eps 1e-8, seed 42, cosine, warmup
0, accumulation 4, 2 epochs, 86 steps, gradient checkpointing on. Step-1 loss 1.008013, step-1 grad
norm 34.4, final loss 0.224. Checkpoint-43 written 13:59 and final 19:02 UTC on 2026-08-22: the
second epoch took about 5 h 3 min, about 423 s/step. Adapter: 230 language-model LoRA modules (460
fp32 tensors) plus 108 untouched vision-tower modules.

**Stopped: P1 under transformers 5.20.2.** 101 GB used and 20 GB available at 39% of the weights
loaded; stopped by hand at 22:07 UTC, node 1 back to 7 GB used. Load probe, 26B (48 GB of
parameters): 5.20.2 took available memory from 113.5 to 17.3 GB at peak; 5.15.1 to 30.0 GB
(`probe_tf*_26b.log`). Re-pinned to Lab 018's stack (see the spec).

**P1 (no model load).** Kept 171, dropped 9 overlong, 1,481,399 tokens, 97,971 labeled, longest
30,553; val 21. `train_ids.jsonl` sha256 `21c9ee21720614a2`, `val_ids.jsonl` `9f5d782d284f83bd`,
byte-identical under 5.15.1 and 5.20.2. Order: 86 steps, epochs end at steps 43 and 86; step 1 is
examples 15, 168, 138, 169 (14,325 / 1,625 / 15,724 / 1,610 tokens). `c1_order_check.py`: the patched
Lab 018 Trainer walks the order exactly. Shared init: seed 42 on the Studio,
`init_adapters_seed42.safetensors` `2f0522b702a619f4`, PEFT translation `init_peft_seed42.safetensors`
`5b8e1bf832c82d0f` (B all zero).

## A0, Studio (22:24 to 22:36 UTC)

bf16 matmul, TFLOPS, median of 3 (forward / transposed product):

| Shape (M x K x N) | Studio |
|---|---|
| square 4096 / 8192 / 16384 | 103.6 / 119.4 / 103.5 |
| q, sliding (8192 x 5376 x 8192) | 120.1 / 120.4 |
| q, full (8192 x 5376 x 16384) | 120.2 / 120.8 |
| k/v, sliding (8192 x 5376 x 4096) | 114.5 / 114.3 |
| o, sliding (8192 x 8192 x 5376) | 112.6 / 112.2 |
| MLP up (8192 x 5376 x 21504) | 120.7 / 121.6 |
| MLP down (8192 x 21504 x 5376) | 95.2 / **66.7** |
| lm_head chunk (2048 x 5376 x 262144) | 120.1 / 123.5 |

Bandwidth (c = a + b, bf16, 4 GiB): 1,083 GB/s. Sustained 8192 matmul for 600 s: 119.8 at the start,
120.6 at the end, no throttling. GPU rail median 199 W, max 398 W. The 66.7 on the transposed MLP-down
product is an outlier I haven't looked into.

## A0, Spark node 1 (22:44 to 22:56 UTC, pinned image)

| Shape | Spark (fwd / transposed) | Studio | Studio / Spark |
|---|---|---|---|
| square 4096 / 8192 / 16384 | 91.3 / 97.7 / 90.2 | 103.6 / 119.4 / 103.5 | 1.13 / 1.22 / 1.15 |
| q, sliding | 97.8 / 96.8 | 120.1 / 120.4 | 1.23 |
| q, full | 97.7 / 96.2 | 120.2 / 120.8 | 1.24 |
| k/v, sliding | 95.5 / 95.0 | 114.5 / 114.3 | 1.20 |
| o, sliding | 90.1 / 89.8 | 112.6 / 112.2 | 1.25 |
| MLP up | 96.0 / 95.2 | 120.7 / 121.6 | 1.26 |
| MLP down | 76.9 / 82.7 | 95.2 / 66.7 | 1.24 / 0.81 |
| lm_head chunk | 89.2 / 93.6 | 120.1 / 123.5 | 1.35 |
| bandwidth, GB/s | 233 | 1,083 | 4.6 |

Sustained 8192 matmul, 600 s: 96.7 at the start, 94.9 at the end. GPU rail (nvidia-smi
`power.draw`) median 87 W, max 90.5 W; SM clock 2,099 to 2,112 MHz; 85 C max.

**So M is about 1.2 and W about 4.6.** Keith's doubt holds on this measurement: the Spark doesn't
out-compute the M5 Ultra GPU at bf16. The Studio is about 1.2x on the job's shapes, at about 2.3x the
GPU-rail power (199 W against 87 W).

## Studio pre-flight and the attention change

- **Step 1, stock attention:** base loss on the first four examples, averaged per micro-batch,
  1.00123 against Lab 018's 1.00801. Capped loss would have been about 0.43, so the uncapped objective
  is confirmed. 230 LoRA modules, 90,030,080 trainable, fp32.
- **Canary, stock attention:** one step on the 30,553-token example, Metal out-of-memory at a 224 GB
  footprint, gradient checkpointing on. No swap growth.
- **Blocked attention** (`blocked_attention.py`, checks in `attn_unit*.py`, `attn_capture.py`,
  `attn_det.py`, `attn_fp32ref.py`). Two bugs found on the way, both in my first version: blocks of
  512 rows hit a less accurate kernel path (now a floor of 1,024 and a tail fold), and an explicit
  boolean mask on the head_dim-512 full layers drifted 0.2 from an fp32 reference (full layers now use
  the kernel's own bottom-right "causal"). On captured layer inputs, blocked equals stock on full
  layers exactly and sits at the same distance from fp32 as stock on sliding layers. Loss on a
  5,085-token example: fp32 attention 1.164, stock 1.113, blocked 1.121.
- **Step 1, blocked:** 0.99324 against 1.00801.
- **Canary, blocked:** 30,553 tokens, one step 114.1 s (includes compile), loss 1.579, grad norm
  97.0, MLX peak 142.5 GB, process footprint 205 GB of 256, swap unchanged.

## C2, Studio (started 22:52 UTC)

`run_studio.sh c2 --mode train --init shared/init_adapters_seed42.safetensors`, blocked attention
2,048. Step 1: loss 1.0182 (micro 1.884 / 0.594 / 0.736 / 0.859), grad norm 10.35, 109.8 s for
33,284 tokens. **Open:** Lab 018's step-1 grad norm was 34.4 from an unseeded init. C1 starts from the
same init as C2, so its step-1 grad norm is the real comparison.

## Spark chain (node 1, started 22:44 UTC)

`run_spark_chain.sh`: A0, then the P2 canary on the 30,553-token example with Lab 018's CANARY path,
then C1 only if the canary leaves at least 15 GB available. Watchdog stops a container at 6 GB.

**C2 stopped by hand at step 17** (23:2x UTC) to investigate a grad-norm mismatch with the Spark
canary (below). Nothing from that partial run is used.

## Spark P2 canary (node 1)

- **Try 1** (22:56): the 31B load reached the end, then C1's StepLog crashed opening a file before
  the output dir existed (my bug, fixed). The load transient took available memory to 6 GB.
- **Try 2** (23:04 to 23:21): the training step on the 30,553-token example completed: loss 1.572,
  grad norm 21.58, 633.6 s, allocator peak 101.4 GB. **Available memory sat at 5 to 6 GB for the
  whole step.** A stale watchdog from try 1 (6 GB stop, same container name; my bug) then stopped
  the container while the adapter was being saved (rc 137). Same example on the Studio (blocked
  attention): loss 1.579, grad norm 97.0, 114 s.
- **The 15 GB gate isn't met.** Lab 018 trained this example in this stack on node 2, so 5 to 6 GB
  is apparently its normal margin, but the spec's gate says this goes to Keith: run C1 at that
  margin, or drop the longest examples on both boxes (declared), or stop. C1 has not started.
- **Two near-misses of my own, both caught by the watchdog:** `spark_grad.py` (a gradient dump, no
  gradient checkpointing, unlike Lab 018's trainer) took node 1 to 2 GB available at 1,625 tokens and
  1 GB at 5,085 tokens before the watchdog stopped it. Node 1 recovered both times (115 GB available,
  no containers). Not to be rerun without checkpointing.

## The gradient investigation (Studio, Spark, CPU)

The Studio-vs-Spark grad-norm gap led to a long check of whether either venue computes the right
LoRA gradient. What held up:

- **"fp32" on the M5 Ultra GPU isn't fp32.** MLX 0.32.4 fp32 matmul, 2048 x 5376 x 2048 against a
  float64 reference: relative error 7.9e-4 on the GPU, 3.0e-6 on the CPU (`mm_precision.py`). So
  GPU-side "fp32 references" and finite differences were noise; several early conclusions of mine
  (that mlx's fused attention backward was broken) didn't survive this.
- **Ground truth on the Studio CPU** (`grad_truth_cpu.py`): whole 31B in fp32, plain attention,
  B = 0 (the real training start), shared seed-42 init, dropout 0. Example k = 168 (1,625 tokens):
  loss 0.5838, LoRA-B grad norm 14.04, 130 s.
- **Both venues' bf16 gradients are only loosely aligned with the truth** on that example:

| Path | Loss | Grad norm | Cosine to truth |
|---|---|---|---|
| CPU fp32 (truth) | 0.5838 | 14.0 | 1 |
| Spark, PyTorch bf16 (Lab 018 stack, sdpa) | 0.6076 | 8.9 | 0.22 |
| Studio, mlx-lm stock fused attention | 0.5940 | 59.7 | 0.28 |
| Studio, blocked plain-ops attention | 0.5869 | 395 | -0.12 |

  Spark and Studio-stock gradients have cosine 0.09 with each other. So at B = 0, per-step gradient
  direction on this model is mostly bf16 noise on both boxes, and grad-norm differences between
  venues mean nothing on their own. With clipping at 1.0 and Adam, only direction matters, and the
  Studio's stock path is on par with the Spark. Plain-ops attention in bf16 is worse and is out.
- **Multi-block check, k = 53 (5,085 tokens).** CPU truth: loss 1.1394, grad norm 536.4 (606 s,
  per-layer checkpointing on the CPU). Studio GPU bf16, cosine to truth: stock fused -0.056; fused
  per block 2,048 rows (what C2 runs) 0.141; fused per block 1,024 rows 0.176; plain ops 0.073.
  Everything is near noise at this length; the blocked fused path is no worse than stock, so C2 uses
  it (`blocked_attention.install(window=1024, block=2048)`, kernel "fused"). A Spark gradient at
  this length wasn't taken (my dump script lacked checkpointing; see above).

## C2, restarted from step 1 (2026-10-21 ~00:2x UTC)

Same command and inputs as the stopped run; the partial run is kept as `out-c2-partial-stopped/`.

## C1, Spark node 1 (started 2026-10-20 23:59:53 UTC)

`run_c1.sh`: Lab 018's trainer patched for shared ids, order and init, pinned image, watchdog stop
at 3 GB (Keith: run at the 5 to 6 GB margin Lab 018 trained at). Node 2 had no containers.

## Step 1, side by side (same four examples, same seed-42 init)

| | Lab 018 (2026-08-22) | C1, Spark node 1 | C2, Studio |
|---|---|---|---|
| Loss | 1.0080128908157349 | **1.0080128908157349** | 1.0182 |
| Grad norm | 34.43 | 34.21 | 10.35 |
| Step time | ~423 s (epoch-2 mean) | 419.6 s | 109.7 s (incl. compile) |
| Memory | | allocator peak 80.7 GB; 28.7 GB available at step end, 8 GB minimum so far | MLX peak 105.7 GB |

C1 reproduces Lab 018's step-1 loss bit for bit, so the shared order, the tokenization and the pinned
stack are Lab 018's. Its grad norm differs slightly because Lab 018's LoRA A init was unseeded and
C1's is the shared seed-42 init (at B = 0 the step-1 gradient depends on A). C2's step 1 repeated
exactly on restart (loss 1.0182, grad norm 10.35), so the Studio run is deterministic. The Studio's
loss is 1% higher and its grad norm a third of the Spark's; per the gradient check above, bf16
gradient size at B = 0 is mostly noise on both boxes, so the comparison that counts is the loss
curve over 86 steps and the held-out validation (E1).

## C1 and C2, full runs (finished 2026-10-21)

Both ran the full 86 steps, rc 0. C1 finished at 10:21 UTC with no watchdog stop. C2 finished at 02:51 UTC
with no swap growth. Per-step logs, power and Lab 018's trainer state are in
`finetune-model-first-tools/results/`.

| | C1, Spark node 1 (PyTorch, Lab 018 stack) | C2, Studio (mlx-lm, blocked attention) |
|---|---|---|
| Training time, 86 steps | 36,885 s (10.25 h) | 9,670 s (2.69 h) |
| Median s/step, steps 5 to 86 | 377.5 | 109.6 |
| Throughput, 2,962,798 tokens | 80 tokens/s | 306 tokens/s |
| Peak memory | allocator 102.4 GB; minimum available 5.7 GB of 121 | MLX 144.0 GB; process footprint 231 GB of 256 |
| GPU-rail power, median | 65.6 W (nvidia-smi) | 135.5 W (IOReport) |
| GPU-rail energy, whole run | ~656 Wh (5 s samples, includes load) | 367 Wh |
| Mean train loss, all / epoch 1 / epoch 2 | 0.2135 / 0.2947 / 0.1324 | 0.2190 / 0.3020 / 0.1360 |
| Mean of last 10 steps | 0.1248 | 0.1296 |
| Median grad norm | 3.73 | 5.59 |

Lab 018's own run on the same order: mean 0.2189, epoch 2 0.1368, last 10 0.1338.

- **R = 3.8 overall** (36,885 / 9,670). The median-step ratio is 3.45, and the per-step ratio ranges
  from 2.5 to 5.4, varying with the step's lengths.
- **The curves track.** The mean per-step |C1 - C2| loss gap is 0.0093. The mean |C1 - Lab 018| gap is
  0.0104 (same order, different init). So the Studio tracks the Spark about as closely as the Spark
  tracks Lab 018's own run.
- **Against the pre-registered readings** (rough, mine; Keith rules). The small job had R ~6. On this
  job R ~3.8, with M ~1.2 and W ~4.6. R moved toward M but stays far above it.
- **Rough achieved compute** (6 x ~30.7B language-model parameters x tokens, including the
  checkpoint recompute but not attention): about 5.5e17 FLOPs. That's ~15 TFLOPS on the Spark (15% of
  its A0 ceiling) and ~57 on the Studio (47% of its ceiling). Neither box is near its compute ceiling,
  and the Spark is much further from it. That points at the stack more than the silicon, which by the
  pre-registration makes the faster Spark software arm (D) the next question. Not run; Keith's call.
- **The Studio number is for the blocked-attention path,** not mlx-lm as shipped (see the spec's
  "As run").

## Forge arm: not run, and why that's the finding (2026-10-21)

Keith asked to run the job on CoreWeave Forge (Serverless SFT) to compare a managed service's effort
against the work C1 and C2 needed. It can't be run as specified. **Forge doesn't offer the dense
Gemma 4 31B.** CoreWeave's available-models page
(docs.coreweave.com/training/serverless-rl/available-models, read 2026-10-21) lists five base models
for Serverless Training: Gemma 4 26B-A4B (rank 1, fixed), Llama 3.1 8B (rank 8), Nemotron 3.5
Lightning 30B-A3B (rank 8), Qwen3.8 27B and Qwen3.6 35B-A3B. LoRA rank and alpha are set per model
and can't be changed. Anything else goes through forge-support@coreweave.com. (Lab 024's notes say
the key listed 13 trainable models; I couldn't reproduce a listing from the ART client or the obvious
endpoints, so the docs page is the record here.)

Keith's ruling: the point was the dense model, so a 26B-A4B substitute isn't run. **A managed
service is only as good as its model selection.** Here the job is the one Lab 018 and this comparison
already ran on owned hardware, and the managed path stops at the menu before any question of effort
or speed comes up. Owned hardware took real work to get there (re-pinning transformers, the load
transient, blocked attention on the Studio), but it could take the model the work called for.

What would also have needed handling on Forge, had the model been offered: ART renders tool calls
through the chat template's history format, the dialect Lab 018's first tune learned and couldn't
execute (workaround: send native-emission text as assistant content); a fixed rank in place of
Lab 018's r32; and evaluating off-Forge, since in Lab 024 Forge's endpoint served the base model for
its own adapter. Nothing was sent to Forge and nothing was spent.

## E1: held-out validation (finished 2026-10-21)

21 held-out examples (6 tasks Lab 018 held out), 11,458 labeled tokens. Scored in one runtime (Studio,
mlx-lm bf16, blocked attention, dropout off). Lab 018's loss: uncapped, chunked, per-example mean,
averaged. PEFT adapters translated in memory by transpose (`e1_val.py`, results `e1_val.json`).

| Adapter | Val loss | Next-token match | Same prediction as C1 |
|---|---|---|---|
| Base (no adapter) | 1.0484 | 92.07% | 93.59% |
| C1, Spark | 0.1536 | 94.75% | 100% |
| Lab 018's original (Spark, Aug 2026) | 0.1534 | 94.75% | 98.36% |
| C2, Studio | 0.1510 | 94.81% | 98.31% |

The Studio adapter is as good as the Spark adapter on held-out data, and agrees with it about as
often as two Spark-trained adapters agree with each other. Lab 018's 22-task loop eval (its tune
scored 14 against the base model's 9) wasn't rerun on either new adapter.

## Lab entry

Scaffolded 2026-10-21 as Lab 027 (`finetune-model-first`, status proposed). The ruling question,
in Keith's words: model selection drives the hardware, and a managed service is only as good as its
model selection. Spec, notes and tools renamed to the slug. Lab detail staged at
`specs/finetune-model-first-lab-detail.md` until publish. When it publishes, Lab 026's first
`opened` answer ("That's running now as the next lab") should be updated to name Lab 027.

## Keith's framing, 2026-10-21: the comparison is at the contract layer

On a proposed Studio serving arm (C1 and C2 served by mlx-lm on the Studio against vLLM on the
Spark), I flagged that runtime differs along with hardware. Keith: "That caveat isn't a confound
but a feature. At this level we aren't comparing straight technical features but consuming services
at the contract layer. Which runtime is a technical implementation detail below the value line."

So the venues are compared as services: does the venue take the model, train it, serve it in the
agent loop, and at what time, margin and outcome. The runtime (vLLM or mlx-lm, stock or rewritten
attention) is how each venue meets the contract, recorded in the detail, not a confound to remove.
This also bears on the peer-check findings that called the 3.8x a stack comparison: at this layer
the stack is part of the venue. What still has to be said plainly is what it took to meet the
contract (the attention rewrite, the pinned release), since that's the cost of the venue.

## E1 reverse runtime check (Spark, PyTorch, 2026-10-21)

`spark_e1.py`, pinned image, same metric as `e1_val.py`. Val loss / next-token match / agreement
with C1: base 1.0679 / 92.05% / 93.56%; C1 0.1510 / 94.78% / 100%; Lab 018 0.1534 / 94.70% / 98.49%;
C2 0.1509 / 94.77% / 98.37%. On the Studio runtime C1 was 0.1536 and C2 0.1510, so the C1-C2 gap is
runtime noise; the runtime doesn't change the order against Lab 018's adapter or the base.

## E2: Lab 018's 22-task loop eval (started 2026-10-21)

Lab 018's published bar: its tuned adapter solved 14 of 22 in the loop against a fair untimed loop
base of 12 (`p7_loop_tuned_think` on node 2, `p7_loop_base31_think` on node 1, both re-counted from
the result files: 14 and 12). Its notes' earlier "9/22" was a timed base run; the published record
uses 12 and calls the two-task gap noise.

**Reference apparatus, recovered from the stopped `tuned-serve` containers** (`docker inspect`):
vLLM 0.25.1, base `google/gemma-4-31B-it` served as `base31`, runtime LoRA `tuned=/lora`,
`--max-lora-rank 32 --max-model-len 131072 --gpu-memory-utilization 0.85 --max-num-seqs 4
--enable-chunked-prefill --enable-prefix-caching`, port 8010, no reasoning parser. Loop:
`run_p7_loop_tuned.sh` (`tools/stateful.py`, `miss22.txt`, TURNS=2, VLLM_THINK=1,
NUM_PREDICT=49152, PAR=2). Runner, loop script and the 22 task files byte-identical on both nodes.

**Spark arms** (`run_e2.sh`: the reference container with only the `/lora` mount changed; the loop
script with only the results dir changed): C2 on node 2 from 14:15 UTC (`results/lab027_c2`), C1 on
node 1 from 14:40 (`results/lab027_c1`). C2 is served as PEFT: its 460 language-model tensors (exact
transpose of the mlx-lm weights) in C1's file, keeping C1's 216 untouched vision-tower tensors (B =
0) and its adapter_config (`build_c2_peft.py`). Adapter live: on one prompt at temperature 0,
`tuned` and `base31` gave the same text but different token logprobs.

**Studio arms** (Keith: the loop and its Docker sandbox run on the main Mac against the Studio through
an SSH tunnel, as Lab 026 did). Runner copied from node 2 to `~/StackBuilder/loopcontrolbench` on the
Mac (hashes match), venv from the node's frozen package list, sandbox image built from the same
Dockerfile. Studio: stock `mlx_lm.server` 0.32.2 bound to localhost, with a 10-line wrapper
(`mlx_serve_alias.py`) so the request model names `tuned` and `base31` resolve to the CLI model with
and without the adapter (the loop sends `model="tuned"` and can't change). Sampling defaults set to
Gemma's generation config (temperature 1.0, top-p 0.95, top-k 64), which vLLM applies by default.
Adapters: C2's own mlx-lm weights; C1 translated from PEFT by transpose (`peft_to_mlx_dir.py`).
Served copies set dropout to 0. Adapter live: logprobs differ from `base31`. **Contract difference,
recorded not removed** (Keith: runtime is below the value line): mlx-lm returns Gemma's thinking in a
separate `reasoning` field and the answer alone in `content`; Lab 018's vLLM, with no reasoning
parser, returns both in `content`. The loop reads `content`.

**One-task validation** on the Studio (click-pr3126): two turns, 14,965 worker tokens, unsolved; C2 on
node 2's vLLM also missed it (15,208 tokens); Lab 018's tuned solved it on turn 2. Session read: the
model reasoned about the right function in `src/click/shell_completion.py` and emitted one
well-formed SEARCH/REPLACE edit that the parser accepted. Not oMLX's broken Gemma 4 path. The Studio
C2 arm started after (`results/lab027_studio_c2`, under caffeinate); C1 follows on the Studio.

**Studio C2 arm, attempt 1: INVALID (my apparatus defect).** macOS's bash 3.2 runs the loop script's
job-control check in a subshell and has no `wait -n`, so the script launched all 22 tasks at once
instead of two. The mlx-lm server batched them and its generation thread died on a Metal
out-of-memory at 10:51 AM CDT; 17 of 22 tasks crashed (3 solved, 2 unsolved, all under a 22-way load
the reference never saw). Archived as `results/lab027_studio_c2_INVALID_bash32_no_wait_n`. The early
per-task timing I quoted from it is void. The Spark arms run bash 5 and aren't affected.
**Attempt 2** (10:57 AM CDT): server restarted; the loop's job control replaced by `run_par.py`, a
two-worker driver that runs the same `stateful.py` command per task, in the same order, with the same
environment, skip-if-done and crash records. Two tasks in flight, as in the reference.

**Studio C2 arm, attempt 2: server out of memory at the 8th task.** With two tasks in flight, the
mlx-lm server's generation thread died at 12:17 PM CDT on a Metal out-of-memory, 1.5 minutes into
decoding after a 46,886-token prefill. Its prompt cache held 10 finished conversations' KV caches,
50.8 GB: mlx-lm defaults to 10 entries with no byte cap, while vLLM's prefix cache lives inside its
fixed 0.85 memory budget and evicts. Tasks 1 to 7 finished under the correct two-task load before the
failure (3 solved, 4 unsolved) and are kept; the 15 that crashed are rerun (Lab 018's own base arm was
resumed the same way, `p7_base31_resume.log`). Server restarted with `--prompt-cache-bytes 32GB`
(reused KV is exact, so the cap changes reuse, not outputs), plus a 30-second memory sampler
(`memlog_e2.log`). Resume started 12:21 PM CDT. Attempt 2's full results are snapshotted in
`results/lab027_studio_c2_attempt2_snapshot`.

That this took two fixes on the Studio and none on the Spark is itself a venue observation at the
contract layer: vLLM's serving defaults held for this model and workload; mlx-lm's needed a cache cap
to survive it.

## Keith, 2026-10-21: the serving runtime is part of software-layer portability

On the mlx-lm prompt-cache failure: "This impacts portability of the software layer. I can take the
vLLM dependent workflow to any provider using vLLM as the underlying surface. While I'd be using the
same OpenAI API, I'd be using different configurations at the application layer to ensure proper
cache management."

The reading: the OpenAI-compatible API is the visible contract; the runtime's operating envelope (KV
cache budget and eviction, concurrency, memory limits) is a second contract the API doesn't show.
vLLM sets it through one budget (`--gpu-memory-utilization`); mlx-lm's server left the prompt cache
unbounded by bytes and needed `--prompt-cache-bytes` to survive this workload. A workflow written
against vLLM moves to any vLLM-based provider with its envelope; moving it to a different runtime
behind the same API takes operating changes. Lab 018 already found the tool-call parser is part of the
serving contract; this adds memory management. Caveat from Lab 018's own record: vLLM to vLLM isn't
free either (the served context cap and the parser had to be diffed against the reference), so the
portable unit is a runtime family plus its recorded envelope, not the API alone.

Candidate for the lab (Keith's call): a call or an opened question along the lines of "the
OpenAI-compatible endpoint isn't the portability layer; the runtime's operating envelope moves with
the workload". Layer placement to be decided with the DAPM reading (portability).

**Keith's placement ruling, 2026-10-21:** "The underlay in either case is ceded if you go a managed
route and retained if you are self serving." The serving runtime under the OpenAI-compatible API
(vLLM, mlx-lm, or a provider's stack) places by who controls its operating envelope, not by which
runtime it is: self-served = Retained; managed endpoint = Ceded, even behind the same API. For Lab
027's portability table: Spark vLLM and Studio mlx-lm serving rows are Retained; a managed
fine-tune-and-serve path (Forge) would be Ceded.

**Studio C2 arm, complete** (resume finished 1:28 PM CDT, 18:28 UTC): **12 of 22 solved**, 10
unsolved, 0 crashed. Solved: click-pr3504, dateutil-pr700, more-itertools-pr1117, more-itertools-pr944,
prettytable-pr344, prettytable-pr345, prettytable-pr445, requests-pr7205, requests-pr7308,
requests-pr7309, requests-pr7433, requests-pr7502. No server deaths after the cache cap; free memory
stayed at or above 46%, swap flat. Loop time about 2.5 hours across the valid part of attempt 2 and the
resume. **Studio C1 arm** started 1:31 PM CDT on the same server flags; C1 adapter live (first-token
logprob -0.0625 against C2's -0.1875 and base 0.0 on the probe prompt).

**Spark C2 arm (node 2), complete** 2:32 PM CDT (19:32 UTC), 5 h 17 min: **13 of 22 solved**, 9
unsolved, 0 crashed. Solved: click-pr3482, click-pr3504, dateutil-pr700, more-itertools-pr1117,
more-itertools-pr944, prettytable-pr344, prettytable-pr345, prettytable-pr445, requests-pr7205,
requests-pr7309, requests-pr7433, requests-pr7502, task-r007. Server stopped; the loop's `lcb-proxy`
container (started by the sandbox runner) stopped after, node 2 idle.

**Node 1 lagging node 2 (Keith).** Not the node: over each whole run vLLM averaged 6.2 generated
tokens/s on node 1 against 5.8 on node 2, same clocks (2,450 to 2,476 MHz), power (36 to 38 W) and no
throttle reasons. Node 1's C1 had solved 4 of 11 when node 2's C2 had solved 12 of 19; unsolved tasks
run both turns, so C1 spends more generation per task. One run each, sampled, so not yet a difference
between the adapters. Keith: which box trained an adapter can't be the cause ("math is math"); any
real difference is the trainer software (PyTorch/PEFT against MLX/mlx-lm: kernels, attention path,
dropout RNG, bf16 rounding order). Labels C1/C2 to be described by trainer stack, not box, in the
write-up. Test being set up: Lab 018's PyTorch stack on the Studio (MPS), same software as C1 on
different hardware.

**Studio C1 arm, complete** 3:17 PM CDT (20:17 UTC), 1 h 46 min: **14 of 22 solved**, 8 unsolved, 0
crashed, no server deaths, free memory at or above 43%. Solved: click-pr3482, click-pr3504,
dateutil-pr700, more-itertools-pr1117, more-itertools-pr944, prettytable-pr344, prettytable-pr345,
prettytable-pr445, requests-pr7205, requests-pr7308, requests-pr7309, requests-pr7433,
requests-pr7502, task-r007. **Correction to my earlier reading:** C1 ran faster than C2 on the Studio,
so "C1 reasons longer" doesn't hold; node 1's mid-run lag was task ordering, and its full-pool time
is converging on node 2's. Studio server stopped; same-stack MPS canary started.

**Same-stack test on the Studio (Lab 018's PyTorch stack on MPS), canary: infeasible as shipped.**
3:20 PM CDT. PyTorch 2.14.1 MPS, transformers 5.15.1, PEFT 0.18.0, C1's trainer with only the device
changed. One step on the 30,553-token example failed in 20 s in transformers' SDPA attention:
`NotImplementedError: MPS all_reduction: tensors requiring 64-bit indexing are not supported
(numel=29871545888)`, i.e. 32 heads x 30,553^2 scores materialized. Peak footprint 170 GB. MPS's
32-bit indexing caps that matrix at about 11,600 tokens; 46 of the 171 training examples are longer.
So on the Studio neither framework's stock attention trains this job (mlx-lm out of memory at 224 GB,
PyTorch MPS at the indexing limit); the Studio venue needs custom attention either way. Swap guard not
triggered. The hardware-against-software question it was meant to answer may be moot: all four E2 arms
landed at 12 to 14 of 22. Keith to rule.

## Has the community solved long-context attention training on Apple silicon? (Keith asked, 2026-10-21)

Searched 2026-10-21. **PyTorch MPS:** no. PyTorch 2.13 added FlexAttention on MPS, but a third-party
test (dev.to, M1 Max) found backward raises "not supported on MPS", and PyTorch's 2.14 blog lists
MPS-native backward as future work. Draw Things' Metal FlashAttention 2.0 has an experimental backward
in their app, not in PyTorch. **MLX upstream: partly.** ml-explore/mlx PR #4563 (merged 2026-09-29)
adds a tiled SDPA backward that never builds the score matrix (reported on an M5 Ultra: up to 24x less
memory, 97 GiB to 4 GiB at batch 4); an Unsloth test PR says it shipped in 0.32.4. **MLX community:**
IonDen/mlx-train-perf (MIT) has a Metal flash-attention forward and backward for training plus a
logit-free fused loss, but only causal masks, head_dim 64/96/128, Llama/Qwen families; it refuses
Gemma.

**Probe on the Studio, MLX 0.32.4** (`sdpa_vjp_probe.py`, gradient of fused SDPA w.r.t. q, k, v, peak
memory above inputs): full-attention shape with `"causal"` (head_dim 512, 4 KV heads) 1.97 GB at
8,192 tokens and 3.66 GB at 16,384 (linear: the tiled backward is in use); sliding-window shape with a
boolean mask (head_dim 256, 16 KV heads) 12.94 GB and 50.25 GB (about 4x per doubling: the full matrix
is still built). So in 0.32.4 the native backward covers string-causal masks and not explicit boolean
masks, and Gemma 4's 50 sliding-window layers fall back to the quadratic path. That's why stock mlx-lm
died at 224 GB. `blocked_attention.py` is effectively the workaround: full layers go through
"causal" (tiled backward), sliding layers through small per-block masks. Upstream, the missing piece is
a sliding-window (or arbitrary-mask) tiled backward; an issue with these numbers would be the useful
contribution (outbound, Keith's call).

**Existing MLX issues and PRs (checked 2026-10-21 with gh, ml-explore/mlx).** No issue covers the SDPA
backward falling back to the full matrix for boolean or sliding-window masks. **PR #4734** "Adding
sliding window attention" (open, jagrit06, MLX maintainer, opened 2026-10-20): adds `window_size` to
`fast::sdpa` with Metal vector and matrix kernels, lets the kernels take array masks with
causal/windowed at once, asks for API comments; its description and comments discuss the forward
kernels and don't say the tiled backward from #4563 will honor the window. An earlier
sliding-window kernel PR (#3552) was closed. Related: #4693 (open, logsumexp from fused SDPA, for
merging attention over KV chunks). **Issue #3860** (closed): fp32 matmul silently defaults to TF32-class
precision (`MLX_ENABLE_TF32=1`), undocumented; this explains the 7.9e-4 fp32 matmul error measured on
the Studio's GPU (`MLX_ENABLE_TF32=0` should give true fp32). Proposed contribution, Keith's call to
post: a comment on #4734 with the probe numbers and the Gemma 4 31B case, asking whether `window_size`
will reach the backward.

**Posted to ml-explore/mlx PR #4734** from Keith's account on his go, 2026-10-21, with the Claude disclosure: https://github.com/ml-explore/mlx/pull/4734#issuecomment-6325994922 (text in `finetune-model-first-tools/mlx-pr4734-comment.md`).

**Spark C1 arm (node 1), complete** 3:21 PM CDT (20:21 UTC), 5 h 41 min: **11 of 22 solved**, 11
unsolved, 0 crashed. Node 1's sandbox proxy stopped after; node 1 idle.

### E2 tally (Lab 018's 22-task loop, one sampled run per arm)

| Adapter (trainer stack) | Served on | Solved | Loop time |
|---|---|---|---|
| Lab 018 tuned (PyTorch/PEFT), Aug 2026 | Spark, vLLM | 14 | about 9 h |
| Lab 018 base (no adapter), Aug 2026 | Spark, vLLM | 12 | |
| C1 (PyTorch/PEFT, Lab 018 stack) | Spark node 1, vLLM | 11 | 5 h 41 min |
| C1 | Studio, mlx-lm | 14 | 1 h 46 min |
| C2 (MLX, blocked attention) | Spark node 2, vLLM | 13 | 5 h 17 min |
| C2 | Studio, mlx-lm | 12 | about 2.5 h of loop time (resumed) |

Every arm lands at 11 to 14 of 22: the band Lab 018 called noise (its tuned 14 against base 12). No
trainer stack or serving venue separates from the others, and none clears the base model by more than
two tasks. The same adapter moved by three tasks between venues (C1: 11 on vLLM, 14 on mlx-lm), which
is the run-to-run spread at one sample per arm. On loop time the Studio served each adapter in a third
to a half of the Spark's time, on this one run each.

**Step-1 loss on the Studio, 0.993 against 1.018 (peer-check finding 11).** 0.99324 is a plain forward
in eval mode (pre-flight); 1.0182 is the loss value_and_grad returns in training mode with per-layer
checkpointing, on the same four examples with B = 0 (so LoRA and dropout can't move it). MLX's forward
under autodiff differs from its plain forward: 0.003 to 0.012 even with the whole model in fp32 on a
1,625-token example (`fused_bisect.py`), and fp32 matmul on the GPU is TF32-class by default (MLX issue
#3860). 0.025 in bf16 on these examples is larger than that and isn't fully explained.