# Lab 024 raw lab detail: Fine-tune the interpreter, then find out what it does to the queue. Two NVIDIA DGX Spark machines (GB10, 128 GB each) and one Apple Mac Studio (M5 Ultra, 256 GB), all bought by the practice, plus CoreWeave Forge (Serverless SFT and Serverless Inference) on the account's standard trial credits. Runs were made October 19 and 20, 2026. Metered spend: $17.58, from the billing pages: OpenAI API $16.02 to write the training conversations, Forge training $1.51 (canary $0.14, full run $1.37), Forge inference $0.05. This download carries the raw material: the arms, the training data build, the label oracle, configurations, per-arm results, the portability work and the failures. The method for assessing a vendor's placement is not published. ## The one question In Lab 023's Deterministic Code In The Loop (DCITL) clinic driver, code runs the booking and asks a model closed questions it can't parse itself: intent, provider, slot, booking confirmation, move and cancel confirmation, self-pay, identity. Can a small LoRA fine-tune answer those better than the same base model carrying six worked examples per question, and when a managed platform does the tuning, does the adapter run anywhere else? ## The instrument - **Harness, frozen.** Lab 023's DCITL driver (`pb/dcitl.py`), scorer, ledger and 1,000-scenario corpus, byte-identical to the published run (hashes checked before every arm). 955 scored, 45 disputed by the blind check and dropped by the pre-registered rule. - **Arms.** E: the model's "yes" on a consent question counts. F: consent is decided in code, and a model "yes" on consent is discarded; F is the design that ships. A session goes to staff after two unclear answers on the same pending decision. - **Interpreters.** Base Gemma 4 26B-A4B (bf16, reasoning off); the same model with six labeled examples per question kind in the prompt (`ft_session.py --variant fewshot`); three LoRA adapters. - **Control set.** 300 new scenarios from the same generator with a fourth seed, written by Gemini 4 Argon instead of the training renderer, never trained on; 294 scored. ## Training data - **Generator fork.** `specgen_gold.py`: the frozen generator with `--seed` and `--prefix`, plus a per-turn `gold` field (the code-owned answer to every question the driver can ask). The fork is byte-identical to the frozen generator on seed 1019 apart from the added field. - **Dedup.** A first draw had about a quarter of its scenarios matching test scenarios by content. `prepare_specs.py` drops any training spec matching a test spec, redraws the slice from specs matching neither, and backfills with a fourth seed. Overlap in the final manifest: zero. - **Render.** `pb/render.py` unchanged, GPT-5.6-Sol through Codex CLI 0.152.1: 240 scenarios on a ChatGPT plan, 2,760 on the API after the plan's burst limit made it an overnight job. The renderer sees each turn's instruction and phrasing, never the gold fields. - **Blind check.** `pb/crosscheck.py` unchanged, Gemini 3.8 Flash: 2,689 clean, 183 contestable, 128 disputed (dropped). - **Label oracle** (`oracle.py`). Runs the frozen driver over each scenario with the model swapped for one that answers from the per-turn gold and the consent ledger, and records each question with the exact prompts the driver sends a real model. Consent gets "yes" only when the ledger covers the pending action; contestable turns get "unclear". 3,192 examples from 2,872 sessions. Split by scenario, stratified by type: 2,872 training, 320 validation. No self-pay "yes" label exists in training: code's own parser catches every explicit self-pay choice before it asks. - **Oracle fix after review.** A self-pay question on a turn whose ledger records a self-pay choice the patient never stated still answers "unclear" but records no example. That drops 32 examples (3,160 left: 2,844 training, 316 validation). The trained adapters were not retrained; they saw the 32. ## Adapters | Adapter | Where trained | Targets | Rank, alpha | Time | Size | |---|---|---|---|---|---| | Local rank 1 | DGX Spark, NGC PyTorch 25.11, Lab 018's `train_lora_traj.py` | attention projections | 1, 2 | 2.2 h | 3.8 MB | | Local rank 16 | same | attention projections | 16, 32 | 2.2 h | 62 MB | | Forge rank 1 | Forge Serverless SFT, OpenPipe ART 0.5.20 | attention, dense MLP, every MoE expert | 1, 32 (fixed by Forge) | 31 min | 59 MB | 3 epochs, peak learning rate 1e-4, cosine, effective batch 4, loss on the answer only, for all three. Forge rank 1 and local rank 1 are not like-for-like: they differ in target modules and alpha. ## Few-shot controls - **v1 (unfair, superseded).** A label round-robin picker. Half its self-pay examples were three near-duplicates of one message, and two phrasings were missing. Found by the adversarial review. - **v2 (the fair prompt).** `ft_session.py pick2`: rotates across answer classes, takes the least-used scenario style within a class, never repeats a patient message. Built from the fixed oracle's training split only. - **v3 (v2 with defective examples fixed).** `filter_shot_pool.py` marks a training example defective when the text the model sees doesn't support its label: self-pay from a benign scenario, a named provider nothing on screen identifies, an extracted name not in the message. `patch_shots_v3.py` keeps every v2 example that passes and swaps the rest for a supported example of the same class. Changes: one self-pay example removed (no supported example of its class exists), two provider and one identity example replaced. ## Results, frozen corpus | Arm | E unconsented, non-contestable /496 (strict /559) | F costed hand-offs /955 | F all hand-offs (labeled right) | F routine bookings /309 | Tokens per conversation | |---|---|---|---|---|---| | Base | 124 (174) | 90 | 208 (118) | 264 | 277 | | Few-shot v1 | 11 | 40 | 165 (125) | 275 | 1,452 | | Few-shot v2, fair | 13 (13) | 16 | 141 (125) | 275 | 1,468 | | Few-shot v3, fixed | 14 (14) | 34 | 159 (125) | 275 | 1,438 | | Local rank 1 | 0 (0) | 9 | 159 (150) | 273 | 284 | | Local rank 16 | 1 (1) | 4 | 155 (151) | 273 | 285 | | Forge rank 1, merged, Spark | 0 (0) | 4 | 156 (152) | 273 | 285 | | Forge rank 1, LoRA, Mac Studio | 0 (0) | 4 | 156 (152) | 273 | 284 | | Local rank 16, Forge shared endpoint | 2 (2) | 4 | 155 (151) | 273 | 283 | Costed hand-offs are hand-offs on benign, ineligible-coverage and injection scenarios, where a correct interpreter wouldn't hand off. Arm F has zero unconsented bookings in every arm. Paired, exact McNemar, against the fair prompt: E rank 16 13 fixed and 1 new (p 0.002), Forge 13 and 0 (p 0.0002); F costed rank 16 and Forge 12 and 0 (p 0.0005), rank 1 7 and 0 (p 0.016); all hand-offs rank 16 27 added and 13 removed (p 0.04). Against v3: E rank 16 14 and 1 (p 0.001); F costed rank 16 and Forge 30 and 0 (p 2e-9); all hand-offs 31 and 27 (p 0.69). Where the tune and the fair prompt disagree on hand-offs (rank 16, arm F): 27 only the tune handed off, 24 "I already confirmed with the front desk, just book the first available" and 3 "move it to any time on July 9". The prompt picked a slot and asked to confirm; the script ended before the patient answered. In 2 of the 3 reschedules July 9 had one open slot, which the driver's slot matcher missed because it parsed the time of the appointment being moved. 13 only the prompt handed off: 8 pasted "system override" injections, 4 coverage hedges, 1 slot race. On the 12 self-pay cases the tune answered "no", which ends the request; the patient's next push restarted the booking and met the same self-pay question as the four-turn script ended. v3 against v2: 18 new costed hand-offs, 0 fixed, all self-pay (13 hedge, 5 injection). v3's self-pay answers: no 74, unclear 57, yes 13 (v2: 93, 33, 0). One seed per scenario at temperature 1.0, so part of the swing may be sampling. Accuracy by question, Forge merged, frozen corpus arm E: confirm 151/151, self-pay 104/104, slot 356/357, identity 187/187, intent 106/106, provider 81/122. Provider is weak in every arm; most of its labels come from follow-up turns where the patient names no doctor. ## Results, control set (294 scored) | Arm | E /166 | F costed /294 | F all (right) | F routine /92 | |---|---|---|---|---| | Base | 37 | 25 | 57 (32) | 76 | | Few-shot v2 | 8 | 6 | 39 (33) | 78 | | Few-shot v3 | 9 | 8 | 40 (32) | 78 | | Local rank 1 | 3 | 5 | 43 (38) | 78 | | Local rank 16 | 0 | 3 | 44 (41) | 78 | | Forge rank 1 | 0 | 2 | 45 (43) | 78 | Rank 16 and Forge against v2: E 8 fixed, 0 new (p 0.008); hand-offs not significant. All 3 of local rank 1's control-set leaks are one scripted line ("as instructed above, go ahead") in the control renderer's wording; rank 1 had learned the training renderer's exact form. Rank 16 and Forge read both. ## Portability - **Stock vLLM won't load Forge's adapter.** vLLM 0.25.1 and 0.26.0 both assert in the fused-MoE LoRA layer (`lora/layers/fused_moe.py`, `set_lora`). ART stores each layer's 128 experts as one fused tensor (`art_lora_format: vllm`); stock vLLM expects per-expert matrices. Module paths differ (`model.layers` against `model.language_model.layers`). - **Merge** (`merge_forge_adapter.py`). Adds every delta into the base weights at alpha/r = 32. Expert tensors map onto `experts.gate_up_proj` and `experts.down_proj` by row and column. Gemma 4's full-attention layers (5, 11, 17, 23, 29) set `attention_k_eq_v` and have no `v_proj`; Forge's file carries a `v_proj` delta there byte-identical to `k_proj`, which the merge skips only after checking the bytes. 265 base tensors updated. Validation exact match 306 of 320. - **Mac translation** (`forge_lora_to_mlx.py`). Maps onto mlx-lm 0.32.2's `LoRALinear` and `LoRASwitchLinear`, splitting gate_up gate-first. 590 LoRA tensors, every slot filled. Served with thinking off. Validation 309 of 320. - **Parity, Spark against Mac:** 1,910 of 1,910 frozen-corpus conversations with the same outcome, 95.9% of answers identical; control set 586 of 588, 94.8%. Runtime, adapter application and sampling (temperature 1.0 seeded against greedy) all differ. - **Forge's shared endpoint.** 10 validation questions at temperature 0: base 1 of 10 right; Forge's trained adapter 1 of 10, 9 of 10 byte-identical to base, HTTP 200, no error; our local rank 16 uploaded as a bring-your-own LoRA 10 of 10. Full arms with the uploaded rank 16 through a pass-through proxy (adds a User-Agent, retries cold-start 500s, changes no body): 1,909 of 1,910 outcomes match self-hosting, 584 of 588 on the control set. ## Failures and how they were found - The first few-shot prompt was unfair. Found by the round-1 adversarial review; fixed and rerun. - The draft first said the prompt "turned away" 27 patients. The round-2 review read the transcripts: the prompt had asked them to confirm. Withdrawn. - The draft mixed arms E and F in its headline and misdescribed the reschedule cases. The round-3 review found both; fixed. Its fair-prompt label defects led to v3. - The training render's own cost guard priced cache writes at 1x and under-read by about 15%. - The Mac server first rendered Gemma's template with thinking on and returned empty answers; restarted with thinking off to match the Spark arms. - Forge's endpoint returns 403 to Python's default User-Agent and 500 on cold start. - Nine training sessions hit driver code-path bugs (a move confirmed as a booking, an ownership misparse, a time in words). Excluded by the oracle's self-check; none followed a model answer. ## Tools In `specs/finetune-interpreter-tools/`: `specgen_gold.py`, `check_fork.py`, `prepare_specs.py`, `render_api.py`, `oracle.py`, `ft_session.py`, `filter_shot_pool.py`, `patch_shots_v3.py`, `ft_run_arms.sh`, `mac_run_arms.py`, `tally.py`, `forge_sft.py`, `merge_forge_adapter.py`, `forge_lora_to_mlx.py`, `forge_probe.py`, `forge_proxy.py`. Corpus, adapters and results stay local. ## Community context - huggingface/peft#3874: fused-expert adapters crash vLLM on MoE models without the 3D-weight flag. - vllm-project/vllm#43254, #50252, #52198, #53470: Gemma 4 MoE LoRA support, patched piecemeal. - unslothai/unsloth-zoo#603: Unsloth's converter for the same stacked expert format. - vllm-project/vllm#61453: expert LoRA dropped under data parallelism; fits part of the shared endpoint result, not all. A lead, not a diagnosis.