# Lab 017 raw lab detail: Beyond CUDA

Working notes behind the published entry. Everything measurable ships here: the
availability-ladder raw data, every bench table, the training configs, the friction
log including the author's own process mistakes, the cost ledger with the staging
split, and the full story of the quality gate that got overruled.

**What stays back.** The 290-example training set and the frozen 30-question eval
set are the practice's proprietary instrument, held back since Lab 002; this lab
reused both unchanged, which is exactly what makes it a replication. The trained
adapters stay local. Correspondence with Hot Aisle is paraphrased per house rule.

**Redactions.** VM IP addresses are replaced with labels (BOX-1, BOX-2). They were
ephemeral and are already recycled, but the habit is the point.

---

## The one question

Lab 002 proved the weights are the asset you keep, measured entirely on NVIDIA
silicon. If CUDA is a requirement, that verdict has a dependency nobody priced. If
CUDA is a default, the weights are portable one level deeper than Lab 002 claimed:
not just the artifact, but the workflow that produces it. Same base model, same
training set, same recipe, same frozen eval. Change only the silicon.

## The apparatus

- **Training venue:** Hot Aisle MI300X VMs. 1x (192GB HBM3, 8c Xeon 8462Y+, 224GB
  RAM, 12TB NVMe, $2.99/hr, 1-minute minimum) and 2x (26c, 448GB RAM, $5.98/hr,
  60-minute minimum). Ubuntu 24.04, ROCm 7.2.4, provisioned via REST API against
  published shelf inventory.
- **Serving venues:** the same MI300X class; an Apple M5 Max MacBook Pro (36GB,
  MLX 0.31.3, fp16); two NVIDIA DGX Sparks (GB10, 128GB each) clustered TP=2 over
  dual 200Gb RoCE with the pinned vllm 0.25.1 + ray 2.54.0 container from Lab 014.
- **Stack:** rocm/vllm:latest (stable) for Llama-family work; rocm/vllm-dev:nightly
  where noted; LLaMA-Factory 0.9.5 pip-installed in-container; DeepSpeed 0.19.5
  (ZeRO-3 + CPU offload) for the 70B.
- **Recipe (identical to Lab 002):** LoRA r16, target all modules, lr 1e-4, 3
  epochs, effective batch 8, bf16, cutoff 4096. 111 steps every time.
- **Weights provenance:** Llama 3.1 8B from the practice's own cache. 70B base from
  an ungated public mirror, config and safetensors index SHA-256 verified against
  the official repo's files before any training; tokenizer files taken from the
  official gated repo via the practice's authenticated box. The practice's tokens
  never touched rented hardware.

## The availability ladder (raw)

| Venue | AMD datacenter GPU | Account truth at probe time |
| --- | --- | --- |
| AWS | None. Only AMD GPU is g4ad (Radeon Pro V520, 2020 graphics part) | g4ad in 8/17 regions, spot placement score 3/10 everywhere; NVIDIA accelerators rent fine and the practice has used them |
| GCP | Catalog lists accelerators | Standing probe record: no accelerator provisioning delivered at any tier, including Xeon 6 CPU instances |
| Azure | MI300X priced in 17 regions ($48/hr od, $8.87/hr spot at the cheapest) | Offered to this subscription in 2 regions; MI300X family quota 0; spot quota 3 vCPUs; regional cap 10 vCPUs vs a 96 vCPU minimum box |
| Hot Aisle | MI300X, MI355X | Live inventory API: 3 VMs on shelf; provision POST returned a running VM in seconds |

Timeline, account-zero to silicon (all UTC, from the platform's own API): account
created 00:45:14; API key minted ~22:30 (the 22-hour gap was the author's
calendar, not a queue; the comped credit was already applied but self-serve
payment was live the whole time); provision POST 22:39:40; SSH ~22:42; first
inference token ~22:47. Active human time: roughly 15 minutes.

## Training (raw)

| Run | Hardware | Wall | Cost | Loss |
| --- | --- | --- | --- | --- |
| 8B LoRA (Lab 002 baseline) | 1x DGX Spark | 35 min | $0 marginal | 0.34 → 0.22 |
| 8B LoRA (this lab) | 1x MI300X | 5.8 min | $0.29 | 0.56 → 0.225, same 111 steps |
| 70B LoRA (this lab) | 2x MI300X, ZeRO-3 + CPU offload | 89.2 min | $8.89 | 0.512 → 0.118 |

70B attempts: (1) failed, tokenizer missing, the author killed the weight download
two files early; (2) failed, GPU OOM, plain ZeRO-3 left 120MB free on 192GB cards;
(3) clean with parameter and optimizer offload to the 448GB of system RAM.

## Serving (raw)

Gemma 4 31B dense bf16, the loop-control grief model (512-token streamed outputs):

| Venue | conc 1 | conc 8 | conc 32 | conc 64 |
| --- | --- | --- | --- | --- |
| 2x Spark TP=2 (Lab 014 recipe) | ~10 tok/s | | | |
| 1x MI300X, nightly image | 55.8 | 309 agg | 1,207 agg | 1,974 agg (~$0.42/M output tokens at $2.99/hr) |

Tuned 70B, identical weights and adapter everywhere:

| Venue / method | 1-stream | best aggregate |
| --- | --- | --- |
| 1x MI300X, merged | 25.7 | 1,182 (c=64) |
| 2x MI300X, merged, TP=2 | 42.3 | 1,774 (c=64) |
| 1x MI300X, runtime LoRA, CUDA graphs on | 22.9 | 177.8 (c=8) |
| 2x Spark, runtime LoRA, TP=2, eager forced | 3.2 | 12.1 (c=4); c=8 collapses to 0.7/stream |

Tuned 8B on the M5 Max (MLX, fp16, unquantized): 26.1 tok/s decode, 16.1GB peak.
The laptop outran Lab 002's Spark serving of the same weights (14 tok/s).

The Spark cluster note that matters: with runtime LoRA under default CUDA graph
capture, decode collapsed to 0.1 tok/s and the engine watchdog killed the run.
Forcing eager mode fixed it. The identical config ran untouched, graphs on, on the
stock ROCm container. The lag was on the NVIDIA stack this time.

## The quality gate, including its collapse

The original Lab 002 judge script is unrecoverable and its strict judge (Gemma 4
31B on Bedrock) is no longer offered there. The gate was rebuilt from the frozen
contract: Gemma 3 27B (Bedrock OpenAI-compatible endpoint) plus Gemini 2.5 Pro
(Vertex), temperature 0, rubric checks 2-4, accept only on agreement.

Calibration against Lab 002's archived tuned-model outputs: the published 70%
scored 27% under the rebuilt gate. The rebuilt gate reads "grounded" harder. All
comparisons below are same-instrument relative; the absolute 70% is not comparable
and is not used.

| Model | JSON valid | Accepted (rebuilt gate) | Quote fidelity (mechanical) |
| --- | --- | --- | --- |
| Spark-trained 8B (archived outputs) | 29/30 | 27% | |
| AMD-trained 8B, Mac-served | 30/30 | 33% | 99% of quotes in context (1 fabricated of 89) |
| AMD-trained 70B, MI300X-served | 30/30 | 83% | 100% (0 fabricated of 89) |

The 83% did not survive. The author read the outputs side by side and could not
perceive the claimed difference. A deterministic check followed: every evidence
quote string-matched (exact, then 4-gram fuzzy at 0.8) against the cached
retrieval contexts. Both models cite real quotes at parity. The judge pair was not
detecting fabrication; it was expressing a strictness preference about
claim-support. The 2.5x quality claim is withdrawn. Four instruments, four
verdicts on one set of outputs: lenient gate, saturated; strict gate, 70B by 2.5x;
mechanical verification, parity; human reading, parity.

What survives every instrument: training-venue parity (the AMD-trained 8B matches
the Spark-trained original), and 30/30 format reliability on both AMD-trained
models.

## Friction log (complete)

1. HF offline-cache resolution bug in the stable ROCm image's Python stack (calls
   the hub API despite a complete local cache). Fix: point the config at the
   snapshot path. Would bite identically on CUDA. ~5 minutes.
2. Stable rocm/vllm image did not know the gemma4 architecture; the nightly did.
   Day-N support lag, one image tag wide. (Lab 002 needed a special CUDA build for
   the same family; the lag is ecosystem-wide, CUDA got its build first.)
3. Stale apt index on a fresh VM blocked python3-venv until apt-get update. Venue
   image hygiene, minutes.
4. Author error: killed the 70B download after the weight shards but before the
   tokenizer files. One relaunch.
5. Author error: a training monitor pattern missed the final error form; a dead
   run sat 40 minutes (~$4) looking alive. Monitor coverage is a discipline, not
   a default.
6. Spark cluster: CUDA graph capture + runtime LoRA hang (above). Eager mode is
   the workaround at this stack version.
7. Root-owned lock files in a previously-dockered HF cache blocked host-side
   downloads. Fresh HF_HOME sidestepped it.

Nothing ROCm-specific broke at any point in the campaign. The list above is the
entire bill.

## Cost ledger (the staging finding)

Total: $29.75 GPU rental (from a disclosed $100 vendor credit) plus about $3 of
judge and embedding APIs (AWS, Google Cloud), self-funded.

| Session | Compute | Staging + debug | Ratio |
| --- | --- | --- | --- |
| 8B arm (BOX-1, first life) | ~$0.44 | ~$1.85 | 4:1 |
| Gemma bench (BOX-1, second life) | ~$0.10 | ~$3.40 | 34:1 |
| 70B train (BOX-2) | ~$9 | ~$4 | 0.4:1 (DC-speed pull) |
| 70B quality eval (BOX-1, third life) | ~$0.80 | ~$1.50 | 2:1 |

The meter buys bandwidth before it buys FLOPs. Recycled VMs keep nothing between
lives (no snapshot, volume, or cache service on the venue; each life re-pulled the
same 40GB serving image), and the transfer path prices the staging: the venue's
own network pulled public weights at ~106-220MB/s while home-network rsync ran
~35MB/s. Where a gated model forced the token-stays-home rule, the slow path was
the price of that rule, paid knowingly.

## Reproduce it

Base weights are public (gated and mirrored; verify the mirror against the
official index hashes as done here). The recipe is printed above in full. The
serving configs are stock container invocations with the flags listed. What you
cannot reproduce without building your own is the training set, the eval set, and
the judges; that is the practice's instrument, and after this lab's gate collapse,
building your own validator and then validating it is not a step this detail would
let you skip anyway.