{ "slug": "beyond-cuda", "lab_number": 17, "title": "Beyond CUDA: the lock was never the silicon", "type": "editorial", "status": "published", "sponsor": null, "author": "Keith Townsend", "date": "August 11, 2026", "date_iso": "2026-08-11", "layers": [ "layer0", "layer2b", "layer2c" ], "axes": [ { "instrument": "4plus1", "layers": [ "layer0", "layer2b", "layer2c" ], "has_dapm_table": true } ], "supersedes": [], "superseded_by": null, "corrects": [], "corrected_by": [], "vendors": [ { "key": "hotaisle", "role": "cloud" }, { "key": "amd", "role": "hw" }, { "key": "apple", "role": "hw" }, { "key": "nvidia", "role": "hw" }, { "key": "microsoft", "role": "ref" } ], "themes": [ "model-ownership", "ai-factory-economics", "serving-economics", "validator-authority" ], "finding": "Lab 002 rerun off NVIDIA end to end: the workflow is portable and CUDA is a default, not a requirement; what actually gates the path is who will sell you the silicon, and whether your acceptance gate measures what you think it does.", "question": "Lab 002 ruled that the weights are the asset you keep, and every step of that ruling ran on NVIDIA: the box trained, the box served, and the cloud comparison was NVIDIA-backed. So the verdict carried an untested assumption. Does the workflow that produces and serves owned weights actually require CUDA, or is CUDA just where everyone happens to be standing? This lab reran the Lab 002 fine-tune on a rented AMD MI300X, brought the weights home to Apple silicon, and held everything else constant: same base model, same training set, same frozen thirty questions, same recipe down to the learning rate.", "load": "The Lab 002 fine-tune and serving workload, rerun with the silicon as the only variable: Low-Rank Adaptation (LoRA) training on rented AMD Instinct MI300X, serving on an owned Mac and on the rented card, quality judged under one rebuilt gate.", "verdict": { "scope": "The workflow is portable. The same configuration file, the same trainer, the same serving container ran on AMD silicon with zero code changes, trained the same LoRA six times faster than the owned box, and the tuned model held quality parity under every instrument that survived scrutiny. CUDA is a default, not a requirement. What actually gates the non-CUDA path is two things this lab measured: who will sell you the silicon at all, and whether your acceptance gate is measuring what you think it measures.", "independence": "Self-funded editorial with one comped input, disclosed: Hot Aisle’s CEO credited $100 to the account after it was opened, and the lab spent $29.75 of it. The credit funded the compute, not the answer. Hot Aisle did not commission, review, or see this lab before publication, had no say in what ran or shipped, and self-serve payment was available the whole time, so the credit never gated access. Correspondence with Hot Aisle is paraphrased where it appears. Disclosure: Google Cloud is a client of The CTO Advisor LLC, and it appears here twice: it failed the availability probe this lab reports, and it supplied one of the two judges (Gemini 2.5 Pro on Vertex AI, part of about $3 in self-funded API spend alongside AWS). Hot Aisle, AMD, Apple, NVIDIA, AWS, and Microsoft are not clients. The ruling is the author’s alone.", "calls": [ { "kind": "do", "text": "rent the training run. The 8B fine-tune cost 29 cents and 5.8 minutes on a rented MI300X; the 70B cost $8.89. The weights come home either way. If you own the training set and the recipe, training compute is an errand, not a capital decision." }, { "kind": "dont", "text": "infer capacity from a catalog. AWS rents NVIDIA accelerators fine, and this practice has rented L40S there; what it rents is no AMD datacenter-class part at all. Azure prices the MI300X in 17 regions and offered this account 2, behind a quota of zero. GCP would not provision anything with an accelerator attached. The neocloud published shelf inventory in an API response and delivered a running VM in seconds. For silicon off the CUDA path, access is a product surface, and the hyperscalers do not sell it." }, { "kind": "do", "text": "treat the mainstream stack as already portable. PyTorch, the trainer, and vLLM installed clean on ROCm every time. The one real lag was day-zero support for a brand-new architecture, and NVIDIA felt a version of that same lag inside this lab too." }, { "kind": "dont", "text": "let one judge gate pick your model tier. Four instruments scored the same outputs and returned four verdicts. The strict judge pair said the 70B beats the 8B by 2.5x; mechanical quote verification and human reading said parity. If your gate is a model judge, you are buying the judge’s taste." } ] }, "objection": { "q": "But nobody got fired for buying NVIDIA, and the software moat is real.", "a": [ "It is, at the frontier. Lab 002 hit a four-wall dependency cascade trying to fine-tune a brand-new architecture on CUDA, and this lab hit the same species of wall on ROCm: the stable container did not know the newest model family, and the fix was a nightly image. The moat exists where the weights are newest.", "However, for the workload that pays the bills, a Llama-family LoRA on the standard stack, the moat has been abstracted away. The trainer pip-installed clean. The serving container was stock. Nothing in the path knew or cared that CUDA was not underneath it. And the two-Spark cluster needed a workaround this lab’s AMD boxes did not: the graph-capture path hung under runtime LoRA on the NVIDIA stack while the ROCm container ran the identical configuration untouched. Day-N polish is workload-specific. Neither vendor owns it." ] }, "numbers": [ { "label": "Account-zero to MI300X shell", "value": "~15 active minutes", "note": "provision POST to first inference token: ~8 min, image-pull bound" }, { "label": "8B LoRA fine-tune", "value": "5.8 min · $0.29", "note": "identical recipe, same 111 steps; Spark baseline was 35 min" }, { "label": "70B LoRA fine-tune", "value": "89 min · $8.89", "note": "2x MI300X, ZeRO-3 with CPU offload; the tier that was relationship-gated out of Lab 002" }, { "label": "Gemma 4 31B dense, one MI300X", "value": "55.8 tok/s single-stream", "note": "10 tok/s on 2x clustered Sparks; 1,974 tok/s aggregate at 64 streams ≈ $0.42/M output tokens" }, { "label": "Tuned 70B, same weights, three venues", "value": "25.7 · 42.3 · 3.2 tok/s", "note": "one MI300X merged · two MI300X · two Sparks; rented single card beats the owned cluster 7.2x" }, { "label": "Quality, one rebuilt gate", "value": "AMD-8B 33% vs Spark-8B 27%", "note": "training-venue parity; 30/30 valid JSON on both AMD-trained models" }, { "label": "Quote fidelity (mechanical)", "value": "8B 99% · 70B 100%", "note": "the strict gate’s 2.5x claim for the 70B is withdrawn: judge taste, not fabrication" }, { "label": "Whole-lab cost", "value": "$29.75 GPU + ~$3 APIs", "note": "staging exceeded compute roughly 2:1 across the campaign" } ], "opened": [ { "q": "Who validates the validator?", "a": "Four instruments disagreed about a 2.5x quality gap on the same thirty outputs. Before an acceptance gate picks your model tier, something has to pick the gate, and nothing in the current tooling does that job. This lab leaves that question standing." }, { "q": "Does the day-zero support asymmetry actually close?", "a": "The newest model family needed a nightly image on ROCm and a special build on CUDA. One image tag of distance, measured once. Whether that gap holds, widens, or closes at the next major model drop is measurable the day it happens." }, { "q": "Does the fine-tune survive quantization?", "a": "The designated 16GB fair-venue Mac serves this model only quantized. The unquantized arm ran on a larger machine, so the behavior of these weights under 8-bit serving is unmeasured here." } ], "not_proved": [ "Mainstream-recipe portability only. The bleeding edge, newest architectures and exotic quantization, was not retested on ROCm beyond the one image-tag lag this lab hit.", "Thirty questions, one task, one domain. Quality parity is directional, not a benchmark.", "The original Lab 002 gate is unreproducible; absolute comparison to the published 70% is broken and replaced by same-instrument relative comparison, disclosed above.", "The 70B’s 83% under the strict gate is reported and withdrawn as a quality claim; it measures judge behavior, not verified grounding.", "One neocloud tested. Hot Aisle’s access model may not generalize across the tier.", "The two-Spark 3.2 tok/s is a floor of this stack version, not a ceiling; the graph-capture workaround may be fixed upstream." ], "executive_summary": [ "Lab 002 proved the weights are the asset, entirely on NVIDIA silicon. This lab moved every stage of that workflow off CUDA and measured what actually changed. Training: the identical LoRA recipe ran six times faster on a rented MI300X than on the owned Spark, for 29 cents. Serving: the tuned weights came home to a Mac and outran the original box. Quality: parity with the Spark-trained original under every instrument that survived scrutiny. The silicon was never the lock.", "The two real findings sit either side of the compute. Below it: access. Three hyperscalers could not or would not rent the AMD part this lab needed, while a neocloud published its shelf inventory in an API and delivered in seconds. Above it: validation. The rebuilt judge gate claimed the 70B beat the 8B by 2.5x, and the claim did not survive a human read or a deterministic quote check. Which model tier pays off turned out to be a property of the validator, not the model." ], "dapm": [ { "instrument": "4plus1", "layer": "layer0", "placement": "Delegated", "note": "Training and speed-tier serving rented per-minute against published shelf inventory. The Sparks and the Mac retain the floor: the weights run at home on nobody’s quota." }, { "instrument": "4plus1", "layer": "layer2b", "placement": "Retained", "note": "Same artifact served on three vendors’ silicon; venue chosen per workload, not per platform commitment. Venue fluidity is what retention looks like now." }, { "instrument": "4plus1", "layer": "layer2c", "placement": "Retained", "note": "The weights come home, and the layer splits: the acceptance gate is revealed as its own authority surface. Whoever defines the validator defines which model tier wins." } ], "seam_map": null, "writeup": [ "The availability ladder answered Lab 002’s own open question first. Does a neocloud actually deliver access? Four venues, one afternoon. GCP: no accelerator provisioning at any tier, including a CPU. AWS: rents NVIDIA accelerators fine, and this practice has rented L40S there, but its only AMD GPU is a 2020 graphics part, and an L40S is not an answer to a 192GB high-bandwidth-memory question. Azure: the only hyperscaler that sells the MI300X, in 2 of its 17 priced regions for this account, family quota zero, spot quota 3 virtual CPUs against a 96-CPU box. The scope of that ladder is AMD Instinct-class silicon, not accelerators generally. The wall is specific to leaving CUDA. Hot Aisle: a GET request returned shelf quantity, a POST returned a running VM, and the first inference token came 8 minutes after the provision call.", "Training moved without friction that mattered. Identical recipe, identical data. The MI300X ran Lab 002’s 35-minute fine-tune in 5.8 minutes for 29 cents, converging to the same loss over the same 111 steps. The 70B, the tier that was relationship-gated out of existence in June, trained in 89 minutes for $8.89 across two cards. The frictions worth logging were a stale package index, an offline-cache bug in the image’s Python stack, and the author’s own process mistakes. Nothing ROCm-specific broke, at any point, in the whole campaign.", "Serving came home. The merged 8B served unquantized on the Mac at 26 tokens per second: the laptop outran the Spark serving the same weights in Lab 002. The dense 31B that crawls at 10 tokens per second across two clustered Sparks ran 5.6x faster on one rented card, and at 64 concurrent streams the card delivered commodity-token economics on owned weights, about 42 cents per million output tokens. Lab 002 found that custom weights cost $27.50 per million on the managed floor. That floor was never physics. It was packaging.", "The dual-card question answered itself. A 70B fits whole on one 192GB card; the second card bought 1.6x single-stream speed for 2x cost, a latency purchase, not an efficiency one. And the owned two-Spark cluster, serving the same tuned 70B, delivered 3.2 tokens per second. The rented single card beat the owned cluster 7.2x. Owned compute at this tier is a sovereignty floor, not a serving tier. The weights run at home on nobody’s quota; when speed matters, you rent the hour.", "The gate ate the headline. The original judge script was lost and its judge model retired, so the gate was rebuilt from the frozen contract and calibrated against Lab 002’s archived outputs. The rebuilt gate read grounded more strictly: the archived 70% baseline scored 27% under it. Fine. Re-baseline, compare relatively. The AMD-trained 8B matched the Spark-trained original within noise, so the training venue cost nothing. Then the strict gate said the 70B wins by 2.5x, and that number survived exactly until a human read the outputs. The wording difference is not perceptible. Mechanical quote verification agreed with the human: both models cite real quotes at parity, 99 and 100 percent, one fabricated citation out of 178. The judge pair was not detecting fabrication. It was expressing taste.", "Four instruments, four verdicts, one set of outputs. The lenient original gate said the 8B was saturated. The strict rebuilt gate said the 70B wins by 2.5x. Deterministic verification said parity. Human reading said parity. So the question an Enterprise Architect actually asks, does the bigger model pay off, has no answer until you answer a prior question: measured by what? The validator is the control point, and the validator itself requires validation. That is the campaign’s standing thesis arriving from the measurement side, and it is the finding this lab keeps." ], "abstraction": [], "method": [ "Self-funded editorial, no sponsor; the comped Hot Aisle credit is disclosed in the independence line. Every measurement ran under one frozen contract: same training set, same eval set, same retrieval index, same prompts as Lab 002. The only variable was silicon.", "Training was Low-Rank Adaptation (LoRA) via the same trainer and recipe as Lab 002, on rented MI300X capacity. Serving was measured on the rented cards, an owned Mac (MLX, fp16), and the owned two-Spark cluster, with the same bench harness throughout. The judge gate was rebuilt from the frozen contract after the original was lost, calibrated against archived outputs, and then itself became a finding when mechanical verification and human reading overruled it.", "The raw detail ships at the download link: the availability-ladder data, every bench table, the training configs, the complete friction log including the author’s own mistakes, and the cost ledger with the staging split. The training set, eval set, adapters, and judges stay with the practice." ], "assessments": [ { "instrument": "4plus1", "vendor": "AMD Instinct, EPYC & Enterprise AI Suite", "layer": "Layer 0 · Compute", "grade": "The Second Source — Silicon Across All Three Sub-Layers", "assessed_date": "August 10, 2026", "version": "v1.0 - Initial Assessment", "source_url": "https://layer2c.com/assessment/amd", "snapshot_date": "2026-08-11" }, { "instrument": "4plus1", "vendor": "AMD Instinct, EPYC & Enterprise AI Suite", "layer": "Layer 2B · Runtime", "grade": "Serving Catalogue on a Standard Endpoint, No Distributed Inference Plane", "assessed_date": "August 10, 2026", "version": "v1.0 - Initial Assessment", "source_url": "https://layer2c.com/assessment/amd", "snapshot_date": "2026-08-11" }, { "instrument": "4plus1", "vendor": "AMD Instinct, EPYC & Enterprise AI Suite", "layer": "Layer 2C · Reasoning", "grade": "Structurally Out of Reach Without a Data Layer", "assessed_date": "August 10, 2026", "version": "v1.0 - Initial Assessment", "source_url": "https://layer2c.com/assessment/amd", "snapshot_date": "2026-08-11" }, { "instrument": "4plus1", "vendor": "NVIDIA AI Platform", "layer": "Layer 0 · Compute", "grade": "NVIDIA Strength — Silicon Authority", "assessed_date": "July 31, 2026", "version": "v1.5 - Idle-Silicon Economics Reconciliation", "source_url": "https://layer2c.com/assessment/nvidia", "snapshot_date": "2026-08-11" }, { "instrument": "4plus1", "vendor": "NVIDIA AI Platform", "layer": "Layer 2B · Runtime", "grade": "NVIDIA Authority — Inference + Agent Runtime", "assessed_date": "July 31, 2026", "version": "v1.5 - Idle-Silicon Economics Reconciliation", "source_url": "https://layer2c.com/assessment/nvidia", "snapshot_date": "2026-08-11" } ], "publisher": "The Advisor Bench LLC", "program": "Layer2C Labs", "url": "https://labs.layer2c.com/labs/beyond-cuda", "markdown_url": "https://labs.layer2c.com/labs/beyond-cuda.md", "json_url": "https://labs.layer2c.com/labs/beyond-cuda.json" }