# The CPU exit is a batch lane, not a serving lane

> Lab 007 · Editorial lab · Status: published  
> Published by The CTO Advisor LLC · Layer2C Labs

**Question:** Google promotes the C4 virtual machine for GPU-comparable inference through a customer claim it publishes and features, Intel’s own posts echo the language, and the supporting performance chart compares the new Xeon to the older Xeon. Meanwhile a custom model no garden will host needs compute you control, and the NVIDIA GPU requests this program filed on June 29 were still unusable two weeks later. This lab put a LoRA-tuned Gemma 4 26B mixture-of-experts on the Xeon lane and measured which workload shapes it can actually carry. Across two serving stacks, two prompt shapes, and concurrency 1 through 16, zero of 22 measured configurations met the interactive latency bar. The shape could deliver throughput or interactive latency, not both.

**Load:** Gemma 4 26B-A4B (25.2B parameters, 3.8B active) with the lab-six LoRA merged in, served on a c4-standard-32-lssd (Intel Xeon 6 6985P-C, Granite Rapids, AMX) in us-east4 at $1.99 an hour. Two serving stacks (llama.cpp Q8_0 and vLLM bf16), two workload shapes (1K-in/1K-out synthetic matching the vendor benches, and 6K-in/400-out retrieval-augmented prompts), concurrency 1 through 16, 22 measured cells. The GPU comparator could not be provisioned; the commodity API was sampled as the floor it is, for the managed models it alone serves. A post-bench TPU probe tested whether GCP’s third compute lane had a quota or a capacity gate; it measured access only, not inference.

## Executive Summary

The pitch under test was "GPU-comparable performance for small and mid-size models": language Google features through a published customer claim and Intel echoes in its own posts, supported by a chart that compares the new Xeon to the old Xeon. The lab bought the comparison the chart skips. A LoRA-tuned Gemma 4 26B mixture-of-experts, merged on-instance, served on the largest Granite Rapids shape this project could provision, measured across two stacks, two workload shapes, and four concurrency levels.

First, the custom model fit and ran. That is not nothing: it is a real exit from model-garden dependency, on self-serve capacity, at $1.99 an hour. Second, interactive service failed everywhere. No cell of the 22 meets a 5-second first token and 15 tokens per second per stream simultaneously, and the failure splits by stack: llama.cpp owns decode and loses prefill by a factor of seven; vLLM owns prefill and decodes at roughly half llama.cpp’s single-stream rate. Both hit the same short-prompt aggregate ceiling near 50 tokens per second, which is consistent with the software seam both share: dense math reached the matrix units, expert execution did not.

What survives is a batch lane with unusual availability properties: $10 to $14 per million output tokens at sustained saturation, compute cost only, for a model no garden will host, on capacity that reached SSH in 71 seconds and negotiated its quota in minutes. The NVIDIA comparator that would resolve the tradeoff could not be provisioned at all: denied in 2.1 seconds, twice, while the program’s June 29 NVIDIA requests stood unusable at day 14. The TPU door, probed for scope, is different material: no quota gate fired at single-chip scale, two zones failed on capacity, and a v5e chip went from cold API to READY in 17 minutes in the third. Whether a TPU can serve this custom model is unmeasured; whether one is rentable is now not in question. Availability is part of the architecture, and this lab measured it with timestamps.

The commodity API floor was sampled for scale: 611-millisecond first token and $2.50 per million output tokens (Vertex standard on-demand list, observed July 13, 2026, thinking disabled), for a managed provider model that cannot serve or reproduce the custom weights. That number is not a competitor to the CPU lane. It is the price available when the application can accept a provider-managed, non-equivalent model, and the gap between $2.50 and $10-with-broken-latency is one measured premium, in July 2026, for retaining custom weights on self-provisioned infrastructure.

## DAPM Table — Authority Verdict

| Layer | Placement |
| --- | --- |
| layer0 | Delegated |
| layer2c | Retained |

## Detailed Writeup

The bench was designed to be unfair to the thesis in the vendor’s favor. The model should be the computationally favorable case for CPU serving: a sparse mixture-of-experts activating 3.8B of 25.2B parameters, exactly the class the "small to mid-size" claim covers. Its 25.2B-parameter structure also tests whether the serving software can place and execute expert weights efficiently, and that test turned out to be the finding. The stacks got a shootout first and the two strongest carried the matrix. Quantization stayed at int8 or better. The prompts matched the vendor bench’s own shape in one arm, so the comparison meets the marketing on its own terms before the second arm asks the question the marketing skips: what happens at the 6K-token prompts retrieval-augmented workloads actually send.

What happens is prefill. Decode held its rate at long context, 21.9 to 23.3 tokens per second on llama.cpp, so the model reads back at a usable pace once it starts. But llama.cpp prefills this shape at roughly 169 tokens per second, which puts the first token 36 seconds out on a real prompt. vLLM prefills around five times faster and gets the first token to 6.2 seconds, still past the bar, and pays for it with decode at roughly half llama.cpp’s single-stream rate. The lab looked for a configuration that passes both bars and there is not one among the ordinary configuration-level fixes it tried. The closest cells still fail: llama.cpp misses the first-token bar by about 10 percent on the short prompt, and vLLM misses the decode bar by about 24 percent. The frontier is empty, and reporting the near-misses honestly is what makes the emptiness credible.

The second finding is the best explanation for the first. Advanced Matrix Extensions are the load-bearing feature of the entire CPU-inference pitch, and on this model generation neither stack ran the mixture-of-experts layers on them. llama.cpp’s AMX buffer repacked every attention and dense tensor, 206 of them, and none of the expert tensors; vLLM’s CPU expert kernel asserts SiLU activation and Gemma 4’s experts use GELU, so the flagship AMX serving path cannot run the flagship open MoE’s experts at all. The shared throughput ceiling is consistent with both stacks running the same expert math on the same vector units. This was observed, not isolated experimentally, and it is software, not silicon destiny. It is also the current truth of a marketing claim written in the present tense.

The economics sort the workloads cleanly, with the caveat stated: these are compute costs at sustained saturation, excluding idle capacity, redundancy, and the humans who operate the thing. On those terms the lane produces custom-model tokens at $10 to $14 per million with no latency contract. The commodity API produces managed-model tokens at $2.50 with a 611-millisecond first token, and will not host the custom weights that are the reason this lab exists. The NVIDIA GPU that would resolve the tradeoff lists at $3.99 an hour for a four-L4 shape that fits the model in bf16, and this project’s request for it was denied in 2.1 seconds, then the half-size fallback within seconds. The same automation approved CPU family bumps in minutes, the same day, on the same project — and granted a TPU chip with no quota conversation at all. On this project, that asymmetry was the durable fact: the performance numbers will move with software releases, and the doors were policy. Whether another project’s doors behave the same way is a one-hour test, and it belongs before any architecture that assumes the GPU lane opens.

Availability got measured rather than asserted. Eighty-three capacity probes across two passes, business hours and 10:40 PM Central, found the tight zones tight around the clock: everything in us-central1-a stocked out in both passes, a Granite-Rapids-specific hole in us-south1 at all hours, and one lonely diurnal flip in us-east4-c. Two regions carried a C4 family quota of zero for this project by default, the same zero-by-default pattern the GPU lane wears everywhere. The pattern is not a global CPU squeeze; it is regional pressure with batch-shaped overnight demand in the flagship region, and a lane that stayed open somewhere at every hour probed.

## Assessments at the Time of the Lab

| Vendor | Layer | Grade | As assessed |
| --- | --- | --- | --- |
| Google Cloud AI Infrastructure | Layer 0 · Compute | TPU + GPU Full Stack | July 12, 2026 |

## Method and Disclosure

Editorial and self-funded: $37 total, all phases, on-demand rates, no vendor involvement. Disclosure: Intel and Google Cloud are clients of The CTO Advisor practice; this lab was not commissioned, funded, or previewed by either or any vendor, and the editorial ruling was formed from the bench alone. The marketing claims under test are Intel’s "GPU-comparable performance and lower TCO" positioning for Xeon 6 inference and the Intel/Google Cloud C4 launch material; the CPU-to-CPU chart referenced in the standfirst is the published C4-versus-C3 total-cost-of-ownership comparison (1.7x, GPT-OSS 120B, 1K/1K prompts). Links ship in the raw detail.

Instance shapes, stack versions, serving flags, AMX dispatch evidence, the full 22-cell matrix, quota timestamps, both capacity-census passes, and the AWS probe ship in the raw lab detail, including the $22.50 of idle burn from an overnight auth outage. API pricing is Vertex standard on-demand list as observed on July 13, 2026 (gemini-2.5-flash, us-central1, thinking disabled). CPU platform observations (which C4 shapes carried Granite versus Emerald Rapids) are as observed on the tested dates and zones. The custom model is the lab-six adapter merged on-instance; its training data and the corpus taxonomy are proprietary and appear nowhere. Retrieval-shape prompts were synthetic, built from shape statistics only.

---
*Layer2C Labs · The CTO Advisor LLC · labs.layer2c.com*