The CPU exit is a batch lane, not a serving lane
Google promotes the C4 virtual machine for GPU-comparable inference through a customer claim it publishes and features, Intel’s own posts echo the language, and the supporting performance chart compares the new Xeon to the older Xeon. Meanwhile a custom model no garden will host needs compute you control, and the NVIDIA GPU requests this program filed on June 29 were still unusable two weeks later. This lab put a LoRA-tuned Gemma 4 26B mixture-of-experts on the Xeon lane and measured which workload shapes it can actually carry. Across two serving stacks, two prompt shapes, and concurrency 1 through 16, zero of 22 measured configurations met the interactive latency bar. The shape could deliver throughput or interactive latency, not both.
By Keith Townsend · July 14, 2026
The verdict
Scoped to a 32-vCPU Granite Rapids shape, which is not the biggest C4 that exists but is the biggest this project could reach through the self-serve quota path, and to this class of model: a mid-size sparse mixture-of-experts served as a merged custom model. The ruling classifies workload shapes. It does not bless or condemn any application; an application owner locates their workload in the taxonomy and reads their row. A post-bench TPU probe tested whether GCP’s third compute lane had a quota or a capacity gate; it measured access only, not inference, and the accelerated-lane findings are otherwise NVIDIA-specific.
Disclosure: Intel and Google Cloud are both clients of this practice. Neither commissioned, funded, previewed, or had any say in this lab, and the claims under test are their public marketing surfaces. No vendor paid for this answer; the bench ran on $37 of self-funded cloud spend.
Video
How I know
The pitch under test was "GPU-comparable performance for small and mid-size models": language Google features through a published customer claim and Intel echoes in its own posts, supported by a chart that compares the new Xeon to the old Xeon. The lab bought the comparison the chart skips. A LoRA-tuned Gemma 4 26B mixture-of-experts, merged on-instance, served on the largest Granite Rapids shape this project could provision, measured across two stacks, two workload shapes, and four concurrency levels.
First, the custom model fit and ran. That is not nothing: it is a real exit from model-garden dependency, on self-serve capacity, at $1.99 an hour. Second, interactive service failed everywhere. No cell of the 22 meets a 5-second first token and 15 tokens per second per stream simultaneously, and the failure splits by stack: llama.cpp owns decode and loses prefill by a factor of seven; vLLM owns prefill and decodes at roughly half llama.cpp’s single-stream rate. Both hit the same short-prompt aggregate ceiling near 50 tokens per second, which is consistent with the software seam both share: dense math reached the matrix units, expert execution did not.
What survives is a batch lane with unusual availability properties: $10 to $14 per million output tokens at sustained saturation, compute cost only, for a model no garden will host, on capacity that reached SSH in 71 seconds and negotiated its quota in minutes. The NVIDIA comparator that would resolve the tradeoff could not be provisioned at all: denied in 2.1 seconds, twice, while the program’s June 29 NVIDIA requests stood unusable at day 14. The TPU door, probed for scope, is different material: no quota gate fired at single-chip scale, two zones failed on capacity, and a v5e chip went from cold API to READY in 17 minutes in the third. Whether a TPU can serve this custom model is unmeasured; whether one is rentable is now not in question. Availability is part of the architecture, and this lab measured it with timestamps.
The commodity API floor was sampled for scale: 611-millisecond first token and $2.50 per million output tokens (Vertex standard on-demand list, observed July 13, 2026, thinking disabled), for a managed provider model that cannot serve or reproduce the custom weights. That number is not a competitor to the CPU lane. It is the price available when the application can accept a provider-managed, non-equivalent model, and the gap between $2.50 and $10-with-broken-latency is one measured premium, in July 2026, for retaining custom weights on self-provisioned infrastructure.
What the bench measured
The detail
The bench was designed to be unfair to the thesis in the vendor’s favor. The model should be the computationally favorable case for CPU serving: a sparse mixture-of-experts activating 3.8B of 25.2B parameters, exactly the class the "small to mid-size" claim covers. Its 25.2B-parameter structure also tests whether the serving software can place and execute expert weights efficiently, and that test turned out to be the finding. The stacks got a shootout first and the two strongest carried the matrix. Quantization stayed at int8 or better. The prompts matched the vendor bench’s own shape in one arm, so the comparison meets the marketing on its own terms before the second arm asks the question the marketing skips: what happens at the 6K-token prompts retrieval-augmented workloads actually send.
What happens is prefill. Decode held its rate at long context, 21.9 to 23.3 tokens per second on llama.cpp, so the model reads back at a usable pace once it starts. But llama.cpp prefills this shape at roughly 169 tokens per second, which puts the first token 36 seconds out on a real prompt. vLLM prefills around five times faster and gets the first token to 6.2 seconds, still past the bar, and pays for it with decode at roughly half llama.cpp’s single-stream rate. The lab looked for a configuration that passes both bars and there is not one among the ordinary configuration-level fixes it tried. The closest cells still fail: llama.cpp misses the first-token bar by about 10 percent on the short prompt, and vLLM misses the decode bar by about 24 percent. The frontier is empty, and reporting the near-misses honestly is what makes the emptiness credible.
The second finding is the best explanation for the first. Advanced Matrix Extensions are the load-bearing feature of the entire CPU-inference pitch, and on this model generation neither stack ran the mixture-of-experts layers on them. llama.cpp’s AMX buffer repacked every attention and dense tensor, 206 of them, and none of the expert tensors; vLLM’s CPU expert kernel asserts SiLU activation and Gemma 4’s experts use GELU, so the flagship AMX serving path cannot run the flagship open MoE’s experts at all. The shared throughput ceiling is consistent with both stacks running the same expert math on the same vector units. This was observed, not isolated experimentally, and it is software, not silicon destiny. It is also the current truth of a marketing claim written in the present tense.
The economics sort the workloads cleanly, with the caveat stated: these are compute costs at sustained saturation, excluding idle capacity, redundancy, and the humans who operate the thing. On those terms the lane produces custom-model tokens at $10 to $14 per million with no latency contract. The commodity API produces managed-model tokens at $2.50 with a 611-millisecond first token, and will not host the custom weights that are the reason this lab exists. The NVIDIA GPU that would resolve the tradeoff lists at $3.99 an hour for a four-L4 shape that fits the model in bf16, and this project’s request for it was denied in 2.1 seconds, then the half-size fallback within seconds. The same automation approved CPU family bumps in minutes, the same day, on the same project — and granted a TPU chip with no quota conversation at all. On this project, that asymmetry was the durable fact: the performance numbers will move with software releases, and the doors were policy. Whether another project’s doors behave the same way is a one-hour test, and it belongs before any architecture that assumes the GPU lane opens.
Availability got measured rather than asserted. Eighty-three capacity probes across two passes, business hours and 10:40 PM Central, found the tight zones tight around the clock: everything in us-central1-a stocked out in both passes, a Granite-Rapids-specific hole in us-south1 at all hours, and one lonely diurnal flip in us-east4-c. Two regions carried a C4 family quota of zero for this project by default, the same zero-by-default pattern the GPU lane wears everywhere. The pattern is not a global CPU squeeze; it is regional pressure with batch-shaped overnight demand in the flagship region, and a lane that stayed open somewhere at every hour probed.
You measured a 32-vCPU shape. The 144- and 288-vCPU Granite Rapids shapes would fix the prefill problem.
They might. This project could not reach them through the self-serve path to find out. The C4 family limit started at 24 vCPUs per region for this project, the quota automation approved a bump to 96 in the one region that had no Granite Rapids capacity, and capped it at 36 in both regions that had the silicon. The global request for 128 vCPUs was denied outright. For this project, on these dates, the bigger-shape rescue existed in the catalog and not in the quota system. Timestamps ship in the raw detail; whether an account with deeper history gets a different answer is exactly the kind of thing an architect should test on their own project before designing around it.
What a bigger shape buys is prefill, at a linearly bigger hourly rate. Whether cost per token holds, improves, or degrades at 144 vCPUs is unmeasured, and this lab will not project it: memory bandwidth per core, NUMA topology, and the expert path all move at once. That measurement is the designed follow-up, and the grant is half the experiment.
What core count is unlikely to fix is the decode seam. In both tested stacks, independently, the mixture-of-experts tensors stayed off the Advanced Matrix Extensions (AMX) path: llama.cpp repacked 206 attention and dense tensors into its AMX buffer and zero expert tensors, and vLLM’s fused expert kernel rejects Gemma 4’s GELU activation outright. More cores may lift expert decode; they do not put the experts on the matrix units. What Intel’s marketing calls AMX inference is, on the flagship open MoE today, AMX prefill and attention stapled to vector-unit decode. That is a software seam, and it is the constraint that matters at every shape until it closes.
Where each layer belongs
| Layer | Placement |
|---|---|
Layer 0 · Compute Compute & Network Fabric Rented in every arm; GCP owns the layer end to end. Inside it, two lanes with different doors: the general-purpose lane negotiated quota in minutes and provisioned in under a minute and a half; the NVIDIA lane denied in 2 seconds and has held two external requests unusable past two weeks; the TPU lane, probed for scope, showed no quota gate at single-chip scale and failed on capacity instead, granting a chip in one of three zones in minutes. Three doors, three materials: CPU is a negotiation, TPU is a capacity lottery, NVIDIA is a policy wall. Substrate choice inside a delegated layer is now an availability decision before it is a performance one. | Delegated |
Layer 2C · Reasoning Agentic Infrastructure — The Reasoning Plane A custom LoRA merged into open weights is the strongest Retained position at this layer: no garden hosts it, so no garden can withdraw it. The lab priced what exercising that position costs on the one self-serve substrate: retaining the model was technically possible; retaining an interactive latency contract was not, constrained by the serving software on one side and the inaccessible GPU lane on the other. | Retained |
The two questions this lab now knows to ask
Prefill scales with cores, so the biggest C4 shapes might drag the 6K-prompt first token under the bar, and whether cost per token survives the bigger hourly rate is equally unmeasured. The quota automation capped this project at 36 vCPUs in every region that had the silicon, so the question is as much about the door as the shape. The experiment is designed and cheap; the grant is the experiment.
Two census passes say the pressure is real but regional: persistent stockouts in us-central1 that worsened overnight, a round-the-clock Granite Rapids hole in us-south1, open capacity everywhere else at every hour probed. That is consistent with others discovering the same exit, and it is two timestamps from one account. The census script is saved and re-runs in minutes; the trend line is the answer, and the question keeps its question mark.
What it did not prove
- It did not measure a GPU or a TPU. The G2 arm could not be provisioned, which is the lab’s premise wearing its own evidence, but it means the GPU column prices from list rates, not from a bench. "GPU-comparable" was retired against its own latency bar, not against a measured GPU. The TPU probe measured the door only: a single v5e chip (16GB) cannot hold this model anyway, multi-chip serving needs a TPU-native stack, and none of that was benched. The accelerated-lane findings are NVIDIA-specific and say nothing about TPU serving.
- It did not test the biggest shapes, and it does not project their economics. The 144- and 288-vCPU Granite Rapids machines might fix prefill; whether cost per token holds at their hourly rates is unmeasured. The quota ceiling that kept them out of this lab is documented with timestamps for this project, not asserted as anyone else’s ceiling.
- The AMX-expert split is today’s software, not a permanent property, and the shared-ceiling explanation is an inference from dispatch evidence, not an isolated experiment. If either stack ships a GELU-capable AMX expert path, the decode picture moves and the ruling is dated on purpose.
- It says nothing about the custom model’s quality. Lab six judged those weights; this lab only established that the merged custom model serves at base-model speed.
- The census is pattern evidence from one account over two days, not a market study. A small AWS probe (no per-family quota gate, first-try capacity in two of three regions, against a much smaller Xeon 6 footprint) ships in the raw detail as a pointer, not a finding.
Notes from the author, Keith Townsend
I went in expecting to price a serving tier and came out holding a workload classifier. The question I brought was whether the Xeon lane could serve my model. The question the data answers is which shapes of work it serves, and that is a more useful instrument than the one I designed.
The after-hours hypothesis was mine and it was wrong in the way I like being wrong: I guessed capacity would relax overnight, and instead us-central1 got tighter at 10:40 PM Central. Batch demand does not go home. That single inversion taught me more about who else is in this lane than the daytime probes did.
The two-second GPU denial next to the four-minute CPU approval is the sentence I will be repeating in briefings for a year. Same project, same automation, same day. The quota system has an opinion about what you should be building on, and it delivers it faster than any account team.
Assessments at the time of the lab
Method and disclosure
Editorial and self-funded: $37 total, all phases, on-demand rates, no vendor involvement. Disclosure: Intel and Google Cloud are clients of The CTO Advisor practice; this lab was not commissioned, funded, or previewed by either or any vendor, and the editorial ruling was formed from the bench alone. The marketing claims under test are Intel’s "GPU-comparable performance and lower TCO" positioning for Xeon 6 inference and the Intel/Google Cloud C4 launch material; the CPU-to-CPU chart referenced in the standfirst is the published C4-versus-C3 total-cost-of-ownership comparison (1.7x, GPT-OSS 120B, 1K/1K prompts). Links ship in the raw detail.
Instance shapes, stack versions, serving flags, AMX dispatch evidence, the full 22-cell matrix, quota timestamps, both capacity-census passes, and the AWS probe ship in the raw lab detail, including the $22.50 of idle burn from an overnight auth outage. API pricing is Vertex standard on-demand list as observed on July 13, 2026 (gemini-2.5-flash, us-central1, thinking disabled). CPU platform observations (which C4 shapes carried Granite versus Emerald Rapids) are as observed on the tested dates and zones. The custom model is the lab-six adapter merged on-instance; its training data and the corpus taxonomy are proprietary and appear nowhere. Retrieval-shape prompts were synthetic, built from shape statistics only.
Download the raw lab detail (Markdown)