The CPU exit is a batch lane, not a serving lane

Scoped to a 32-vCPU Granite Rapids shape, which is not the biggest C4 that exists but is the biggest this project could reach through the self-serve quota path, and to this class of model: a mid-size sparse mixture-of-experts served as a merged custom model. The ruling classifies workload shapes. It does not bless or condemn any application; an application owner locates their workload in the taxonomy and reads their row. A post-bench TPU probe tested whether GCP’s third compute lane had a quota or a capacity gate; it measured access only, not inference, and the accelerated-lane findings are otherwise NVIDIA-specific.

Disclosure: Intel and Google Cloud are both clients of this practice. Neither commissioned, funded, previewed, or had any say in this lab, and the claims under test are their public marketing surfaces. No vendor paid for this answer; the bench ran on $37 of self-funded cloud spend.

Google Cloud promotes the C4 virtual machine for inference through a customer claim it publishes and features: GPU-comparable performance for small and mid-size models. Intel echoes the language in its own posts. The chart underneath compares the new Xeon to the older Xeon, a 1.7x total-cost-of-ownership win over the prior generation. Read that twice. The evidence offered for a GPU-comparable claim is a CPU-to-CPU comparison. So somebody had to buy the comparison the chart skips, and I had a reason to be that somebody. I own a custom model no garden will host: the lab-six Low-Rank Adaptation (LoRA) adapter merged into Gemma 4 26B, a sparse mixture-of-experts activating 3.8B of its 25.2B parameters. Meanwhile the NVIDIA GPU requests this program filed on June 29 were still unusable two weeks later. The question stopped being academic. Can the Xeon lane carry a custom model, and which workload shapes can it actually serve?

The bench was designed to be unfair to my own skepticism, in the vendor's favor. A sparse mixture-of-experts should be the computationally favorable case for CPU serving, exactly the class the small-to-mid-size pitch covers. I served it on a c4-standard-32-lssd, Intel Xeon 6 Granite Rapids silicon with Advanced Matrix Extensions (AMX), at $1.99 an hour. That was the largest shape this project could reach self-serve, a bound I'll come back to. Two serving stacks carried the matrix after a shootout: llama.cpp at Q8_0 and vLLM at bf16. Two workload shapes: 1K-token prompts matching the vendor benches, and the 6K-token retrieval-augmented generation (RAG) prompts real applications actually send. Concurrency 1 through 16. That's 22 measured cells.

The frontier is empty

Credit first, because it's real. The custom model fit and ran, and the merged adapter served at base-model speed, a 0% penalty. That is a genuine exit from model-garden dependency, on capacity nobody had to approve. However, interactive service failed everywhere. My bar was a user-experience judgment, not a vendor number: first token under 5 seconds, because five seconds is the patience edge of an interactive tool, and 15 tokens per second per stream, a comfortable reading pace. Zero of 22 cells met both.

The failure splits by stack, in opposite directions. llama.cpp owns decode, 21.9 tokens per second on the long prompt, and takes 36.7 seconds to produce the first token. vLLM answers in 6.2 seconds and decodes at 11.4. Prefill is the villain: llama.cpp prefills this shape at roughly 169 tokens per second, and vLLM prefills around five times faster yet still misses the bar. The near-misses are worth reporting honestly, because they make the emptiness credible. llama.cpp clears everything except the first-token bar on the short prompt, over by about 10 percent. vLLM misses the decode bar by about 24 percent. There is no cell where a user waits an acceptable time and then reads at an acceptable pace. Concurrency makes it worse. llama.cpp fell to 3.9 tokens per second per stream at 16 streams, and both stacks hit the same short-prompt aggregate ceiling of 45 to 55 tokens per second. Two independent codebases converging on one ceiling is a clue.

The experts never touched the matrix units

The clue leads to the second finding, which is the best explanation for the first. AMX is the load-bearing feature of the entire CPU-inference pitch, and on this model generation neither stack ran the mixture-of-experts layers on it. llama.cpp repacked 206 attention and dense tensors into its AMX buffer and zero expert tensors. vLLM's fused expert kernel asserts SiLU activation, and Gemma 4's experts use GELU, so it rejects them outright. Two stacks, independently, stapling AMX prefill and attention to vector-unit decode. What the marketing calls AMX inference is, on the flagship open mixture-of-experts today, AMX for everything except the experts. I observed this through dispatch evidence rather than isolating it experimentally, and it's software, not silicon destiny. It is also the current truth of a claim written in the present tense.

What survives is a batch lane

The economics sort the workloads cleanly, with the caveat stated: compute cost at sustained saturation, excluding idle time, redundancy, and the humans who run the thing. On those terms the lane produced custom-model tokens at $10 to $14 per million. The vendor-shaped workload bottomed at $10.13, the RAG shape at $13.74, and the latency contract was broken in both. So the honest fit is latency-tolerant batch work on a custom model: corpus sweeps, evaluation runs, classification passes, overnight generation. One concession exists. A patient single operator at 1K prompts and concurrency 1 gets a 1.1-second first token from vLLM at 11 tokens per second. One person, one stream at a time. Don't generalize it into a user-facing tier.

Two comparison points frame the price. The commodity API floor came in at $2.50 per million output tokens with a 611-millisecond first token, sampled on July 13, 2026. That number is not a competitor, because the managed model behind it cannot host my weights at any price. The gap between $2.50 and $10-with-broken-latency is one measured premium, in July 2026, for keeping custom weights on infrastructure you control. The second comparison never happened, and that absence is evidence. The four-L4 NVIDIA shape that would resolve the tradeoff lists at $3.99 an hour, and my request for it was denied in 2.1 seconds. The half-size fallback died within seconds too. Same project, same day, same automation: the CPU family bumps approved in 4 to 6 minutes. A Tensor Processing Unit (TPU) probe, run for scope only, found a third door made of different material. No quota gate fired at single-chip scale, two zones failed on capacity, and a v5e chip reached READY in 17 minutes in the third. Serving on it stayed unmeasured, and the accelerated-lane findings here are NVIDIA-specific.

Availability got measured rather than asserted. The CPU lane went console to Secure Shell (SSH) in 44 to 71 seconds, and 83 capacity probes across two passes found regional pressure rather than a global squeeze: us-central1-a stocked out around the clock, a Granite Rapids hole in us-south1 at all hours, open capacity somewhere at every hour probed. My after-hours hypothesis was wrong in the way I like being wrong. I guessed capacity would relax overnight, and us-central1 got tighter at 10:40 PM Central. Batch demand doesn't go home.

The obvious objection says I measured a 32-vCPU shape and the 144- and 288-vCPU machines would fix prefill. They might. I couldn't reach them to find out. The C4 family limit started at 24 vCPUs per region, the automation approved a bump to 96 in the one region with no Granite Rapids capacity, capped it at 36 in both regions that had the silicon, and denied the global request for 128 outright. For this project, on these dates, the bigger-shape rescue existed in the catalog and not in the quota system. And what core count is unlikely to fix is the seam: more cores may lift expert decode, but they don't put the experts on the matrix units.

Bound the ruling honestly. This lab measured no GPU and no TPU inference; "GPU-comparable" was retired against its own latency bar, not against a measured GPU. It says nothing about the model's quality, which lab six judged. The whole bench cost $37, including $22.50 of idle burn from an overnight auth outage, disclosed because that's the job. I went in expecting to price a serving tier and came out holding a workload classifier, which is the more useful instrument. The verdict: this lane delivers throughput or interactive latency, not both, so treat it as a batch lane and route your work accordingly. And the two-second GPU denial sitting next to the four-minute CPU approval is the sentence I'll be repeating in briefings for a year. The quota system has an opinion about what you should be building on, and it delivers that opinion faster than any account team.

The numbers

Where each layer belongs

Assessments at the time of the lab

Method and disclosure

Editorial and self-funded: $37 total, all phases, on-demand rates, no vendor involvement. Disclosure: Intel and Google Cloud are clients of The CTO Advisor practice; this lab was not commissioned, funded, or previewed by either or any vendor, and the editorial ruling was formed from the bench alone. The marketing claims under test are Intel’s "GPU-comparable performance and lower TCO" positioning for Xeon 6 inference and the Intel/Google Cloud C4 launch material; the CPU-to-CPU chart referenced in the standfirst is the published C4-versus-C3 total-cost-of-ownership comparison (1.7x, GPT-OSS 120B, 1K/1K prompts). Links ship in the raw detail.

Instance shapes, stack versions, serving flags, AMX dispatch evidence, the full 22-cell matrix, quota timestamps, both capacity-census passes, and the AWS probe ship in the raw lab detail, including the $22.50 of idle burn from an overnight auth outage. API pricing is Vertex standard on-demand list as observed on July 13, 2026 (gemini-2.5-flash, us-central1, thinking disabled). CPU platform observations (which C4 shapes carried Granite versus Emerald Rapids) are as observed on the tested dates and zones. The custom model is the lab-six adapter merged on-instance; its training data and the corpus taxonomy are proprietary and appear nowhere. Retrieval-shape prompts were synthetic, built from shape statistics only.

The essay on this page was rendered from the lab’s frozen structured record by a model writing under this practice’s voice specification, with a deterministic gate checking every figure against the record before publication. The record is the canonical surface: where prose and record disagree, the record rules.