# Lab 7 — Raw lab detail: The CPU exit is a batch lane, not a serving lane

Layer2C Labs, editorial, self-funded. Published 2026-07-14. This document carries the
substrate engineering behind the ruling at labs.layer2c.com/labs/gemma4-xeon-inference.

**Publish boundary.** Everything about the substrate ships: shapes, stacks, flags, numbers,
quota timestamps, gotchas, spend. What stays proprietary: the training data and taxonomy
behind the custom adapter, the advisory corpus, and the analytical conclusions drawn from
it. The retrieval-shape prompts used here were synthetic, generated from shape statistics
(token counts and document-chunk structure), not from the corpus.

## The vehicle

- Base: google/gemma-4-26B-A4B-it (25.2B total, 3.8B active, MoE, 128 experts / 8 active
  + 1 shared, 256K context). Weights: ~49GB bf16, 26.9GB Q8_0 GGUF, ~15GB int4 (not used;
  the lab's quality floor was int8).
- Custom model: a rank-32 LoRA (attention + MLP projection targets) trained on this base
  in a prior lab, merged into full weights **on-instance** with peft (bf16, rc=0, ~14 min
  on 32 vCPU), then converted to Q8_0 GGUF for llama.cpp. Only the ~148MB adapter ever
  crossed the wire; base weights pulled from Hugging Face on the GCP side.
- Serving decision: **merged only.** Runtime-adapter serving costs 8.7% on llama.cpp
  (adapter matmuls fall off the AMX buffer type — the server log literally warns
  "cannot use buft AMX, fallback to CPU") and vLLM's CPU Fused-MoE-LoRA path is
  unsupported ("Monolithic kernels are not supported for Fused MoE LoRA"). Merged custom
  equals base speed within noise on both stacks, confirmed at matrix scale.

## The hardware (and the shape gotcha)

- **C4 ≠ Granite Rapids.** As observed 2026-07-13/14: plain c4-standard-2..96 shapes ran
  Xeon Platinum 8581C (Emerald Rapids); Granite Rapids (Xeon 6985P-C, 6900P series) lived
  in the -lssd variants, -metal, and the 144/288-vCPU shapes. `--min-cpu-platform "Intel
  Granite Rapids"` is rejected for C4 ("C4 VM does not support minCpuPlatform"); platform
  is selected by shape. Verify with lscpu, not the catalog.
- Measurement shape: **c4-standard-32-lssd**, us-east4-a. 32 vCPU (16 physical cores,
  single NUMA node), 120GB DDR5, 5×375GB Titanium local SSD. All-core turbo 3.9GHz.
  amx_tile / amx_bf16 / amx_int8 present.
- Rate (SKU catalog, us-east4, on-demand): 32 vCPU × $0.03465 + 120GiB × $0.003938 +
  1875GiB lssd × $0.16/mo ≈ **$1.99/hr** (+~$0.033/hr for the 300GB hyperdisk boot disk).
- Memory fit observed: Q8_0 server RSS 29.5GB; bf16 HF peak 35-40GB; bf16 vLLM + 16GB KV
  ~62GB system. A 30GB c4-standard-8 thrashes on the Q8_0 (26.9GB) — don't bother.
- GCE virtual PMU exposes no AMX/memory-bandwidth counters. AMX verification must be
  dispatch-evidence based (below).

## Stack shootout (Phase 0, concurrency 1, 1K in / 512 out)

| Stack | Quant | Decode tok/s | TTFT @1K | AMX evidence | LoRA mode |
|---|---|---|---|---|---|
| llama.cpp (native GNR build, fa on, -t 32) | Q8_0 | 23.5 | 4.78s | weights in ggml "AMX" buffer type | runtime or merged |
| vLLM 0.25.0 +cpu wheel (VLLM_CPU_SGL_KERNEL=0) | bf16 | 11.66 | 1.20s | oneDNN verbose: 2567/2567 matmuls brg_matmul:avx10_1_512_amx (dense layers only) | merged only |
| IPEX 2.8 ipex.llm.optimize (torch 2.8) | bf16 | 4.71 | 4.31s | n/a | runtime |
| Transformers 5.13 + SDPA + static KV (reference) | bf16 | 5.78 | 5.77s | n/a | runtime |

IPEX is a dead end for new model families: frozen at torch 2.8, predates this model,
generic path only, loses to plain Transformers on decode. The Phase 0 "winner" call
(llama.cpp) was superseded by the matrix decision to run both leaders, because prefill
and decode winners are different stacks.

**The AMX/MoE seam (the load-bearing engineering fact).** Neither stack runs the
mixture-of-experts tensors on AMX. llama.cpp's AMX buffer repacked 206 attention and
dense-FFN tensors and zero `ffn_*_exps` tensors (verbose-load evidence run, `-lv 5`,
grep buft/repack). vLLM's CPU fused-MoE kernel asserts SiLU activation and this model's
experts are GELU, so experts take the native non-AMX fallback. Both stacks therefore run
expert decode on the same vector units, which is consistent with their converging
aggregate ceilings. This is serving-software state as of July 2026, not silicon.

## The matrix (22 cells, c4-standard-32-lssd, $1.99/hr)

Bar: TTFT p95 < 5s AND ≥15 tok/s decode per stream. **Zero cells pass.** Closest misses:
llama.cpp vendor c1 over the TTFT bar by ~10%; vLLM vendor c1 under the decode bar by ~24%.

Custom = merged LoRA. Vendor shape = 1024 in / 1024 out. RAG shape = 6144 in / 400 out
(synthetic, seeded, unique heads to defeat prefix caching; cache_prompt off for honest
cold TTFT; temp 0).

| Stack | Model | Shape | Conc | TTFT p50/p95 (ms) | Decode/stream | Agg tok/s | $/1M out |
|---|---|---|---|---|---|---|---|
| llama.cpp | custom | rag | 1 | 36,652 / 37,132 | 21.9 | 6.7 | $83.00 |
| llama.cpp | custom | rag | 4 | 86,842 / 138,164 | 3.4 | 6.7 | $82.63 |
| llama.cpp | custom | rag | 8 | 181,133 / 281,725 | 0.8 | 3.1 | $177.74 |
| llama.cpp | custom | rag | 16 | 188,840 / 314,473 | 0.1 | 0.5 | $1,228.40 |
| llama.cpp | custom | vendor | 1 | 5,510 / 5,539 | 22.8 | 20.8 | $26.63 |
| llama.cpp | custom | vendor | 4 | 21,494 / 21,517 | 11.2 | 36.5 | $15.14 |
| llama.cpp | custom | vendor | 8 | 30,692 / 42,165 | 6.6 | 48.1 | $11.49 |
| llama.cpp | custom | vendor | 16 | 48,470 / 80,408 | 3.9 | 54.6 | $10.13 |
| llama.cpp | base | rag | 1 | 34,616 / 34,643 | 21.9 | 7.6 | $72.73 |
| llama.cpp | base | rag | 8 | 109,377 / 157,025 | — | 0.2 | $3,071.00 |
| llama.cpp | base | vendor | 1 | 5,443 / 5,460 | 22.9 | 20.4 | $27.15 |
| llama.cpp | base | vendor | 8 | 30,720 / 42,287 | 5.9 | 44.8 | $12.35 |
| vLLM | custom | rag | 1 | 6,208 / 6,400 | 11.4 | 9.7 | $57.11 |
| vLLM | custom | rag | 4 | 9,850 / 21,022 | 6.1 | 21.7 | $25.44 |
| vLLM | custom | rag | 8 | 9,999 / 43,747 | 3.8 | 29.0 | $19.06 |
| vLLM | custom | rag | 16 | 10,254 / 91,406 | 2.3 | 40.2 | $13.74 |
| vLLM | custom | vendor | 1 | 1,085 / 1,144 | 11.5 | 11.3 | $48.79 |
| vLLM | custom | vendor | 8 | 4,327 / 8,025 | 6.0 | 47.3 | $11.69 |
| vLLM | base | rag | 1 | 6,415 / 6,454 | 11.3 | 9.3 | $59.57 |
| vLLM | base | rag | 8 | 10,230 / 46,865 | 3.7 | 25.9 | $21.34 |
| vLLM | base | vendor | 1 | 1,133 / 1,155 | 11.4 | 11.3 | $49.14 |
| vLLM | base | vendor | 8 | 5,927 / 8,168 | 5.8 | 45.8 | $12.08 |

$/1M output = instance rate ÷ aggregate output. Compute cost only, at sustained load;
excludes idle time, redundancy, and operations. Dropped for budget: vendor push past c16
(saturation evident: +13% aggregate for 2× streams c8→c16), base c4/c16, vLLM vendor
c4/c16, runtime-adapter rows, the 12B dense contrast arm.

Prefill rates that explain the RAG TTFTs: llama.cpp ~169 tok/s prefill at this shape;
vLLM ~5× faster. Decode held at long context (21.9-23.3 on llama.cpp), so the RAG
failure is prefill, not decode.

## The doors (availability, measured with timestamps, all on one project)

**NVIDIA (policy wall).**
- 2026-06-29: neocloud (CoreWeave) access requested 14:20; hyperscaler H100 committed
  request filed 21:11Z. At bench time (day 14): neocloud unresolved; H100 "partially
  approved to 1" but unusable behind GPUS_ALL_REGIONS = 0.
- 2026-07-13 23:01Z: L4 self-serve requests (global 0→4 and regional us-east4 1→4)
  **denied in 2.4s and 2.1s**. Fallback ask (→2): denied in 4s and 7s. Threshold
  sensitivity does not rescue the GPU lane.
- G2 list rates (SKU catalog, us-east4): g2-standard-24 (2×L4) ≈ $1.99/hr;
  g2-standard-48 (4×L4, fits bf16) ≈ $3.99/hr. Unrentable at those prices without quota.

**CPU (negotiation).**
- C4 family default for this project: 24 vCPU/region (invisible in the legacy quota list;
  lives in the Cloud Quotas API as CPUS_PER_VM_FAMILY). Two regions carried family
  quota = 0 by default (us-west1, us-west2).
- Self-serve automation, same day as the GPU denials: family 24→96 approved in ~6 min
  (us-central1, the region with no GNR capacity); capped at 36 in both regions that HAD
  GNR capacity (us-east4 ~7 min, us-east1 ~1.5 min); global 32→128 denied, retry at →96
  approved in ~4 min. Ask for exactly what you need.
- Console-to-SSH, clean measurements: 44-71s (71s for the GNR -lssd shape). One 8.5-min
  outlier on a reattached staged boot disk (guest-side first-boot work; API accepted in 33s).
- Quota approved ≠ capacity exists: with all gates clear, c4-standard-96 stocked out in
  all four us-central1 zones (Monday, business hours).

**TPU (capacity lottery, probed for scope 2026-07-14).**
- Cold project (TPU API never enabled): API enabled in 7s. No quota gate fired at any
  point, single-chip scale.
- v6e-1 us-central1-a: "Insufficient capacity" (83s). v6e-1 us-east5-b: accepted, ~2.5 min
  CREATING, then "There is no more capacity in the zone." v5litepod-1 us-west4-a:
  **CREATED, READY in 2.5 min** — cold-API-to-chip 17 minutes. Deleted immediately.
- Scope: door only. A single v5e chip (16GB) cannot hold this model; TPU serving was not
  benched. Gotcha: --version takes the bare runtime name (v2-alpha-tpuv5-lite), not the
  runtimeVersions resource path.

**Capacity census (GCP, 83 probes, two passes).** Real creates of c4-standard-4-lssd
(GNR) vs c4-standard-4 (Emerald Rapids) vs n4-standard-4 (control), deleted on success.
Pass 1 Monday ~17:00-18:30 ET; pass 2 22:40 CT (03:40Z).
- Persistent, not diurnal: us-central1-a stocked out everything (GNR, ER, N4) both
  passes; us-south1-a is a GNR-only hole at all hours (ER and N4 fine).
- One diurnal flip: us-east4-c (GNR stockout by day, open by night). Inverse-diurnal in
  us-central1-b/c: ER open by day, stocked out at 22:40 CT — batch-shaped overnight demand.
- Open at all hours probed: us-east1 (×3 zones), us-east4-a/b, us-east5, us-west3/4,
  europe-west1/3/4, asia-northeast1, asia-southeast1.
- AWS pointer (small probe, 3 regions, 2-vCPU instances, 02:35 CT): no per-family quota
  gate (Standard-vCPU pool, 1,920 in each region probed), first-try capacity for
  c8i.large/m8i.large in us-east-1 and us-west-2, us-east-2 untestable (no default VPC).
  Much smaller Xeon 6 regional footprint than GCP. A pointer, not a finding.

## API floor (Phase 3)

gemini-2.5-flash, Vertex v1beta1, us-central1, 50 sequential RAG-shape requests
(6,168 in / 400 out), observed 2026-07-13. **thinkingBudget: 0 is required** — default
thinking silently consumes maxOutputTokens (13-token answers, TTFT 2.7-3.0s) until
disabled. With thinking off: TTFT p50 611ms / p95 878ms, total p50 2.84s, effective
decode ~180 tok/s/stream. Standard on-demand list: $0.30/1M in, $2.50/1M out.
The API serves the base model only; it cannot host the custom weights at any price.
Surface notes: v1 endpoints 404 on this project (v1beta1 works); newer Flash IDs 404 —
2.5-flash was the newest this project resolves.

## Operational gotchas worth stealing

- **Hugging Face xet transfer hangs silently on GCE** (30 min dead, incomplete files
  frozen). HF_HUB_DISABLE_XET=1 pulled 29GB in ~2 min over classic HTTP.
- vLLM's +cpu GitHub release attaches x86_64 AND aarch64 wheels; filter your download.
- lssd shapes cannot set-machine-type; stage-small/measure-big works via boot-disk swap
  (create the measurement instance on the staged disk). Budget an extra first-boot delay.
- The GGUF tokenizer's 1.46× token inflation seen in Phase 0 was prompt pathology
  (whitespace runs in synthetic text), not a llama.cpp property: realistic prose counted
  6,145 vs HF's 6,144.
- Bench honestly: llama-bench numbers do not include serving overhead. The Phase 0
  kill-criterion pass (4.78s TTFT) became 5.51s on the real serving path, which inverted
  a pass into a fail.
- An overnight gcloud reauth outage left the measurement box idle 11.3h (~$22.50 of the
  $28.20 matrix spend). nohup + on-disk RESULT_JSON logs meant zero data loss. Run an
  idle-stop watchdog; assume your auth dies at 2am.

## Spend

Phase 0: $8.70. Matrix + arms: $28.20 (bench compute ~$4.20; idle burn $22.50; hyperdisk
$0.50; Vertex $0.33). Census + TPU probe: ~$0.50. **Total ≈ $37.** Cap was $135 across
phases; teardown verified to zero instances/disks/TPUs after each phase.

## Disclosure

Intel and Google Cloud are clients of The CTO Advisor practice. Neither commissioned,
funded, previewed, or had any say in this lab; the claims under test are their public
marketing surfaces (the "GPU-comparable" language is a SAS customer quotation that
Google publishes and features in its C4 launch post, echoed in Intel's own posts), and
the ruling was formed from the bench alone. No vendor paid for this answer.

## Vendor response (addendum, 2026-07-14)

Intel responded to the published lab. Paraphrased fairly, in four parts: (1) the
upstream multi-user path on Xeon is vLLM/SGLang, with speculative decoding as a decode
enhancement; (2) llama.cpp is positioned as single-user, with Xeon support enabled via
an OpenVINO backend; (3) the GPU-comparable positioning applies in GPU-limitation
scenarios — memory per device, models spanning devices without interconnect,
input-token spill to CPU, an overloaded single GPU; (4) the value case named is small
and medium models in deterministic agentic workloads (LangGraph, Pydantic, DSPy
class), not one-shot long-prompt dumps.

The lab's read: point 3 scopes the launch claim to GPU-limitation scenarios, which is a
narrower statement than the marketing tested here. Points 1 and 2 contain testable
claims: this lab measured vLLM to concurrency 16 (the multi-user ceiling published above
is that measurement; the dispatch evidence shows the GELU expert path off AMX), while
SGLang, speculative decoding, and OpenVINO-backed llama.cpp were not tested and are fair
follow-up arms. Point 4 names a workload class this lab did not measure. A follow-up lab
on the vendor's named configuration and the agentic workload shape is planned; the
ruling above stands as measured until that bench runs.

## Sources under test

- Intel/Google Cloud C4 + Xeon 6 launch positioning ("GPU-comparable performance," "up to
  1.7x TCO"): the TCO chart compares C4 (Xeon 6) to C3 (4th-gen Xeon) on GPT-OSS 120B at
  1K/1K — a CPU-to-CPU denominator. Intel Community: "Google Cloud C4 VMs + Intel Xeon 6
  Delivers Instant AI ROI"; Hugging Face blog: "Google Cloud C4 Brings a 70% TCO
  improvement on GPT OSS with Intel and Hugging Face"; Google Cloud blog: "C4 VMs based
  on Intel 6th Gen Xeon Granite Rapids now GA."