# Lab 008 — Raw lab detail ## Renting the chip was the easy part The substrate record behind Lab 008 (gemma4-tpu-inference). Every provisioning attempt, quota mechanic, serving error, config fact, gotcha, and dollar, on the dated run. This is the evidence under the ruling on the lab page, published so the reasoning survives inspection. **The boundary.** What ships here: TPU provisioning and quota behavior with timestamps, the model's own public configuration, the memory-fit math, the vLLM-TPU serving errors verbatim, the substrate gotchas, and the spend. What does not: a custom LoRA adapter was staged for a planned custom-model arm, but the serving attempts ran on the public base model, and the adapter, its training data, and the corpus taxonomy are proprietary and appear nowhere. Internal project and bucket identifiers are generalized. Referenced performance numbers are attributed to their source labs, and the one external benchmark to its author. **Frame.** All timestamps UTC, 2026-07-14, on a single self-serve GCP project. On-demand rates, no vendor involvement. Google Cloud is a client of this practice; this lab was not commissioned, funded, or previewed by Google or any vendor. --- ## The model under test `google/gemma-4-26B-A4B-it`. Public config (`config.json`, fetched HTTP 200): - Architecture: `Gemma4ForConditionalGeneration` (multimodal image-text-to-text wrapper); text tower `gemma4_text`, mixture of experts, `enable_moe_block=true` - `num_experts=128`, `moe_intermediate_size=704`, `hidden=2816`, 30 layers, `hidden_activation=gelu_pytorch_tanh` - `num_attention_heads=16`, `num_key_value_heads=8`, `head_dim=256` - `num_global_key_value_heads=2`, `global_head_dim=512`, `sliding_window=1024` - Weights ~51.6GB bf16 (49.9 + 1.7 safetensors) Two load-bearing facts. **`num_global_key_value_heads=2` caps pure tensor parallelism at tp ≤ 2** (tp=4 and tp=8 fail to compile). And the **heterogeneous head dims** (256 sliding, 512 global) break vLLM chunked prefill (needs `--disable_chunked_mm_input`). --- ## Phase -1 — the multi-chip door (provisioning pre-flight) Does the multi-chip shape a 26B model needs provision on the self-serve lane? Every attempt logged with UTC timestamps and failure class. **v6e-4 (Trillium), catalogued in us-east5-a, us-east5-b, us-central1-a:** | Attempt → result | Zone | Class | Message | |---|---|---|---| | 16:45:30 → 16:46:04 | us-east5-a | CAPACITY | "There is no more capacity in the zone" | | 16:46:04 → 16:48:38 | us-east5-b | CAPACITY | "There is no more capacity in the zone" | | 16:48:38 → 16:48:48 | us-central1-a | CAPACITY | "Insufficient capacity." | 0 of 3 zones. Quota was fine (`tpu-v6e` limit 16/zone). Pure capacity, not policy. **v5litepod-8 (v5e-8):** - 16:49–16:51 | us-east5-b and us-west4-a | QUOTA: `429 RESOURCE_EXHAUSTED`, "Quota limit `TPUV5sLitepodServingPerProjectPerZoneForTPUAPI` has been exceeded. Limit: 4 in zone." - Note: gcloud masked this as a generic "network connectivity" error. The real 429 was only visible in the full `HttpError` body. If you probe this door, read the raw error, not gcloud's summary. - Reading: not capacity. A quota gate. `tpu-v5s-litepod-serving` defaults to 4 chips/zone. v5e-8 needs 8, so it is policy-blocked below the model's need. **v5litepod-4 (v5e-4, fits the limit-4 quota):** - 17:05:48 → 17:05:55 | us-east5-b | CAPACITY-dry - 17:05:55 → 17:08:30 | us-west4-a | SUCCESS, READY ~2.5 min, deleted clean. Same zone that granted Lab 007's single chip. **Default quota picture (this project, 2026-07-14):** `tpu-v6e` = 16/zone; `tpu-v5s-litepod-serving` = 4/zone; `TPU_LITE_DEVICE_V5` (single device) = 0. Phase -1 cost ~$0.15. No resource stranded. --- ## The quota door (extends Lab 007's ledger) - 17:20:04 | Filed increase: `TPUV5sLitepodServingPerProjectPerZoneForTPUAPI` 4 → 8, zone us-west4-a. - ~17:41 | GRANTED. Effective limit us-west4-a now 8, reconciling cleared. Turnaround under ~21 minutes, self-serve, no human or sales contact. - Reading: the v5e serving quota negotiates like the CPU door (Lab 007: CPU family bumps auto- approved in 4–6 min), the opposite of NVIDIA's 2-second policy denial. The TPU wall that actually bit this model was v6e capacity, not policy. - Gotcha: the quota is a **per-zone chip-sum**, not per-node. With a live v5e-4 (4 chips) up, a v5e-8 (8) create returned `429 "Limit 8 exceeded"` (4 + 8 = 12 > 8). Delete the smaller node to free the quota before the larger one will provision. --- ## Phase 0 — serving attempt (vLLM-TPU, v5e-8) Shape: v5litepod-8, us-west4-a, 8 chips single host (2×4), 16GB HBM/chip, 377GB host RAM, 88GB free boot disk. Runtime `v2-alpha-tpuv5-lite`. Bare runtime (no torch_xla / jax / vllm preinstalled), Docker 27.0.3. Stack: vLLM-TPU nightly (image `af8a30d9fc2c`, ~12h old). Base model; the custom arm was descoped, base and custom perform the same per Lab 007. **Arch: supported.** `tpu_inference` ships a native JAX model (`models/jax/gemma4_mm.py`) with `load_weights` and `process_unquantized_moe_weights`. The Gemma 4 mixture of experts is not rejected at the architecture level. Contrast Lab 007, where both CPU stacks dropped the GELU experts off the matrix path. **bf16: does not serve.** TP=8, bf16, `max-model-len 4096`: - `HBM RESOURCE_EXHAUSTED` (XLA E0101 `RuntimeProgramAllocationFailure`) inside `jit_process_unquantized_moe_weights` at model-load: needs 2.54G contiguous, only 1.39G free per chip. 1.39G free on a 16GB chip after weight-load means ~14.6G consumed per chip. The experts are **replicated, not sharded** across TP=8 (a sharded MoE would leave ~9–10G free). - Retry `--gpu-memory-utilization` 0.5 and 0.85: identical OOM. The HBM-fraction lever does not change XLA weight-load free memory on the JAX path. - So even at 128GB aggregate HBM, bf16 does not serve on v5e via vLLM-TPU: the experts don't shard, and no single 16GB chip holds the model. **On-the-fly quantization: not supported.** `--quantization fp8` → `KeyError "None of the keys ['quant_method'] found in config"`. The TPU JAX path does not dynamically quantize; `--quantization fp8` requires a pre-quantized checkpoint whose `config.json` already carries `quantization_config.quant_method`. Supported TPU quant modules present: `fp8`, `compressed_tensors`, `mxfp4`. No dynamic int8 or AQT path. **No expert-parallel flag.** The `vllm-tpu` serve help (95 lines) exposes no expert-parallel or MoE-sharding knob. Expert parallelism, the path that fits, lives in MaxText/JetStream, not this stack. MaxText+JetStream was not attempted: it is a strictly larger vendor-locked detour (Hugging Face → Orbax scanned-checkpoint conversion plus XLA compile) that would only reconfirm the same finding. **The tp ≤ 2 math.** From the config, 2 global KV heads cap pure TP at 2. At tp=2, 51.6GB bf16 shards across only 2 chips → ~26GB/chip, over the 16GB v5e budget. The model does not fit at any legal TP degree on 16GB chips. It fits on 32GB v6e chips (~26GB < 32GB), which is why the external v6e-4 benchmark serves it and v5e cannot. --- ## Substrate gotchas (publishable, reproducible) - **Hugging Face Xet trap.** `google/gemma-4-26B-A4B-it` is Xet-backed. The Xet client (`hf_xet`, `HF_XET_HIGH_PERFORMANCE=1`) stalled from the TPU node (~0 bytes in 20+ min); a plain curl of `resolve/main` returns a 1KB Xet-bridge pointer, not the bytes. Workaround: `HF_HUB_DISABLE_XET=1` forces the classic LFS path → ~215 MB/s, 49.9GB in ~4 min. Open HTTP/LFS, no vendor tool, but the Xet default is an operational landmine. - **Do not restart docker on a TPU-VM.** A `systemctl restart docker` left an orphan `dockerd` and tripped systemd's start-limit; the half-dead daemon replayed stale container state (phantom "container already in use") for ~15 min and cost real time and money. Recovery: `pkill -9 dockerd/containerd-shim; systemctl reset-failed docker; systemctl start docker`. The TPU runtime agents run under docker and auto-recovered. Self-inflicted, recorded for honesty. - **TPU-VM ssh drops (rc255)** on long-held or background-launched sessions. Detach with `setsid` + `nohup` + `