Renting the chip was the easy part
Google markets the Tensor Processing Unit (TPU) as the price-performance home for Gemma-class inference, and Lab 007 showed the door opens fast: a chip in minutes, a quota bump in minutes, where the NVIDIA lane says no in seconds. This lab set out to serve a mid-size Gemma 4 mixture-of-experts on the lane and fill a latency matrix. It never filled the matrix, because renting the chip turned out to be the easy part. On the silicon you can actually rent self-serve, a bring-your-own model does not fit, and getting it to serve means adopting Google’s stack or quantizing off-box. The performance was already trusted work in earlier labs. The friction was the finding.
By Keith Townsend · 2026-07-14
The verdict
Scoped to the self-serve, on-demand path on a single GCP project, to the reachable v5e silicon (Trillium v6e capacity was dry on the measured date), and to the open-source vLLM-TPU serving stack. It is a ruling about the level of effort to serve a bring-your-own mid-size mixture-of-experts on the lane, not a claim that Google’s own stack cannot serve Gemma. The served model’s performance is referenced from Lab 006 and Lab 007, not re-measured here. This lab measured the friction, not the tokens per second.
Disclosure: Google Cloud is a client of this practice. Google did not commission, fund, preview, or have any input into this lab. The claim under test is Google’s own public positioning of the Tensor Processing Unit (TPU) as the price-performance home for Gemma-class inference, and the ruling was formed from roughly $25 of self-funded, on-demand probes. No vendor paid for this answer.
Video
How I know
The pitch under test is Google’s positioning of the Tensor Processing Unit (TPU) as the price-performance home for Gemma-class inference. Lab 007 had already shown the door opens: a single chip reached READY in about 17 minutes on a project with no TPU history, while the NVIDIA lane denied its requests in two seconds. This lab asked the next question, whether a mid-size mixture-of-experts you bring yourself actually serves on the lane, and it stopped buying answers the moment the effort became the answer.
The reachable self-serve silicon is v5e, because Trillium (v6e) capacity was dry in every catalogued zone on the measured date. On v5e the bf16 model does not fit. Gemma 4 26B has two global key/value heads, which caps tensor parallelism at two, and at that degree the weights land on two chips at roughly 26GB each against a 16GB budget. Adding chips does not help, because the open-source vLLM-TPU stack replicates the 128 experts rather than sharding them. There is no tensor-parallel degree that serves bring-your-own bf16 on this 16GB hardware. On the 32GB Trillium chips it does fit, and an independent benchmark clocks the same mixture-of-experts there at about 457,000 tokens per second. Trillium was the shape that came back capacity-dry.
Two paths get you to a served model, and both are detours. Run Google’s MaxText and JetStream, which ship expert-parallel sharding and are the supported route that fits, at the cost of adopting Google’s checkpoint format, mesh config, and compiler. Or quantize to four bits off-box and serve that on vLLM, which Lab 006 already showed is quality-neutral on this model. The single-chip host can’t even do the quantizing itself. It has 47GB of RAM against a 50GB model.
So the ruling is about level of effort, not tokens per second, and the performance is referenced rather than re-bought: Lab 006 for four-bit quality and speed, Lab 007 for the interactive frontier and the $2.50 commodity floor. The chip is self-serve and the quota negotiates in minutes. Serving your own model is where the lane charges you, and it charges in engineering.
What the bench measured
How the lanes compare
| Lane / config | Serves BYO model? | First token | Decode tok/s | $ / 1M out | Source |
|---|---|---|---|---|---|
| Xeon C4 (CPU), vLLM + llama.cpp bf16 / int8. The comparator frontier. | Yes, but 0 of 22 cells cleared the bar | 6.2s – 36.7s | 11.4 – 21.9 | $10 – 14 | Lab 007 |
| Commodity API (Gemini Flash) Base model only. Cannot host your weights. | Base only | 611ms | — | $2.50 | Lab 007 |
| TPU v5e, bf16, vLLM (bring your own) tp ≤ 2, experts replicate; OOM on a 16GB chip. | No — does not fit | — | — | — | This lab |
| TPU v6e-4, bf16, vLLM (same MoE, external) Independent Google Developer Expert benchmark, attributed. Same model (google/gemma-4-26B-A4B-it) on a v6e-4 pod via vLLM. v6e’s 32GB chips hold bf16 where the reachable 16GB v5e can’t. Trillium was capacity-dry in this lab’s probe. Aggregate figure, not per-stream. | Yes — the MoE runs | <1.2s (low conc.) | 457k tok/s (agg.) | — | dev.to GDE (ext.) |
| TPU v5e, 4-bit, vLLM (single chip) Lab 006 showed 4-bit at ~29 tok/s on the GB10, +25% vs bf16, quality at parity. TPU latency and cost unmeasured; checkpoint built off-box. | Fits one chip | unmeasured on TPU | unmeasured on TPU | unmeasured | This lab + 006 |
| TPU, bf16, MaxText + JetStream Expert parallelism shards the 128 experts. The fitting route; vendor stack, not benched. | Fits (expert-parallel) | unmeasured | unmeasured | unmeasured | This lab |
The detail
The bench was designed to serve the model and fill a latency matrix. It became a record of why the matrix does not fill on the reachable path, which is the more useful result. Every wall is documented with a timestamp, and together they measure the level of effort the price-performance chart leaves out.
Start with availability, because it decides the shape. Trillium is Google’s current flagship TPU generation, the shape a 26B model wants, and on the measured date a v6e-4 create failed on insufficient capacity in all three zones that list it. The quota was fine. This is the same capacity-not-policy pattern Lab 007 found at single-chip scale, now at the multi-chip shape a 26B model needs. The lane that grants a chip in minutes did not grant this shape at all. And the shape matters, because Trillium is where this model actually runs. An independent Google Developer Expert benchmark served the same 26B mixture-of-experts on a v6e-4 pod at about 457,000 tokens per second aggregate, with first token under a second and a half at low concurrency. The 32GB Trillium chips hold at bf16 what the 16GB v5e chips cannot. So the silicon that serves the model is precisely the silicon that was not rentable, and the reachable silicon does not fit it. That is the whole ruling in one line.
That forces v5e, and v5e forces the memory question. The model’s own configuration answers it. Two global key/value heads cap tensor parallelism at two, so bf16 shards to two chips at about 26GB each and a 16GB chip runs out of memory on load. The instinct is to add chips. It doesn’t work, because the serving stack replicates the experts. On a mixture-of-experts the expert weights are most of the model, and replicated experts mean every chip carries nearly the whole thing regardless of the chip count. Expert parallelism is the fix, and expert parallelism is Google’s MaxText, not the open-source vLLM path. That is the boundary this lab actually found, and it is worth naming plainly: the TPU itself is not the portability wall, the serving runtime is. The same weights on the same silicon serve or fail depending on whether they run through vLLM or through Google’s stack. Lab 007 measured hardware portability. This one measured runtime portability, and the runtime is where the model quietly changes hands.
The quota door, by contrast, behaved well, which sharpens the ruling rather than softening it. A serving quota increase from four to eight chips auto-approved in about 21 minutes, self-serve, no sales contact, the same negotiable behavior Lab 007 saw on the CPU lane and the opposite of the NVIDIA policy wall. So the friction is not the door. A CTO can rent a TPU faster than a GPU and negotiate its quota in minutes. What the CTO cannot do, on this hardware with this stack, is drop a bring-your-own mid-size mixture-of-experts onto it and serve. That takes the vendor’s toolchain or an off-box quantization pass, and either one is the level of effort this lab set out to name.
You gave up too early. Google’s own MaxText path serves Gemma 4 on TPU fine, with day-zero support.
Granted, and that’s the point, not the rebuttal. MaxText with expert parallelism is the route that fits, it is real, and it is supported. This lab does not claim a TPU cannot serve Gemma. It measures what it takes to serve a mid-size mixture-of-experts you brought yourself, on the hardware you can actually rent self-serve, with the serving stack most teams already run.
On that reachable path the answer is blunt. Bring-your-own bf16 does not fit and does not serve on vLLM-TPU. Getting to a served model means one of two detours. Adopt Google’s training-lineage toolchain, converting your weights into its scanned checkpoint format and running its mesh config and compiler. Or build a four-bit checkpoint off-box and ship that. Both are fine engineering. Neither is rent-a-chip-and-serve-your-model.
So the level of effort is the finding. The chip is self-serve and the quota negotiates in minutes. The serving stack is a decision, and on this lane the low-effort default is Google’s, not yours. That’s a legitimate answer for a CTO to plan around. It just isn’t the one the price-performance chart implies.
Where each layer belongs
| Layer | Placement |
|---|---|
Layer 0 · Compute Compute & Network Fabric Rented, GCP owns the lane end to end. Three doors into it, now measured at the shape a 26B model needs: CPU a negotiation settled in minutes (Lab 007), NVIDIA a policy wall that denies in two seconds (Lab 007), TPU a mixed door. The TPU quota negotiates like CPU, a four-to-eight serving bump auto-approved in about 21 minutes, but the Trillium capacity the price-performance story assumes was dry in all three catalogued zones on the measured date. Substrate choice inside this delegated layer is an availability decision before it is a performance one. | Delegated |
Layer 2C · Reasoning Agentic Infrastructure — The Reasoning Plane The model is the operator’s to keep. On the reachable TPU silicon, keeping it served is where the effort lands. A bring-your-own bf16 mixture-of-experts does not fit on v5e, and the two paths that serve it both reach up from Layer 0 into how the model is expressed: Google’s expert-parallel MaxText stack (its checkpoint format, mesh config, and compiler), or an off-box quantization pass into a four-bit checkpoint. Neither withdraws the model, so this stays Retained. But the serving toolchain is Google’s to define on this lane, and that is the level of effort the marketing omits. | Retained |
The two questions this lab now knows to ask
The model serves well once it fits (Lab 006, referenced), but the TPU-native latency under a 6K retrieval prompt is unmeasured. Standing up MaxText was named as the fitting path, not benched. That measurement is the designed follow-up, and it is a larger, vendor-locked build.
One measured date, all three catalogued zones dry, quota fine. Lab 007 ran an 83-probe capacity census for the CPU lane and turned an anecdote into a trend line. The same census for v6e would settle whether the marketed shape is reliably rentable or reliably not. The script re-runs in minutes. The question keeps its question mark.
What it did not prove
- It did not measure TPU serving latency or throughput. Nothing served on TPU in this lab. The performance claims are referenced: the v6e-4 throughput from an external, attributed Google Developer Expert benchmark, four-bit quality and speed from Lab 006 (NVIDIA GB10), and the interactive frontier and API floor from Lab 007 (Xeon). The external v6e-4 number establishes the model has run on TPU. It is not first-party, not peer-reviewed, and not a number this lab measured or stands behind as its own.
- It did not run MaxText or JetStream. Expert parallelism is named as the fitting path from the model config and Google’s own documentation, not benched. Whether it clears the interactive bar, and at what cost, is unproven and is the next lab.
- The tensor-parallel cap and the expert replication are read from the model config and one vLLM-TPU image on the measured date. A future vLLM-TPU that ships expert parallelism moves the ruling. It is dated on purpose.
- The capacity and quota findings are one project, one date, one self-serve path. Not a market study, and not a claim about reserved capacity or an account with a sales relationship.
- It says nothing about model quality. Lab 006 judged that. This lab only established the effort to get the model serving, not how well it answers.
Notes from the author, Keith Townsend
I set out to benchmark TPU latency. I came home having measured effort instead. The number I can stand behind isn’t tokens per second. It’s that a rentable chip and a servable model are two different things, and the gap between them is engineering I didn’t have to do on a GPU.
Someone handed me a spec that said Gemma 4 has four global key/value heads, so cap tensor parallelism at four. The config said two. I almost rented a v5e-8 for an hour to prove a topology that can’t compile. Verify the config before you rent the chip. I keep relearning that one.
The quota bump approving in 21 minutes, next to a model that still wouldn’t fit, is the whole lab in one image. The door isn’t the problem. The room behind it is.
Assessments at the time of the lab
Method and disclosure
Editorial and self-funded: roughly $25 total, on-demand rates, no vendor involvement. Disclosure: Google Cloud is a client of this practice; this lab was not commissioned, funded, or previewed by Google or any vendor, and the ruling was formed from the probes alone. The claim under test is Google’s public positioning of TPU as the price-performance home for Gemma-class inference. An editorial lab discloses the relationship and rules anyway.
What was measured here: TPU provisioning and quota behavior with timestamps, the model’s tensor-parallel ceiling from its own configuration, the bf16 memory fit on v5e, and the vLLM-TPU expert-replication behavior on a nightly image, all on a single GCP project on the dated run. What was referenced, not re-measured: the v6e-4 throughput for the same mixture-of-experts, four-bit quality and throughput from Lab 006, and the interactive frontier and commodity API floor from Lab 007. No TPU serving latency was measured, because the level of effort to reach a served model was the finding. The custom adapter, its training data, and the corpus taxonomy are proprietary and appear nowhere.
One external source is cited, with attribution: an independent Google Developer Expert benchmark of the same 26B mixture-of-experts on v6e-4. It is used to establish that the model has been served on TPU, not as a first-party or peer-reviewed result. This is an editorial lab, not a sponsored or academic one, so an attributed community benchmark clears the bar for that single, bounded claim. A dense-model result would not have, because the difficulty here was the mixture-of-experts specifically.
Download the raw lab detail (Markdown)