Borrow the vendor’s plumbing, not its judgment

The cost verdict is commodity knowledge, and it sits below as evidence. These are the rulings that change how the stack gets scored: where authority sits, and what you cede without noticing.

Self-funded. No vendor paid for this answer.

Where does authority actually sit when you split an AI stack between the public cloud and a box on your desk? That is the question this lab was built to answer. I built a retrieval-augmented generation (RAG) pipeline over thirteen years of my own transcripts, roughly 237K segments across 5,815 videos, 2013 to 2026. The data plane went to AWS. The compute stayed home on an NVIDIA DGX Spark, below the cloud's managed abstraction. I validated on a 5,000-segment subset, then ran the full corpus, not a sample, and watched where the decisions landed across the 4+1 stack.

The economics turned out to be the boring part. Embedding and indexing the entire corpus cost $0.84, one time, in roughly 41 minutes, at 337K segments an hour with 0 failures. Keeping the index runs about $0.07 a month. The entire 13-year generation pass would have cost about $0.20 at cloud rates. Retrieval was never the constraint either: latency from the Spark ran p50 280ms and p95 505ms, and most of that was my own network path, since the S3 Vectors service itself sits around 100ms. Cheap, fast, better than expected. If the story ended there, this would be a commodity pricing note. It doesn't end there.

The wrapper made decisions I never gave it

The managed service, Bedrock Knowledge Bases, looked like the product. When I went to stand it up, the control plane was gated: it would not create through the API even for an administrator account. So I rebuilt the same data plane on the primitives underneath. Titan Text Embeddings v2, a 1024-dimension cosine S3 Vectors index, a query. It worked. And the rebuild exposed the real cost of the managed path, which was never orchestration. It was chunking. The managed wrapper re-chunks on its own strategy and turns on parsing you did not ask for, and that would have broken label-filtered retrieval outright. Bring-your-own kept one segment as one vector with its labels attached, and that is what made hybrid retrieval work: semantic similarity plus metadata filtering in one call. Chunking determines what you can retrieve. That decision is yours. Don't cede it.

So the ruling on AWS runs opposite to the one you'd expect from a story about a gated console. AWS's strength is the composable primitives underneath the managed wrapper, not the wrapper. The wrapper is convenience; the primitives are the capability. Score AWS on the primitives. Borrow the vendor's plumbing, and keep your own judgment on chunking.

But what about utilization?

The obvious objection to any cloud-wins cost verdict is that I didn't keep the box busy, and the on-prem case is to keep the box busy so the capex pays off. Fair. I ran that math. Pegged at 100% for three years, the Spark generates roughly 1.9 billion tokens, which spreads the $4,699 to about $2.50 per million tokens, plus around $0.40 in power. Call it ~$2.90 per million, fully utilized. The same model on Bedrock is about $0.40 per million. So even maxed out for three years straight, the owned box costs roughly seven times more per commodity token. Utilization is the wrong lever. The box is memory-bandwidth-bound at about 20 tokens a second, so it produces too few tokens an hour for the capex to ever spread thin enough to win. And to hold 70 to 90% you'd need a constant firehose of batch work feeding a box that loses even when fed. You can't util your way past a throughput ceiling.

That doesn't make the Spark a failure, and the fair way to say it is that the box did real work. It built and ran the whole pipeline end to end. Local generation with Gemma 4 26B-A4B through vLLM held ~20-23 tokens a second, and one model-class note is worth keeping: a mixture-of-experts (MoE) model with ~4B active parameters beat a dense 8B on both speed and capability on this hardware, because single-stream decode is memory-bandwidth-bound and active parameters are what move. But the same model runs 136-214 tokens a second on Bedrock, 7 to 9x faster, for less money. So place the box by its authority, not its benchmarks. The Spark keeps real local execution authority. Its place is development and portability, not commodity inference, where the cloud wins on cost and speed.

The architecture itself passed, which is the result the economics can obscure. The 4+1 seams fell where the model predicted. Splitting the stack with the data plane in the public cloud and the reasoning at the edge is an operable partition, not just a diagram: the I/O across that seam, retrieved context down to the box and queries up, was never the bottleneck. The constraint lived inside Layer 2, in local generation throughput, not at the boundary between layers. The designed partition holds.

And the harder half of the stack stayed home either way. The reasoning plane that yields a true answer, Layer 2C in the 4+1 model, is the gates, the judges, and the reproducibility around the model, not the weights. Generating LLM findings is free and easy. Generating true ones is not. It takes a quantitative gate, cross-run reproducibility, and a strong judge, and judge strength dominates: a mid-tier judge will rubber-stamp a confidently wrong pattern that a strong one refutes. The fix wasn't the most expensive judge. A cheap strict judge plus one frontier judge, trusted where they agree, did the work, and most naive discoveries did not survive that gate. Cede the model and you still own the harder half. The lab even caught me being wrong along the way: my first notes said a frontier model wasn't available on one cloud. It was. I'd failed to access it, which is a different problem, and I corrected the record instead of shipping the mistake.

What a dollar didn't settle

The honest bounds matter as much as the verdict. This lab did not prove on-prem inference never pays; it proved commodity base-model generation, on this box, for this workload, loses to the cloud on cost and speed. It did not settle custom or fine-tuned model economics, because the cost verdict was measured against commodity per-token pricing, which exists only for base models. The moment a workload needs a fine-tuned model, AWS stops selling tokens and charges a floor instead. Lab two ran that regime and reversed the verdict: against a floor, the owned box wins custom serving above roughly 12 to 20% utilization. It did not prove managed RAG is bad; it proved the wrapper can hide control points, chunking above all. And it did not test capability against the frontier. An 8B-class model is a cost-structure vehicle, not a frontier contender. There's a second axis the throughput numbers hide, too. Faster is not better. The MoE model won single-stream speed, but a dense Gemma 4 31B reasoned better over the same data, and stepping up to foundation models reasoned better still. That moves the economics from cost-per-token to value-per-answer. Cheap and fast minimizes the token bill. It does not maximize the worth of the answer.

So the verdict travels in one line: borrow the plumbing, keep your own chunking. What the lab leaves standing is a question I didn't consider worth asking until the bench showed how good the native floor was. The AWS-native data plane was cheap and did hybrid retrieval in one call. So where does VAST Data, or any performance or portable data platform, earn its keep over S3 Vectors: on latency, on portability, on retained governance? Now there is a measured floor to test it against.

The numbers

Where each layer belongs

Assessments at the time of the lab

Method and disclosure

Self-funded, with no sponsor, so the lab is free to mix competitors. That cross-vendor mix is the editorial signature, nobody here is selling you one box. Every lab, editorial or sponsored, is held to the same method and the same editorial control; sponsored labs simply center on the sponsor’s architecture.

Validated on a 5,000-segment subset, then run across the full corpus plus a 2026 refresh. Generation used Gemma 4 26B-A4B via vLLM on the DGX Spark, chosen over a benchmarked Llama 3.1 8B and a dense Gemma 4 31B. The data plane used Bedrock Titan Text Embeddings v2 into a 1024-dimension cosine S3 Vectors index, with labels kept filterable for hybrid retrieval.

Substrate engineering and the corpus findings are captured as working lab notes, not shipped. The classification taxonomy, the tagged corpus, the trained judges, and the specific conclusions stay proprietary. What ships is the pattern and the substrate verdict, enough to recognize and abstract.

The essay on this page was rendered from the lab’s frozen structured record by a model writing under this practice’s voice specification, with a deterministic gate checking every figure against the record before publication. The record is the canonical surface: where prose and record disagree, the record rules.