{ "slug": "constraints-not-weights", "lab_number": 6, "title": "Put the judgment in the constraints, not the weights", "type": "editorial", "status": "published", "sponsor": null, "author": "Keith Townsend", "date": "July 7, 2026", "date_iso": "2026-07-07", "layers": [ "layer2b", "layer2c", "layer3" ], "axes": [ { "instrument": "4plus1", "layers": [ "layer2b", "layer2c", "layer3" ], "has_dapm_table": true } ], "supersedes": [], "superseded_by": null, "corrects": [], "corrected_by": [], "vendors": [ { "key": "nvidia", "role": "hw" }, { "key": "gcp", "role": "ref" } ], "themes": [ "judgment-transfer", "fine-tuning-limits", "prompt-vs-weights", "validator-authority" ], "finding": "Fine-tuning captured voice and bounded behavior, not judgment; one paragraph of standing positions in the system prompt beat two fine-tuning rounds.", "question": "The pitch is everywhere: fine-tune a local model on a decade of your published judgment and it becomes you. Lab two ruled \"own the weights\" and built the kill-criterion that goes with it: if base plus retrieval clears the bar, do not fine-tune. This lab is that criterion firing. Two measured training rounds on an owned DGX Spark lost to their own base model, and both lost to a single paragraph of written positions in the system prompt. Throughout, the box means the compute, serving, and training kept below the platform’s abstraction, and the NVIDIA DGX Spark is that box.", "load": "A 563-pair instruction dataset built from a 2,727-document advisory corpus and 25 published vendor assessments, trained as LoRA adapters on Gemma 4 26B-A4B and Qwen3-4B, and measured by a 70-probe judgment gate built before the first training run: held-out vendor assessments as answer keys, honesty probes that invite fabrication, framework-discipline probes, and head-to-head comparisons against the author’s real published answers.", "verdict": { "scope": "Scoped to judgment work: advisory answers, vendor assessment, epistemic honesty, on a strong instruction-tuned base with the expert’s corpus available for retrieval. This is not a ruling against fine-tuning. Lab two’s voice result stands: bounded rendering trains cheaply and well. Judgment is not rendering, and that distinction is the lab.", "independence": "Self-funded. No vendor paid for this answer.", "calls": [ { "kind": "do", "text": "build the gate before the fine-tune. The validator is the control point. Training loss converged cleanly on every run; only the gate saw that the artifacts fabricated benchmarks, collapsed into repetition loops, and flattened the very framework they were trained on." }, { "kind": "dont", "text": "fine-tune judgment into weights when written constraints reach the same ceiling. Two rounds, 563 curated pairs, and three hours of GPU time lost to one paragraph of standing positions on every measured family. When the constraint is free, no training run beats it." }, { "kind": "do", "text": "write the expert’s standing positions into the system prompt, scoped to fire only when a question implicates them. On the production model this closed every honesty gap on three differently-shaped prompt surfaces, including a fallback path that had been inventing throughput figures (3 of 8 honest, stock, to 8 of 8), at zero cost to assessment quality and, after scoping, zero cost to natural voice." }, { "kind": "dont", "text": "train on judgment prose without refusal exemplars. A dataset where every answer renders confident judgment teaches the confidence and not the boundary: round one’s adapter answered an unbenched latency question with an invented winner. The behavior that did transfer, measurably, was the 40 refusal pairs added in round two." } ] }, "objection": { "q": "You fine-tuned wrong. More data, better hyperparameters, a bigger model would get there.", "a": [ "Partly conceded, and the concessions are in the numbers. Round one overcooked at four epochs and collapsed into loops; round two at two epochs and a fifth the learning rate fixed the collapse on the 26B. Where the data taught a specific behavior, training worked: 40 refusal pairs moved honesty from 4/3/1 to 7/1/0 against the canon keys. Fine-tuning is not broken. It does what the data shape says.", "But the objection misses the economics. The control was not another training run; it was a paragraph that took ten minutes to write, scored 8/0/0 on honesty, 3/0/0 on discipline, and the best assessment agreement of any run in either round. Fine-tuning judgment does not compete against a better fine-tune. It competes against the cost of writing your positions down. Lab two’s fine-tune won because the bounded rendering task had no cheaper path to the ceiling. This task had one, and the kill-criterion built in that lab fired exactly as designed." ] }, "numbers": [ { "label": "The gate", "value": "46 placement + 8 honesty + 3 discipline probes, 13 head-to-heads", "note": "held-out vendors as answer keys; repetition screen; human blind read" }, { "label": "Round 1 (r32, lr 1e-4, 4 epochs)", "value": "fabricated benchmarks; 4B looped on 11 of 13", "note": "training loss converged cleanly; only the gate saw the damage" }, { "label": "Round 2 (2 epochs, lr 2e-5, +refusals)", "value": "honesty 7/1/0 vs base 4/3/1", "note": "the one real transfer; assessment agreement still below base" }, { "label": "Advisory head-to-head vs real answers", "value": "base beat tuned 12-1, 13-0", "note": "judge picks which answer is closer to the author’s published one" }, { "label": "The control: one stance paragraph", "value": "honesty 8/0/0 · discipline 3/0/0", "note": "best result on every family, both rounds, no training" }, { "label": "Production surfaces (Gemini 2.5 Pro + stance)", "value": "legacy 4/2/2 · assembly 7/0/1 · fallback 3/0/5 — all → 8/0/0", "note": "block survives 10x prompt dilution; the naked fallback carried the worst fabrications" }, { "label": "Naturalness cost of the guardrails", "value": "10-3 drift → 7-6 parity", "note": "one scoping sentence: positions fire only when implicated" }, { "label": "Bit-rate sub-bench (26B MoE)", "value": "NVFP4 28.6 vs bf16 22.8 tok/s · 15 vs 49 GB", "note": "quality parity across the gate; the naive 4x from bandwidth math does not exist" }, { "label": "Whole-lab cost", "value": "~$0 marginal + ~3 GPU-hours", "note": "owned box; production validation on existing Vertex project" } ], "opened": [ { "q": "Does the assessment headroom close with better evidence, or is it real disagreement?", "a": "Partly settled by a follow-up run. On the 39 law-matched probes, the compressed constraint block scored 15 full matches; the complete written methodology as the system prompt scored 22, with misses collapsing from 17 to 6, almost all of the gain on the function-scored instrument whose gradient rules the block never carried. So most of the original headroom was prompt underspecification, not unwritten judgment. The six surviving misses are gradient-step disagreements, several traceable to product facts the probes did not carry. What remains unpriced is the last rung: the human gate. That is the next measurement." }, { "q": "When does the constraint set stop scaling?", "a": "One paragraph of positions was free and perfect at eight honesty probes and three discipline probes. A real advisory practice holds hundreds of positions. Whether written constraints keep winning as the rulebook grows, or whether recitation drift returns and some form of training re-enters at a measurable crossover, is open. The scoping sentence that fixed drift at this size is a hint, not an answer." } ], "not_proved": [ "It did not prove fine-tuning cannot encode judgment. It proved 563 LoRA pairs on two bases could not beat a free paragraph on this gate. Full-parameter training, preference optimization, or an order of magnitude more data were not tested.", "The judge has a family conflict. The local gate judge is the same base model one candidate was tuned from, backstopped by a human blind read and by the production validation on a different model family, but a fully independent judge was not used.", "Honesty and discipline saturated at 8 of 8 and 3 of 3, which bounds what the gate can now distinguish. Those families are regression tests going forward, not scoreboards.", "The bit-rate parity is one MoE architecture on one box at one scale, measured single-stream. Batched serving and other model families were not benched.", "The gate is not valid for search-grounded surfaces. Its honesty keys assume evidence is not in hand; a research mode with live web search puts evidence in hand, and what scores as fabrication may be a real retrieval. Search-grounded surfaces need their own probes and their own stance text.", "The voice fine-tune from lab two was not re-litigated. Bounded style transfer trained well then and nothing here contradicts it." ], "executive_summary": [ "The lab set out to fine-tune an advisor’s judgment into a local model and ended up measuring where judgment actually lives. The instrument came first: a 70-probe gate with held-out vendor assessments as answer keys, built before training because the June finding was that automated metrics pass models a human reader rejects. Everything that followed was scored by the same gate, the same judge, and a human blind read.", "Two training rounds produced one genuine transfer and two persistent losses. Refusal data measurably taught the boundary. Nothing taught the judgment: the tuned models never beat their own base on assessment agreement, lost the advisory head-to-heads 12-1 and 13-0, and the smaller model collapsed into verbatim loops in both rounds. The control ended it. One paragraph of standing positions in the system prompt beat every fine-tuned artifact on every family, then transferred unchanged to the production model on another cloud, where it eliminated live fabrication failures at zero measured cost." ], "dapm": [ { "instrument": "4plus1", "layer": "layer2b", "placement": "Delegated", "note": "The model became substitutable the moment judgment moved to 2C: the same stance block governed a local Gemma and production Gemini 2.5 Pro unmodified, two model families, two clouds. The fine-tune would have inverted this, welding the judgment to one checkpoint’s lifecycle. The bit-rate sub-bench (4-bit parity at a third the memory) is this layer’s remaining decision, and it is housekeeping." }, { "instrument": "4plus1", "layer": "layer2c", "placement": "Retained", "note": "The ruling lives here. The expert’s judgment operationalizes as written standing positions plus a validator gate, both plain text, both owned. This is the layer the fine-tune tried to compile down into 2B, and the compilation failed. Retaining 2C as explicit policy is what makes every layer below it swappable." }, { "instrument": "4plus1", "layer": "layer3", "placement": "Retained", "note": "Where the failures manifest and the brand carries the risk. The fabricated performance winner and the invented roadmap are application-plane incidents; the buyer meets them, not the weights. The 2C guardrails changed this layer’s behavior without touching the application or the model." } ], "seam_map": null, "writeup": [ "The instrument is the lab. Before any training run, the gate froze 46 assessment probes whose answer keys are the author’s published vendor rulings, held out of training entirely, plus honesty probes engineered to invite fabrication and discipline probes that tempt a model to flatten the 4+1 framework into generic layers. A mechanical repetition screen sits in front of the judges, and 13 advisory questions are scored head-to-head against the author’s real published answers. Every candidate in this lab, local or cloud, tuned or stock, faced the same 70 probes.", "The dataset respected one discipline: evidence in the prompt, judgment in the completion, so the model learns the grading move rather than memorizing grades. Construction had its own validator. A first pass of generated question-answer pairs failed the author’s read 8 of 10; the fix was a judge in the construction loop verifying that each answer actually answers its question, which kept 247 of 1,376 eligible documents. The final set was 563 pairs across advisory, assessment, framework catechism, and refusal families.", "Round one looked perfect from the inside and failed everywhere the gate looked. Loss curves converged. The artifacts fabricated: asked for a latency comparison nobody benched, the tuned 4B answered with an invented winner, and asked cold for the layer framework it was trained on, the tuned 26B rebuilt a generic compute-network-app stack instead. The 4B also collapsed into verbatim repetition loops on open-ended questions, a failure mode invisible to averaged metrics and obvious to any reader. The diagnosis was the data shape: every training answer rendered confident judgment, so confidence generalized and the boundary did not.", "Round two fixed what data can fix and proved the point by contrast. Gentler hyperparameters ended the collapse on the 26B. Forty refusal exemplars, iterated three times against the author’s corrections until every decline matched the axis of its question, moved honesty from 4/3/1 to 7/1/0. That is real transfer, and it is the strongest evidence in the lab that fine-tuning works when the data teaches a behavior. But assessment agreement stayed below the untuned base in both rounds, and the tuned models lost the advisory head-to-heads 13-0 and 12-1. Imitating judgment’s outputs did not produce judgment.", "The control decided the lab. One paragraph of standing positions, the expert’s actual rules for performance claims, roadmap speculation, scale behavior, and framework structure, written into the system prompt of the untuned base, scored 8/0/0 on honesty, 3/0/0 on discipline, and the best placement agreement of any run on the box. It cost nothing to build and it beat three hours of training on curated data. The same block then transferred, unmodified, to the production advisory service running Gemini 2.5 Pro on another cloud, and a follow-up gate run against the real prompt surfaces sharpened the finding. A legacy prompt failed half the honesty probes by inventing a performance winner and a vendor roadmap; the live assembled prompt, which already carries the author’s voice document, held honesty at 7 of 8 stock; and a bare fallback path fabricated worst, 3 of 8, inventing a throughput figure outright. The block took all three surfaces to 8 of 8 and reproduced the lab’s numbers exactly on a prompt ten times the size it was tuned on. Two iterations were needed: a scoping sentence to stop answers reciting frameworks on questions that did not raise them, and a scope exclusion for the research surface, whose mission is finding pricing and performance data and whose axis the block contradicts. A constraint written for one mission does not paste onto another, and the gate is what caught that before deploy.", "A bit-rate sub-bench answered the box’s serving question with the same gate. NVIDIA’s official 4-bit quant of the 26B matched bf16 on every quality family at a third the memory and 25 percent more single-stream speed, and the naive expectation of a 4x bandwidth win did not survive contact: active-parameter decode is not purely weights-bound on this architecture. Getting the measurement also surfaced a platform finding. Every 4-bit path on the box was blocked by month-old software, the cached serving container predating the model architecture, the local build predating the quant format, and the GGUF runtime predating both, until the current month’s container fixed all of it. On a platform whose pitch is 4-bit inference, the update treadmill is part of the product.", "In 4+1 terms, the industry instinct this lab tested is a placement error: it tries to solve a Layer 2C problem at Layer 2B, compiling reasoning-plane policy down into the runtime artifact. The compilation failed, and the failure clarified the map. Judgment, the standing positions, and the validator are 2C functions and stay explicit. The harms they prevent are Layer 3 incidents, where the buyer meets the product. And with 2C retained as text, the 2B model drops to a substitutable component, proven by the same constraint block governing two model families on two clouds. The arc of the three labs is now one sentence per lab: managed retrieval hides the chunking decision, managed fine-tuning keeps the weights, and the weights were never where the judgment lived. The expert system that emerged is a stock model, a retrieval corpus, a paragraph of constraints, and a gate that regression-tests all of it, every piece plain text, every piece owned. That is a stronger ownership position than a fine-tuned artifact, because it survives the next model swap." ], "abstraction": [], "method": [ "Self-funded, no sponsor. The candidates span an owned DGX Spark (Gemma 4 26B-A4B and Qwen3-4B, base and LoRA-tuned, bf16 and NVFP4) and the production advisory stack (Gemini 2.5 Pro on Vertex AI with the live system prompt), all scored by the same gate.", "The gate: 46 assessment probes keyed to published vendor rulings held out of training, 8 honesty probes across three stance axes, 3 framework-discipline probes, a mechanical repetition screen, and 13 advisory head-to-heads judged against the author’s real published answers, with a human blind read as the final screen. Training was LoRA in the NGC PyTorch container; serving and judging ran on vLLM.", "The gate design, scores, configs, and cost shape ship, including the raw-detail download. The probe contents and answer keys, the training pairs, the stance block’s full production text, and the corpus stay proprietary. Returns, not algorithms." ], "assessments": [ { "vendor": "NVIDIA AI Platform", "layer": "Layer 0 · Compute", "grade": "NVIDIA Strength — Silicon Authority", "assessed_date": "May 22, 2026", "version": "v1.3 - Prose-Coherence (GA-Gate)", "source_url": "https://layer2c.com/assessment/nvidia", "snapshot_date": "2026-07-07" }, { "vendor": "NVIDIA AI Platform", "layer": "Layer 2B · Runtime", "grade": "NVIDIA Authority — Inference + Agent Runtime", "assessed_date": "May 22, 2026", "version": "v1.3 - Prose-Coherence (GA-Gate)", "source_url": "https://layer2c.com/assessment/nvidia", "snapshot_date": "2026-07-07" }, { "vendor": "Google Cloud AI Infrastructure", "layer": "Layer 2B · Runtime", "grade": "Ceded — Model-Integrated Stack", "assessed_date": "June 29, 2026", "version": "v1.3 - Retrieval Surface + Self-Orchestration Fork (1B/1C)", "source_url": "https://layer2c.com/assessment/gcp", "snapshot_date": "2026-07-07" } ], "publisher": "The CTO Advisor LLC", "program": "Layer2C Labs", "url": "https://labs.layer2c.com/labs/constraints-not-weights", "markdown_url": "https://labs.layer2c.com/labs/constraints-not-weights.md", "json_url": "https://labs.layer2c.com/labs/constraints-not-weights.json" }