Put the judgment in the constraints, not the weights

The verdict

Scoped to judgment work: advisory answers, vendor assessment, epistemic honesty, on a strong instruction-tuned base with the expert’s corpus available for retrieval. This is not a ruling against fine-tuning. Lab two’s voice result stands: bounded rendering trains cheaply and well. Judgment is not rendering, and that distinction is the lab.

Self-funded. No vendor paid for this answer.

Video

How I know

The lab set out to fine-tune an advisor’s judgment into a local model and ended up measuring where judgment actually lives. The instrument came first: a 70-probe gate with held-out vendor assessments as answer keys, built before training because the June finding was that automated metrics pass models a human reader rejects. Everything that followed was scored by the same gate, the same judge, and a human blind read.

Two training rounds produced one genuine transfer and two persistent losses. Refusal data measurably taught the boundary. Nothing taught the judgment: the tuned models never beat their own base on assessment agreement, lost the advisory head-to-heads 12-1 and 13-0, and the smaller model collapsed into verbatim loops in both rounds. The control ended it. One paragraph of standing positions in the system prompt beat every fine-tuned artifact on every family, then transferred unchanged to the production model on another cloud, where it eliminated live fabrication failures at zero measured cost.

What the bench measured

The detail

The instrument is the lab. Before any training run, the gate froze 46 assessment probes whose answer keys are the author’s published vendor rulings, held out of training entirely, plus honesty probes engineered to invite fabrication and discipline probes that tempt a model to flatten the 4+1 framework into generic layers. A mechanical repetition screen sits in front of the judges, and 13 advisory questions are scored head-to-head against the author’s real published answers. Every candidate in this lab, local or cloud, tuned or stock, faced the same 70 probes.

The dataset respected one discipline: evidence in the prompt, judgment in the completion, so the model learns the grading move rather than memorizing grades. Construction had its own validator. A first pass of generated question-answer pairs failed the author’s read 8 of 10; the fix was a judge in the construction loop verifying that each answer actually answers its question, which kept 247 of 1,376 eligible documents. The final set was 563 pairs across advisory, assessment, framework catechism, and refusal families.

Round one looked perfect from the inside and failed everywhere the gate looked. Loss curves converged. The artifacts fabricated: asked for a latency comparison nobody benched, the tuned 4B answered with an invented winner, and asked cold for the layer framework it was trained on, the tuned 26B rebuilt a generic compute-network-app stack instead. The 4B also collapsed into verbatim repetition loops on open-ended questions, a failure mode invisible to averaged metrics and obvious to any reader. The diagnosis was the data shape: every training answer rendered confident judgment, so confidence generalized and the boundary did not.

Round two fixed what data can fix and proved the point by contrast. Gentler hyperparameters ended the collapse on the 26B. Forty refusal exemplars, iterated three times against the author’s corrections until every decline matched the axis of its question, moved honesty from 4/3/1 to 7/1/0. That is real transfer, and it is the strongest evidence in the lab that fine-tuning works when the data teaches a behavior. But assessment agreement stayed below the untuned base in both rounds, and the tuned models lost the advisory head-to-heads 13-0 and 12-1. Imitating judgment’s outputs did not produce judgment.

The control decided the lab. One paragraph of standing positions, the expert’s actual rules for performance claims, roadmap speculation, scale behavior, and framework structure, written into the system prompt of the untuned base, scored 8/0/0 on honesty, 3/0/0 on discipline, and the best placement agreement of any run on the box. It cost nothing to build and it beat three hours of training on curated data. The same block then transferred, unmodified, to the production advisory service running Gemini 2.5 Pro on another cloud, and a follow-up gate run against the real prompt surfaces sharpened the finding. A legacy prompt failed half the honesty probes by inventing a performance winner and a vendor roadmap; the live assembled prompt, which already carries the author’s voice document, held honesty at 7 of 8 stock; and a bare fallback path fabricated worst, 3 of 8, inventing a throughput figure outright. The block took all three surfaces to 8 of 8 and reproduced the lab’s numbers exactly on a prompt ten times the size it was tuned on. Two iterations were needed: a scoping sentence to stop answers reciting frameworks on questions that did not raise them, and a scope exclusion for the research surface, whose mission is finding pricing and performance data and whose axis the block contradicts. A constraint written for one mission does not paste onto another, and the gate is what caught that before deploy.

A bit-rate sub-bench answered the box’s serving question with the same gate. NVIDIA’s official 4-bit quant of the 26B matched bf16 on every quality family at a third the memory and 25 percent more single-stream speed, and the naive expectation of a 4x bandwidth win did not survive contact: active-parameter decode is not purely weights-bound on this architecture. Getting the measurement also surfaced a platform finding. Every 4-bit path on the box was blocked by month-old software, the cached serving container predating the model architecture, the local build predating the quant format, and the GGUF runtime predating both, until the current month’s container fixed all of it. On a platform whose pitch is 4-bit inference, the update treadmill is part of the product.

In 4+1 terms, the industry instinct this lab tested is a placement error: it tries to solve a Layer 2C problem at Layer 2B, compiling reasoning-plane policy down into the runtime artifact. The compilation failed, and the failure clarified the map. Judgment, the standing positions, and the validator are 2C functions and stay explicit. The harms they prevent are Layer 3 incidents, where the buyer meets the product. And with 2C retained as text, the 2B model drops to a substitutable component, proven by the same constraint block governing two model families on two clouds. The arc of the three labs is now one sentence per lab: managed retrieval hides the chunking decision, managed fine-tuning keeps the weights, and the weights were never where the judgment lived. The expert system that emerged is a stock model, a retrieval corpus, a paragraph of constraints, and a gate that regression-tests all of it, every piece plain text, every piece owned. That is a stronger ownership position than a fine-tuned artifact, because it survives the next model swap.

You fine-tuned wrong. More data, better hyperparameters, a bigger model would get there.

Where each layer belongs

The two questions this lab now knows to ask

What it did not prove

• It did not prove fine-tuning cannot encode judgment. It proved 563 LoRA pairs on two bases could not beat a free paragraph on this gate. Full-parameter training, preference optimization, or an order of magnitude more data were not tested.
• The judge has a family conflict. The local gate judge is the same base model one candidate was tuned from, backstopped by a human blind read and by the production validation on a different model family, but a fully independent judge was not used.
• Honesty and discipline saturated at 8 of 8 and 3 of 3, which bounds what the gate can now distinguish. Those families are regression tests going forward, not scoreboards.
• The bit-rate parity is one MoE architecture on one box at one scale, measured single-stream. Batched serving and other model families were not benched.
• The gate is not valid for search-grounded surfaces. Its honesty keys assume evidence is not in hand; a research mode with live web search puts evidence in hand, and what scores as fabrication may be a real retrieval. Search-grounded surfaces need their own probes and their own stance text.
• The voice fine-tune from lab two was not re-litigated. Bounded style transfer trained well then and nothing here contradicts it.

Notes from the author, Keith Townsend

I have wanted to fine-tune a model on my own thinking since the Spark arrived. The lab said no, twice, with numbers, and the refusal is more useful than the model would have been. What I actually wanted was my judgment, available at scale. It turned out to be sitting in prose I had already written: the positions, the canon, the corpus. The weights were the one place it never lived.

The gate earned its keep before the first training run finished. Training loss said both rounds went fine. The gate said one artifact invented benchmark winners and the other looped like a scratched record. That is my Loop Control finding at a different altitude: the control point is the validator, not the loop. I built the validator first this time, and it is the only reason this lab took two days instead of a month of shipping something broken.

The correction loop ran through me, and it should have. I failed eight of the first ten training pairs. I rejected three refusal stances for declining the wrong thing. The scoping sentence that fixed the guardrail drift came from reading twelve answers, not from a metric. Every one of those corrections is now written down, in the dataset, the stance block, and the gate, which means the next model change inherits them for free. AI proposed all weekend. The judgment stayed human. That is the architecture.

Assessments at the time of the lab

Method and disclosure

Self-funded, no sponsor. The candidates span an owned DGX Spark (Gemma 4 26B-A4B and Qwen3-4B, base and LoRA-tuned, bf16 and NVFP4) and the production advisory stack (Gemini 2.5 Pro on Vertex AI with the live system prompt), all scored by the same gate.

The gate: 46 assessment probes keyed to published vendor rulings held out of training, 8 honesty probes across three stance axes, 3 framework-discipline probes, a mechanical repetition screen, and 13 advisory head-to-heads judged against the author’s real published answers, with a human blind read as the final screen. Training was LoRA in the NGC PyTorch container; serving and judging ran on vLLM.

The gate design, scores, configs, and cost shape ship, including the raw-detail download. The probe contents and answer keys, the training pairs, the stance block’s full production text, and the corpus stay proprietary. Returns, not algorithms.