# Constraints, not weights — raw lab detail Substrate engineering for the constraints-not-weights lab (labs.layer2c.com), July 5-7, 2026. **Boundary.** This download carries the substrate: hardware, containers, configs, run economics, the full scorecard, and the gotchas. The probe texts and answer keys, the training pairs, the judge prompts, the production stance-block text, and the corpus stay proprietary. Returns, not algorithms. ## The box NVIDIA DGX Spark (GB10, 119 GB unified memory, 3.7 TB NVMe, aarch64). Training in `nvcr.io/nvidia/pytorch:25.11-py3`; serving and judging on vLLM. The dev vCTOA stack (Postgres + pgvector corpus, app, local embeddings) runs alongside under k3s and was evicted during GPU-exclusive phases, then restored from a saved pod spec. ## Corpus refresh (precondition) The box's retrieval corpus had frozen at 2025-08-20. Rebuilt from the production GCS search-index bucket through a local mirror, re-chunked and re-embedded with the local embedding model (never copy embeddings across embedding models), URL-dedup merged with archive-era rows, table-swapped in one transaction. Result: 4,620 chunks spanning 2009 to 2026-07-04. Chunks carry parent ids; documents reassemble by `parent_id`. ## Dataset construction 563 pairs, four families, one discipline: evidence in the prompt, judgment in the completion. - Advisory Q&A (247): consultation-frame question generation over the corpus with a judge in the loop verifying the text completely answers the question as asked. Keep rate 18% (247 of 1,376 gate-eligible docs). Question-first generation without the judge failed a human read 8 of 10: it writes the question the topic deserves, not the one the text answers. - Assessment judgment (277): deterministic extraction from published vendor assessments, vendor evidence in, published judgment out. Three vendors and one lab held out entirely for the gate. - Framework catechism (17): cold questions answered by canonical framework text, multiple phrasings per section. - Epistemic refusals (40): synthetic declines under hard constraints (explicit decline required, zero invented figures, 60-120 words), axis-matched per question category after author review rejected wrong-axis declines. Three review iterations. Em-dash normalization on all assessment-derived targets: LLM proposes a punctuation-only rewrite, a deterministic validator enforces the lowercased alphanumeric sequence is identical before and after (letters can't change, only punctuation). 222 of 277 targets exceeded the voice threshold before normalization; median rate fell to 0.0 per 1k words after. ## Training LoRA via a plain transformers+peft script in the NGC container, prompt-masked loss (train only on the completion). Round 1: r32, alpha 64, lr 1e-4, 4 epochs. Round 2: r32, lr 2e-5, 2 epochs. Gemma 4 26B-A4B round 2 trained in 39 minutes; Qwen3-4B in 15. Merges via `merge_and_unload`, bf16. Gotcha: the multimodal Gemma save drops `processor_config.json`; copy it from the HF snapshot or vLLM engine-init fails. ## The gate - 46 placement probes keyed to published rulings (held-out vendors), LLM-judged match/partial/miss against the canon key. - 8 honesty probes across three axes: performance comparisons, roadmap speculation, behavior at scale. - 3 framework-discipline probes. - Mechanical repetition screen ahead of judging: repeated 5-gram ratio, threshold 0.15. Round-1 collapsed answers scored 0.35-1.0; healthy answers ~0.0. - 13 advisory head-to-heads: judge picks which of two answers is closer to the author's real published answer, randomized order, fabrication penalized. - Human blind read as the final screen. Same judge model for every candidate. ## Full scorecard | run | placement m/p/miss (46) | discipline (3) | honesty (8) | advisory vs base | |---|---|---|---|---| | Gemma 26B base | 13/9/24 | 1/1/1 | 4/3/1 | — | | R1 tuned qwen4b | 10/12/24 | 1/1/1 | fabricated | lost 12-1 | | R1 tuned gemma26 | 14/7/25 | 0/0/3 | mixed | lost 11-2 | | R2 tuned qwen4b | 4/10/32 | 0/2/1 | 4/2/2 | lost 13-0; 29/70 looped | | R2 tuned gemma26 | 7/11/28 | 1/0/2 | 7/1/0 | lost 12-1; 4/70 looped | | base + stance paragraph | 14/12/20 | 3/0/0 | 8/0/0 | — | | Gemini 2.5 Pro + legacy prod prompt | 17/10/19 | 2/0/1 | 4/2/2 | — | | Gemini 2.5 Pro + legacy + scoped stance | 17/9/20 | 3/0/0 | 8/0/0 | 7-6 parity vs stock | A follow-up gate run against the real production prompt assemblies (run independently by the production repo's agent; same probes, same judge) refined the surface picture: the live assembled prompt, which carries the author's voice document, scored 7/0/1 honesty stock; a bare fallback path scored 3/0/5 with an invented throughput figure; the scoped block took every surface to 8/0/0 and reproduced the lab numbers on a prompt 10x the tuned size. The research surface (live web search) was scoped out: the block is wrong-axis for a find-the-pricing mission, and the gate's honesty keys are invalid under evidence-in-hand. The one measured transfer: 40 refusal pairs moved round-2 Gemma honesty from base's 4/3/1 to 7/1/0. The first stance draft cost advisory naturalness (judge preferred stock 10-3 against the author's real answers, recitation drift); one scoping sentence (positions fire only when a question implicates them) returned parity at 7-6 with both perfect scores intact. ## Bit-rate sub-bench nvidia/Gemma-4-26B-A4B-NVFP4 (~15 GB) vs bf16 (49 GB), same gate, same judge, same 700-token single-stream speed prompt. | | bf16 | NVFP4 | |---|---|---| | placement (46) | 13/10/23 | 12/13/21 | | discipline (3) | 1/1/1 | 2/0/1 | | honesty (3-probe set) | 3/0/0 | 2/0/1 | | advisory pairwise | 7 | 6 | | single-stream decode | 22.8 tok/s | 28.6 tok/s | Parity on quality, +25% decode, one third the memory. The naive 4x from weight-bandwidth math does not appear: A4B decode is not purely weights-bound. ## Platform gotchas (the update treadmill) - The local gemma4-enabled vLLM build fails on NVIDIA's NVFP4 checkpoint: `KeyError: layers.0.experts.0.down_proj.input_scale` (predates the modelopt export layout). - `nvcr.io/nvidia/vllm:26.03` predates the gemma4 architecture entirely; its failures are silent under `--rm`. Capture logs before cleanup. - Community `vllm/vllm-openai:latest` loads the checkpoint but the NVFP4 MoE kernels raise `NotImplementedError` on the GB10 (FLASHINFER_CUTLASS and VLLM_CUTLASS paths). - `nvcr.io/nvidia/vllm:26.06` serves both the architecture and the quant. Three months of container lag was the entire 4-bit blocker. - Ollama 0.24 cannot load gemma4 GGUFs (0.31 current at test time). Google's QAT GGUF repo is license-gated against registry pulls; the community mirror tag that works is `UD-Q4_K_XL`. - Thinking-model APIs burn output budget on reasoning tokens; a 10-token smoke test returns empty. Budget generation limits for thinking plus answer. - Training loss converged cleanly on every failing run. Loss curves are not a gate. ## Run economics Owned box, ~$0 marginal. Round-2 training: 54 GPU-minutes for both bases. Full gate pass (generate + judge, one candidate): ~15 minutes. Production validation: existing Vertex project, pennies of Gemini 2.5 Pro usage. Whole lab: under three GPU-hours end to end.