Manufacturing discipline, not prompt engineering: poka-yoke, MSA, FMEA for AI code

agentic-gates blog · audience: developers, tech leads and software architects

The AI software world talks like a factory. Pipelines, gates, quality metrics, occasionally six sigma. The vocabulary of manufacturing quality has arrived in the tooling. The discipline behind it has not.

A real factory does not release a machine tool for series production because the operator says the part looks good. It runs a gauge and process capability study on standardized reference parts and measures whether the machine holds the tolerances that matter, under load. For an AI code builder, such studies are hard to find. People rely on a handful of green unit tests as the quality proof.

Green tests are no seal of quality for stochastic code and prompt engineering is why the industry keeps hoping they are. You cannot talk a stochastic model into deterministic reliability. Ninety-five percent right is fine for a chatbot and unacceptable for a build. The fix is process control, the same move a quality engineer already knows: you qualify a process, monitor it and bound it. Taken seriously rather than as a metaphor, that yields the following disciplines. Every one of them is running code, not slideware and every claim carries its evidence.

1. Poka-yoke: hard blocks and soft metrics

Classic poka-yoke, the Japanese method of mechanical mistake-proofing, keeps a faulty part from ever reaching the next station. For AI code that means splitting every rule into two strictly separated classes:

• Truth-valued rules (hard gates). Conditions with an unambiguous truth value. Does this import cross an architecture boundary, yes or no. A violation blocks the build, unconditionally.
• Shape proxies (soft metrics). Function length, nesting depth and their relatives are monitored continuously and never block the build.

Why this split matters is itself a measured result and a surprising one.

The announcement effect

We ran preregistered pilots (n = 3 per arm, one identical task) on where in the rule pipeline the gaming incentive acts. Announcing a blocking shape threshold before the agent drafts produced the gaming signature in all three runs and the gate never even fired, because the agent complied preemptively. Revealing the same rule only after a first draft, so the gate actually fires and forces repair, produced no gaming in any of the three. Announcing a truth-valued rule, one you can only satisfy and not game, was clean.

A note on scale, stated deliberately: our pilots show this effect consistently across three experimental designs at pilot scale. We say "our pilots show", not "proven".

2. Gauge study before you measure

Industrial metrology has a standing rule, the measurement system analysis: before trusting an instrument, prove that the instrument itself measures correctly.

To qualify our own checks we fault-injected them: 12 of 12 known answer cases caught, on independent re-validation. And we killed the fashionable instrument by its own preregistered criteria. LLM judges, asked to referee code quality pairwise, produced 100% unanimous verdicts that turned out to be position bias: they chose the first-shown option in 32 of 36 verdicts and debiasing across both orders collapsed their consistency to null. A negative result, published in full. It is the strongest reason to trust our positive numbers.

3. Industrial quality frameworks in the build

To move AI generation from the hobby bench into controlled production, the toolkit uses the established instruments of quality management:

• FRACAS (failure reporting, analysis, and corrective action system). An auto-collected list of escaped defects with their containment.
• COPQ (cost of poor quality). The realized cost per merged change, tracked over a shared run id.
• Control plans. Every check carries a named owner and a reaction plan for deviations.
• Preregistration. Every experiment freezes its design by commit hash before any data exists; thresholds are void without it. This discipline holds across all eleven entries of our experiment register.

Two honest boundaries

Engineering honesty includes naming the limits:

• Bench against live production. Capability numbers live on the bench with standard workpieces. We deliberately do not compute capability indices over live, one-off production.
• Proprietary thresholds. We publish the principles, data and methods in full; the concrete gate configuration, thresholds and security specifics stay internal for now.

Conclusion: AI development needs quality engineers

Six sigma was a discipline that made unreliable machines produce reliable parts. The newest machine on the floor is an AI code builder and it needs the same treatment.

Prompt engineering is dead. AI development needs quality engineers.

The free, open-core gates are here: agentic-gates