Critical document reviews with AI: how hardware engineers nip false figures in the bud

aisen blog · article 1 · audience: quality and hardware engineers

An example from practice: a sensor datasheet specifies a minimum operating voltage of 2.7 volts. In the AI summary for the design review the figure suddenly reads 3.3. Both numbers read as equally plausible while only one of them actually exists in the source.

Table 1: The datasheet against the AI summary, checked word for word.

This failure mode is not hypothetical. It has been measured:

• A manual evaluation of machine-generated summaries across all tested models found significant proportions of content with zero backing in the source text [1].
• A preliminary study on one state-of-the-art neural summarization system found fabricated facts in nearly 30 percent of its outputs [2].
• A comprehensive survey on hallucination in large language models describes data fabrication as an inherent feature of generative architecture: prompts can mitigate it, they cannot eliminate it [3].

1. Why a single wrong number decides in hardware design

An operating voltage range is a foundational design choice. Anyone reading 3.3 V instead of the true 2.7 V accidentally rules out an entire class of low-power designs. A single transposed digit in a 40-page document is enough to trigger a flawed decision. Catching it with the naked eye is nearly impossible. Reading those 40 pages is the exact work you offloaded to the AI in the first place.

Position is an additional risk factor. When the decisive tolerance sits in the middle of a long document, retrieval performance in language models degrades measurably [4].

A fluent summary is dangerous precisely because it sounds authoritative. The wrong value creeps into the meeting notes, into the review log and finally into the specification of the subassembly.

2. The one question that really protects your design

Before you trust an AI summary of a technical document, you need proof of exactly one thing:

Did every single value in the response originate verbatim from the source document?

How elegant the response sounds is irrelevant. What matters: Does every number, every unit and every part identifier match the source text word for word? This question can be mechanically verified. It is far more concrete than the vague question "is this summary good" and that precision is what makes it testable.

3. The verification path in three steps

Figure 1: The check as a picture: a claim with a quote passes, a claim without one is stopped.

Image generated by AI

We built a lightweight, free verification tool specifically to close this gap. The path:

1. Mandatory citations. Your prompt requires a verbatim quote from the document for every reported finding.
2. Generate the response. The AI delivers its points alongside the requested quotes.
3. Local audit. A script checks every quote directly on your machine against the original file. Every verified point returns as ACCEPTED. Every point without an exact match returns as REJECTED.

In the opening example the fabricated 3.3-volt figure is caught instantly: REJECTED. The string exists nowhere in the file. The source text reads 2.7 without change.

4. Transparency: what the tool does and what it does not

The tool serves a clearly defined purpose and verifies only one thing: are the cited values real? Nothing more, nothing less:

It eliminates the single hardest silent error to catch manually: the altered or hallucinated value. Before you sign off on an AI-reviewed document, those two minutes are time well spent.

Try it on your own machine

Download the tool free of charge, it runs locally on your computer. Test your next AI summary directly against its source document: agent-quality-control

Sources

[1] J. Maynez, S. Narayan, B. Bohnet, and R. McDonald, "On Faithfulness and Factuality in Abstractive Summarization", Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, arXiv:2005.00661.

[2] Z. Cao, F. Wei, W. Li, and S. Li, "Faithful to the Original: Fact Aware Neural Abstractive Summarization", Proceedings of the AAAI Conference on Artificial Intelligence, 2018, arXiv:1711.04434.

[3] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, "A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions", 2023, arXiv:2311.05232.

[4] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, "Lost in the Middle: How Language Models Use Long Contexts", Transactions of the Association for Computational Linguistics, vol. 12, 2024, arXiv:2307.03172.