Skills, prompts, hacks: all of it is useless without automatic checks
Dr. Aaron Hutzler · 18 August 2026 · 5 min
Eleven AI tip charts came through my feed in a single week. formulas with four building blocks. Skill . A wheel of 50 hacks. I put them side by side and counted what they teach. Then a second stack arrived. The corpus grew to 39 charts without duplicates.
1. What the cheat sheets optimise
Nearly all of them answer the same question: how do I get a better first draft. Assign a role. Add context and constraints. Save the instructions for reuse. Pick the right model for the task. This advice is usable. It raises the average quality of what the machine hands you.
An engineer asks a second question before shipping anything: how do I make sure a bad draft never reaches a customer? So I counted the answers to that one too. The rule for the count: a passage qualifies only when a machine runs the check. That means a test, a or a score with a pass threshold. Rereading the output by hand does not qualify: no program delivers a verdict there.
Figure 1: The count over all 39 charts. Four of the seven with a check come from coding or A/B testing.
The result, plotted in Figure 1: 7 of the 39 charts mention a machine-run check at all. Four of the seven target software engineers or A/B testing. Most mentions are one or two lines, ranked equal to tips like choosing an output style. Exactly one chart treats the check as the foundation: an infrastructure guide where every step ends in a verify command and a human approval. One chart out of all 39 charts.
2. Why that ratio is backwards
A better prompt raises the average draft. The expensive failures do not live in the average. They live in the outliers: the invented number, the confident wrong claim, the broken house rule on page 3. An average does not catch an outlier. A check does. It looks at every single output instead of the typical one. The gauge in Figure 2 does exactly that job.
Figure 2: One crumpled sheet in a row of clean ones. The lamp turns exactly there.
Image generated by AI
Two published measurements show why the input side cannot carry the guarantee alone. One study varied surface features of a prompt that did not change the underlying task and measured the accuracy of the same models on the same tasks: the results moved by up to 76 percent [1]. A second study tested the same AI service twice, three months apart, on identical tasks. On one evaluated task in that study, accuracy fell from 84 percent to 51 percent. Nobody had touched the prompt [2]. A channel this unstable can improve your draft. It cannot certify it.
3. What a check is
A check is a program that ends in a clear pass or fail. It reads the output and tests a defined rule. It applies no judgement and no mood. It grants no benefit of the doubt on draft forty.
Our own marketing runs this way. This article went through it. Every text has a machine-readable spec naming the single claim it makes. Every number in the text is bound to a verbatim quote from its source. On the day of the count the register of that repository listed 38 checks, 31 of them blocking. A text reaches the owner for release only when the whole chain is . My own opinion of a text carries exactly as much weight as the scripts grant it, which is none.
4. The question to take away
The next time a tip chart scrolls past, one filter is enough. Does any line say who checks the output? And is that someone a program? Where the answer is no, the chart teaches better drafts, not trustworthy results. Would your best AI output survive a script that checks every number against its source?
5. Where to go deeper
The pattern has a name in the industry: evaluations, evals for short. A survey from 2023 maps the research field and sorts the methods by what they test and how [3]. The big model vendors recommend the practice themselves. One publishes a guide for building empirical evaluations before a prompt goes to production [4]. Another maintains an for registering and running exactly such tests [5]. For a first own gate in a build , the open-source tool promptfoo runs assertions against model outputs in . It runs there the same way run against code [6].
6. Sources
[1] M. Sclar, Y. Choi, Y. Tsvetkov and A. Suhr, "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design", ICLR 2024, arXiv:2310.11324. https://arxiv.org/abs/2310.11324
[2] L. Chen, M. Zaharia and J. Zou, "How Is ChatGPT's Behavior Changing over Time?", arXiv:2307.09009, 2023.
[3] Y. Chang et al., "A Survey on Evaluation of Large Language Models", ACM Transactions on Intelligent Systems and Technology, arXiv:2307.03109, 2023.
[4] "Create strong empirical evaluations", vendor documentation, 2025. https://docs.anthropic.com/en/docs/test-and-evaluate/develop-tests
[5] "Evals", open-source evaluation framework, vendor repository, 2023. https://github.com/openai/evals
[6] "promptfoo", open-source testing tool, 2025. https://www.promptfoo.dev/