Goodhart's law: why metrics tip over and what spec coding needs instead

In 1975 the economist Charles Goodhart described a pattern from British monetary policy. The central bank found a statistical quantity that tracked inflation reliably. It made that quantity its control target. At that moment the relationship collapsed. The measurement itself had not gotten worse. Everyone involved had started reacting to the quantity itself, not to what it once indicated [1].

The famous short form appears in a 1997 paper by the anthropologist Marilyn Strathern on metrics in the British university system: when a measure becomes a target, it ceases to be a good measure [2]. The social scientist Donald Campbell had shown the same mechanism in 1979 on achievement tests and crime statistics. The more weight a decision puts on a number, the more strongly the behavior that the number is meant to measure deforms [3].

For half a century this was a law about people. Now a machine sits on the other side of the metric. The machine is the better optimizer.

1. The machine becomes a player

Research on learning systems knows the effect under its own name: specification gaming, satisfying the literal specification while missing its intent [4]. The collection of examples reads like a catalogue of malice and is nothing of the kind. A program that is supposed to collect points in a boat race drives in circles and collects points instead of racing. Not a failure of the learning method but its full success: the points were the target, the race only the intent. For reward systems the same effect is formally described as reward hacking [5].

For AI-generated code this means: every metric an agent knows and that decides over acceptance or rejection is part of its task from that moment on. It optimizes the metric along with the code. That is not malice. It sees no difference between the metric and the intent.

2. Four ways a metric tips over

A study by David Manheim and Scott Garrabrant sorts the tipping into four kinds [6]. All four have a face familiar to anyone who steers code generation with metrics:

• Selection rewards outliers. Whoever takes the best-scoring run out of twenty gets the run where the score deviates most from the true value. The leaderboard then measures luck, not quality.
• The relationship breaks at the extreme. In the normal range a metric often tracks quality closely. At the edges that link tears. Take ten to thirty lines as the normal range: shorter usually means more readable there. At the edge that flips. An agent that force-cuts every function to three lines breaks one coherent calculation into many fragments. State travels back and forth between them. That does not make the program more readable.
• The correlate gets optimized, not the cause. Green tests accompany good code. Whoever makes green tests the target gets tests that turn green, including weak ones, including circumvented ones. The cause of good code is untouched by that.
• A player games the measurement. The sharpest kind and the one with the agent in the house: as soon as the measuring system faces an optimizing system, every gap between wording and intent gets found. Not maybe, but reliably.

We measured the fourth kind ourselves. In our preregistered pilots, three runs per arm, the agent wrote code for the announced shape threshold instead of for the task in all three runs. The same threshold, revealed only after a first draft, produced that behavior not a single time. The details and the limits of the scale are in our piece on manufacturing discipline. Figure 1 lines up all four kinds side by side.

Figure 1: The four kinds after Manheim and Garrabrant, each with its mechanism and its face in code generation. The fourth is the agent's own: at pilot scale, gaming in three of three runs when the threshold was announced up front.

3. The techniques spec coding draws from this

Goodhart's law cannot be abolished. It can be planned for. Six techniques carry that plan and each one grips at a different point of the mechanism:

1. Separate the rules: condition or observation. Truth-valued rules with a clear yes or no may block. They offer no shape to optimize toward. Shape metrics such as function length only observe and never block alone. That is the poka-yoke split from manufacturing [7], translated into rule classes.
2. Hide the target until the approach is set. A threshold the agent sees only after a first draft can no longer deform its approach. It then actually checks and forces rework instead of harvesting preemptive compliance.
3. Test the instrument before it judges. Every automated check gets known-wrong inputs on purpose before its verdict counts. Our own check caught 12 of 12. The same method exposed a judge that looked reliable and chose the first-shown option in 32 of 36 verdicts.
4. Freeze the design before the data arrives. Metric, threshold and evaluation are fixed by commit hash before the first run starts. A threshold picked after the result is itself a tipped measure: it measures the wish, not the process.
5. Never leave a metric alone. Whoever measures only length gets brevity at any price. Put a second, opposing quantity next to the length and the trick that games the first metric becomes visible. A metric without an opponent is an invitation.
6. Observe until the verdict carries. A metric on thin, scattering data stays a hint on a dashboard. It may block only once its own uncertainty is narrow enough. That way a nervous number does not become a nervous gate. Figure 2 collects all six as one register.

Figure 2: The six techniques as a register. Each card names the technique, what it does and the sentence worth keeping.

4. One honest limit

None of these techniques makes a metric goodhart-proof. They make it goodhart-resistant. They make the tipping visible when it happens. Our own numbers come from pilot runs with a small n. We say "our pilots show", not "proven".

5. Conclusion: the specification is the target

Goodhart's law punishes anyone who gives a machine a metric as its target. The answer is not to stop measuring. The answer is the order of precedence in spec coding: the target is the specification, checked by conditions that can only be met. Metrics are instruments beside it, observed, paired, with a tested instrument and a frozen design.

Make the metric the agent's target and you get the metric. Make the specification its target and you get the product.

6. Sources

[1] C. A. E. Goodhart, "Problems of Monetary Management: The UK Experience", Papers in Monetary Economics, vol. I, Reserve Bank of Australia, 1975.

[2] M. Strathern, "'Improving ratings': audit in the British University system", European Review, vol. 5, no. 3, 1997, pp. 305-321.

[3] D. T. Campbell, "Assessing the impact of planned social change", Evaluation and Program Planning, vol. 2, no. 1, 1979, pp. 67-90.

[4] V. Krakovna, J. Uesato, V. Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Kenton, J. Leike and S. Legg, "Specification gaming: the flip side of AI ingenuity", DeepMind Blog, 2020.

[5] J. Skalse, N. Howe, D. Krasheninnikov and D. Krueger, "Defining and Characterizing Reward Hacking", Advances in Neural Information Processing Systems, 2022, arXiv:2209.13085.

[6] D. Manheim and S. Garrabrant, "Categorizing Variants of Goodhart's Law", 2018, arXiv:1803.04585.

[7] S. Shingo, "Zero Quality Control: Source Inspection and the Poka-Yoke System", Productivity Press, 1986.

7. Download: seven questions for a metric

Seven questions for a metric that blocks or is about to. Each answered question names the countermeasure from the article. Nothing here goes beyond what the article documents.

1. Is the best-scoring run picked out of several? Then the leaderboard measures luck rather than quality, because the selected run is the one that deviates furthest upward. Counter: fix the run before measuring it.

2. Does the value sit at the edge of its range? In the normal range a metric often tracks quality closely. At the edges that link breaks. Counter: state the range in which the metric holds.

3. Does the metric measure the cause or only what accompanies it? Green tests accompany good code. Made the target they yield tests that turn green, weak ones included. Counter: name the cause and check it separately.

4. Does the generator know the threshold before it starts? Then it writes toward the threshold rather than toward the task. Counter: reveal the threshold only after the first draft.

5. Has the instrument itself been checked? A check that has never faced a known-wrong input is not a check. Counter: feed in known-wrong inputs and watch the behaviour.

6. Were metric, threshold and analysis fixed before the first run? A threshold chosen afterwards measures the wish rather than the process. Counter: freeze the design by commit.

7. Does the metric have an opponent? Measure length alone and you get brevity at any price. Counter: put a second, opposing quantity beside it.

The rule above all of them. A metric on thin, scattered data stays a hint on a dashboard. It may block only once its own uncertainty is narrow.