Goodhart's Law And The BS Test

When discussing the problems of test-based accountability, we've long used Campbell's Law as the go-to framer of the related problems. For the absolute top of the field, get a copy of The Testing Charade by Danielk Koretz. Campbell's law is not very pithy, but it illuminates beautifully:

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

Campbell was a social scientist, and though he died before the modern age of test-driven education really kicked into gear, he was still clear on the problems with the Big Standardized Test:

Achievement tests may well be valuable indicators of general school achievement under conditions of normal teaching aimed at general competence. But when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways. (Similar biases of course surround the use of objective tests in courses or as entrance examinations.)

That's pretty well it. The Big Standardized Tests (and this can be applied to the SAT and ACT as well) don't really tell us what they claim to tell us, and they've warped the whole process of education as well, from months of education sacrificed for test prep to students forced to drop other classes so that they can take "extra" test related classes to the sorting of students into categories-- we don't have to worry about them, these students are hopeless, and these students are close enough to the line that we will invest time and money in pushing them over it.

But as apt as Campbell's Law is, it involves a lot of discussing and explaining, and it doesn't fit easily on a t-shirt. So if you really want to make a short, pithy statement, may I suggest the slightly less well known Goodhart's Law as restated by Marilyn Strathern. Goodheart was an economist and critic of Margaret Thatcher's policies, which led him to this observation circa 1975:

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

In 1997, Strathern, an anthropologist, translated that from economistese into punchy English:

When a measure becomes a target, it ceases to be a good measure.

There we go.

Goodhart is a broader version of Campbell, while Campbell serves as a good explainer of how Goodhart can be true. But for civilians who are just catching on, Goodhart is brief and clear. This law, incidentally, is a fave in the data sciences world, whereas Campbell was more popular with social scientists. So all those data-driven decisions fans may well have already heard of it.

If you are measuring the output of your nail factory, that's useful. But when you start making a particular output a target, that's when you get a factory that produces only tiny nails because that's easier to do. If you measure how many customers your phone bank people talk to each day, that's useful. When you give them a target of so many customers per day to talk to, that's when you get customer service in brusque, one word, unhelpful but quick burst. And (one of my faves) if you measure how many shoes your soviet shoe factory makes, that's useful. But if you make that quantity a target, that's when you get a factory that turns out only left shoes, because that's more efficient.

When you use a standardized test to measure how students are doing, that might be useful for telling you at least a little about how well your class or your school are doing. When you require a school to hit certain targets, that's when you get-- well, you already know what we get. And you test scores no longer tell you anything useful.

So make a note. Goodhart's law is short, clear and will fit on a t-shirt. The kind of t-shirt you could wear to your next data driven decision making in-service, or your next test prep professional development session. It ma be time for me to go into the t-shirt business.