Map vs. Territory

I was reading in Cybernetic Forests, when I came across this quote from "The Map is Eating the Territory" by Henry Farrell, and it's all very technology/AI/academic stuff, but boy, this quote:

To direct these politics, we need to know more about the underlying political economy. So here is my best stab at one aspect of what has been happening over the last couple of decades. Over this period, we have been seeing the rise of new technologies of summarization - technologies that make it cheap and easy to summarize information (or things that can readily be turned into information). As these technologies get better, the summaries can increasingly substitute for the things they purportedly represent.

Farrell is talking about of technopolitical stuff, but if you think of "new technologies of summarization" as, say, testing instruments used to summarize the whole world of student achievement, well, yes. Bullseye.

Test scores (especially when massaged by some cool math-flavored VAM sauce) are supposed to provide a map of the whole territory of student understand, their whole ability to read and math, We call scores above a certain point "Proficiency," as if the score of any student in that range is actually their street address in the land of reading or mathing skills, when in fact that score is more equivalent to a vague "over there" hand wave. 

But having convinced themselves that the test score is a very specific address on a very detailed map, education leaders start trying to change the map. "If I can get the map to show a different address for this student, then the student will move," goes the reasoning.

We mistake the map for the territory. That gives us the mistaken belief that knowing the map is knowing the territory, a problem that is only exacerbated if we hold onto the belief that the map is flawless. And from there we move on to the fallacy that changing the map is the same as changing the territory.

This map-and-territory idea is a close cousin of Campbell's Law:

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

Which is itself a close neighbor of Goodhart's Law:

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

They circle around the same insight for the test-centric approach to schooling-- the insistence that test data are a perfect representation, a perfect map of the student territory, gets us quickly to trying to manipulate and alter that map. If scores on the Big Standardized Test are a perfect map of student achievement, and we fixate on that map as if it were the territory, then we succumb to the fallacy that changing the scores (the map) will somehow change students' reality (the territory). 

It's not just that we have no real proof that the BS Test score map is a perfect representation of the territory. It's that we have absolutely no proof that changing the map changes the territory. IOW, we have no proof that if we get this roomful of students to score higher than the would have (thereby changing the map), they will be wealthier, happier, more successful than they otherwise would have been (changing the territory). 

I'm going to say that again, because it's important, Yes, we have oodles of proof that a high BS Test score correlates with certain conditions, like having a high-income family and desirable life outcomes. That correlation is solid. What we don't have is evidence that if we take students who were destined for a low BS Test score and get them to somehow get a higher score, those students will then achieve higher life outcomes than otherwise expected.

Once more. We've got guys like Raj Chetty who argue thus: Group A has high scores, and Group B has low scores. Group A will go on to make Big Bucks in life and Group B will make Fewer Bucks. What we don't have is any evidence that if we move someone from Group B to Group A, he will go on to make the Big Bucks.

The map is not the territory. It is a proxy, an attempt to represent the territory through a set of symbolic marks. But those symbolic marks, those summaries, those measures-- they are not the thing. In education, teachers live and work in the territory and not on the map--at least they should, as long as folks who can't tell the difference don't take control of the whole business.