The Problem With Comparisons In Education

Which is the best movie: Ghostbusters, Singin' In The Rain, Casablanca, or Avengers: Endgame?

It depends, of course, on how we choose to compare them. Based on level of romance? On the happiest ending? Best dancing? Most money made? Best use of Sigourney Weaver? Criteria make all the difference. But it's not just the criteria; it's the problems with criteria that naturally emerge from the mandate to compare.

One of the driving features of modern education reform has been the mandate to compare. Fans of free market education want to be able to compare schools; several reform programs targeted schools that ranked in the bottom five percent. The New Teacher Project (TNTP) made a huge splash in 2009 with "The Widget Effect" arguing that we should compare teachers and make staffing and pay decisions based on the results. Ranking schools is as important to U.S. News as swimsuits are to Sports Illustrated.

There are problems applying comparisons to education.

Comparisons are not measurements. Pat may be ranked the tallest or shortest student in class, but either way, knowing Pat's ranking does not tell me how tall Pat actually is. "Most improved" may make good advertising copy, but your enterprise can be "most improved" and still be terrible.

This problem only increases as we deal with more complex systems. The better a measure is for making comparisons, the worse it is for actually describing the thing being measured.

If we want to describe what makes a particular school great, or where it is falling short of greatness, we have to talk about a complex web of factors in the school and community--everything from teacher content knowledge to curricular offerings to community socio-economic information to what the stakeholders in the community value and expect from their school, as well as a long-term look at what graduates of the system think five, ten, twenty, forty years later. The list of factors that describe a school, as well as those who work in it, is hundreds of items long.

But creating a clear comparison of thousands of schools based on a hundred-item list of factors is as impossible as ranking great movies. So people who want to compare schools have come up with various truncated lists, lists that are so simplified that they fail to provide any real picture of individual schools.

The answer for modern education reform has been to use standardized math and reading test scores as the measure of schools (and school districts and school teachers). This makes comparison easy because it narrows the long, long list of criteria down to just one. But one data point makes a lousy descriptor of an entire complex organization like a school. And this is a particularly lousy descriptor, because research shows again and again that test scores can be predicted by basic income and demographic data. Students from wealthy families get better test scores.

Worse, when we do a comparison based on a simplified single measure, we encourage folks to pursue that single measured quality. This is exactly what has happened in many schools. While stakeholders may care about the arts programs and school atmosphere and teacher experience and traditions that have made that school great, schools have been told that their greatness will be measured by test scores, and so elements from recess to history class have been dropped so that the school can focus on the single measure. (And for the moment, let's not even get started on the idea that VAM scores would allow us to compare a third grade phys ed teacher to an 11th grade history teacher.)

More complex measures of school quality are possible; Beyond Test Scores by Jack Schneider details the work done with such a model in Somerville, Massachusetts. Certainly many parents engage in informal complex assessments ("I like West Egg High School because the English teachers are really good, they have a great football team, the band is awesome, most of my family has gone there, it's a nice safe school, the principal is a great guy, and it's located close to our neighborhood") but those are as varied as all the stakeholders in a school district.

In the end, education reformers have to face a simple limitation--if an assessment tool is good for comparing and ranking schools, it is not a good tool for describing the strengths and weaknesses of that school. You can do one or the other, but not both. A tool for ranking schools (or teachers or districts) will not provide the information needed to strengthen and improve that school. When creating a tool, the very first question that must be addressed is which goal you want to achieve--do you want to compare schools, or do you want to help them improve. As the saying goes, repeatedly weighing the pig will not cause it to gain weight, but it will also not tell you whether or not the pig would make a good addition to your household.

Originally posted at Forbes.com