Almost unlocking a mystery of SAT scores

So close. Sooooo close!!

Covering the release of SAT scores this year, WTOP news noted two results and put them right in the headline: 

Va., Md. SAT scores rise, but number of test-takers plunges

It's the "but" that tells us that somehow, they didn't manage connect the dots, which is a shame, given there were only two.

Of course, they mean average test scores rose, and understanding averages is important here. It has always been important when reading the latest stories about SAT scores.

Any high school teacher can tell you-- "Let me pick who gets to take the SAT, and I can give you dramatically higher test scores in just a single year!" Because averages.

This year we measured only 5th and 6th graders for height, instead of K-6, and we find the average height of children in our school has increased. We did a survey of car prices; last year we surveyed all dealers, but this year we surveyed only Lexus dealers, and we find that the average price of a car has gone up. 

We've spent years marketing the SAT's flagship product to all students (in some states, we've snookered the government into requiring it), so as we add students who might not have been inclined to take the test to go ahead and take it, the average score is affected. That's how the results for every sub group can go up even as the overall average goes down. 

Now throw in a pandemic year in which students who are having a rough time just don't take the test, leaving it only to those who are well-buffered from the pandemic (and whose buffering is the same sort of socio-economic background that is an advantage on the test) and voila!! Instant increase in average score.

What we have here is just one more example of why test scores from the pandemic are not worth a thing. They can't be compared to any year, they can't really be normed accurately, and they just kind of mean nothing. But we're still going to be subjected to stories that can't manage to draw a line between two data points.