Really Rethinking Standardized Test Scores

Corey DeAngelis, fellow at Cato, turned up at Centre for Education Economics Ltd (CfEE) with an article (since reprinted at Cato's own site) that gets on the Dump Standardized Testing Train. And while I welcome "Rethinking Standardised Test Scores," it suffers from some of the same problems as other reassessments of test-centered education.

DeAngelis, it should be noted, is no friend of public education-- we last met him on this blog when he was arguing that schools should be owned by corporations-- and his background runs from Heartland Institute policy advisor to Risk Management Operations Coordinator for Kohl's. And while his reassessment of testing is welcome, he's going to miss some critical points in the discussion.

He signals one of his omissions in the very first sentence:

Standardised test scores have long been treated as the end-all-be-all of education.

Oh, passive voice, friend of the prevaricator and weasel word aficionado. Who, exactly, has treated test scores as the be-all and end-all? Because such treatment is neither universal nor mysterious and organic, without discernable cause. He does finger "researchers" and "the public at large," but does not acknowledge the many choices made by policy makers and test manufacturing lobbyists and various other powerful education amateurs to place the Big Standardized Test scores at the center of modern education.

Because-- and here's the thing-- you know who didn't want to make test scores the be-all and end-all, who argued against making them the be-all and end-all, who fought and still fight against making test scores the be-all and end-all?

Teachers. Actual professional educators.

DeAngelis is going to take us to a study published by the American Enterprise Institute in March with the title "Do Impacts on Test Scores Even Matter?" This meta-study comes from an interesting angle-- why is it that school choice hasn't caused giant leaps forward in achievement? The answer-- because choice based on the premise that people would choose the best school, but "best school" would be defined by "best test scores" and what happens if that's actually not a great measure of a good school?

The study, by Michael McShane and Patrick Wolf, is pretty plainspeaking. For twenty years "almost every major education reform has rested on a common assumption: standardized test scores are an accurate and appropriate measure of success and failure." Such scores are "convenient," "easier and cheaper" to collect and use. And there was that cool Raj Chetty research that everyone likes to cite even though there are good reasons to believe it's all bunk. McShane and Wolf don't call Chetty bunk, but they do refer to the "supposed truth" in it.

McShane and Wolf also discover that teacher impacts on test scores don't seem to correlate with much of anything, and maybe they aren't very good tools for policy when it comes to teacher pay, retention, etc. So much for the days when reformsters would bemoan how few schools were using VAM data for personnel decisions. But it's the relationship between students and test scores that really bears examining:

For research on test scores to actually be meaningful, the following should be true: The impacts that schools have on math and reading skills will change the trajectories of children’s lives. Otherwise, why would policymakers and researchers put such emphasis on “student achievement” and “student growth”—measures that are based on test scores

This assumption seems uncontroversial. 

Does it seem uncontroversial? Because it seems to me that for twenty years, one group of people has been trying to point out that test scores are NOT a good measure of student achievement, and that test scores correlate with later achievement because test scores correlate with socio-economic status, and so does life achievement. One group has said repeatedly that it is, well, not so much controversial, as just wrong.

Teachers. Actual professional educators.

The AEI report reaches a useful conclusion:

Policymakers need to be much more humble in what they believe that test scores tell them about the performance of schools of choice.

DeAngelis wants to make the additional point that character education is more important than test scores for life achievement. Hard work and respect-- that sort of thing.

He also nods to the work of Jay Greene (University of Arkansas, where DeAngelis has also done some brainwork) who has been way out in front of other reformsters in noting that test scores aren't very useful tools for changing students' futures.

It is a great thing that more and more people are catching on to the fact that the BS Test is not useful, not valid, not measuring much of anything worth knowing, and most definitely not a reliable proxy for student educational achievement.

But there are other important lessons to be learned here, and I don't see any hints that people are even close to learning them.

1) How Did We Get Here?

In all the debunking of test scores, I don't see anyone saying, "We did this."

This goes back to the days when Arne Duncan would say, "Boy, you guys are spending too much time on testing. Where'd you get the idea that testing was such a big deal, anyway?" As if he hadn't personally pushed test centric policies.

To hear reformed reformsters talk, one would assume that tests simply wandered into schools and took over without any help from policymakers, lobbyists, politicians, and rich private self-appointed school-fixers.

It's not that I want to assign blame. It's that I want these education movers and shakers to think about how they got us here so they don't keep doing it. As Daniel Koretz says about Common Core in The Testing Charade:

It's not just the Common Core that has been dropped into schools wholesale before we gathered any evidence about impact; this has been true of almost the entire edifice of test-based reform, time and time again. I'll argue later that putting a stop to this disdain for evidence-- this arrogant assumption that we know so much that we don't have to bother evaluating our ideas before imposing them on teachers and students-- is one of the most important changes we have to make.

Reformsters need some humility. They need to stop assuming that their ideas are so awesome that they don't need to be tested or even, in some cases, explained-- just implemented, quickly and without time for discussion (remember "our children can't afford to wait for us to change things"). And the number one thing that reformsters have aggressively refused to do...?

2) Listen To Teachers, Dammit

There isn't a thing that reformsters are figuring out today that professional educators haven't been saying for twenty years. But reformsters have remained steadfast in their belief that they have nothing to learn by listening to classroom teachers. David Coleman was just one of many reformsters who believed that his lack of teaching and education background made him more qualified than professionals who had devoted their adult lives to education.

As reformster after reformster has written a "Hey, I just figured out this policy doesn't work" piece, I've written over and over some variation of "also, water is wet" or "no shit, Sherlock" or "we've been trying to tell you this for years."

Yet this lesson-- listen to teachers-- hasn't penetrated much at all.

Those Who Don't Learn From the Past...

Just watch. So many reformsters are lined up behind some version of Personalized [sic] Learning, even though it has no basis in real research, has produced no positive results where tried, and has a chorus of teachers hollering "this is a really bad idea." But instead of saying, "Hey, you know, the last time the stars lined up like this, it didn't work out for us," reformsters are saying, "Oh, shut up. We've totally got this."

It's great that we're learning that the BS Test scores are bunk. It would be greater if folks pursued undoing these policies with the same zeal with which they pursued installing them (one more reason that accepting responsibility would be nice). But still, baby steps in the right direction.

It would also be greater if reformsters learned some of the other lessons that come with failed reform strategies. Just go sit the corner for a few minutes and think about what you've done before you head out to do something else.