Wednesday, June 20, 2018

Really Rethinking Standardized Test Scores

Corey DeAngelis, fellow at Cato, turned up at Centre for Education Economics Ltd (CfEE) with an article (since reprinted at Cato's own site) that gets on the Dump Standardized Testing Train. And while I welcome "Rethinking Standardised Test Scores," it suffers from some of the same problems as other reassessments of test-centered education.

DeAngelis, it should be noted, is no friend of public education-- we last met him on this blog when he was arguing that schools should be owned by corporations-- and his background runs from Heartland Institute policy advisor to Risk Management Operations Coordinator for Kohl's. And while his reassessment of testing is welcome, he's going to miss some critical points in the discussion.

He signals one of his omissions in the very first sentence:

Standardised test scores have long been treated as the end-all-be-all of education.

Oh, passive voice, friend of the prevaricator and weasel word aficionado. Who, exactly, has treated test scores as the be-all and end-all? Because such treatment is neither universal nor mysterious and organic, without discernable cause. He does finger "researchers" and "the public at large," but does not acknowledge the many choices made by policy makers and test manufacturing lobbyists and various other powerful education amateurs to place the Big Standardized Test scores at the center of modern education.

Because-- and here's the thing-- you know who didn't want to make test scores the be-all and end-all, who argued against making them the be-all and end-all, who fought and still fight against making test scores the be-all and end-all?

Teachers. Actual professional educators.

DeAngelis is going to take us to a study published by the American Enterprise Institute in March with the title "Do Impacts on Test Scores Even Matter?" This meta-study comes from an interesting angle-- why is it that school choice hasn't caused giant leaps forward in achievement? The answer-- because choice based on the premise that people would choose the best school, but "best school" would be defined by "best test scores" and what happens if that's actually not a great measure of a good school?

The study, by Michael McShane and Patrick Wolf, is pretty plainspeaking. For twenty years "almost every major education reform has rested on a common assumption: standardized test scores are an accurate and appropriate measure of success and failure." Such scores are "convenient," "easier and cheaper" to collect and use. And there was that cool Raj Chetty research that everyone likes to cite even though there are good reasons to believe it's all bunk. McShane and Wolf don't call Chetty bunk, but they do refer to the "supposed truth" in it.

McShane and Wolf also discover that teacher impacts on test scores don't seem to correlate with much of anything, and maybe they aren't very good tools for policy when it comes to teacher pay, retention, etc. So much for the days when reformsters would bemoan how few schools were using VAM data for personnel decisions. But it's the relationship between students and test scores that really bears examining:

For research on test scores to actually be meaningful, the following should be true: The impacts that schools have on math and reading skills will change the trajectories of children’s lives. Otherwise, why would policymakers and researchers put such emphasis on “student achievement” and “student growth”—measures that are based on test scores

This assumption seems uncontroversial. 

Does it seem uncontroversial? Because it seems to me that for twenty years, one group of people has been trying to point out that test scores are NOT a good measure of student achievement, and that test scores correlate with later achievement because test scores correlate with socio-economic status, and so does life achievement. One group has said repeatedly that it is, well, not so much controversial, as just wrong.

Teachers. Actual professional educators.

The AEI report reaches a useful conclusion:

Policymakers need to be much more humble in what they believe that test scores tell them about the performance of schools of choice.

DeAngelis wants to make the additional point that character education is more important than test scores for life achievement. Hard work and respect-- that sort of thing.

He also nods to the work of Jay Greene (University of Arkansas, where DeAngelis has also done some brainwork) who has been way out in front of other reformsters in noting that test scores aren't very useful tools for changing students' futures.

It is a great thing that more and more people are catching on to the fact that the BS Test is not useful, not valid, not measuring much of anything worth knowing, and most definitely not a reliable proxy for student educational achievement.

But there are other important lessons to be learned here, and I don't see any hints that people are even close to learning them.

1) How Did We Get Here?

In all the debunking of test scores, I don't see anyone saying, "We did this."

This goes back to the days when Arne Duncan would say, "Boy, you guys are spending too much time on testing. Where'd you get the idea that testing was such a big deal, anyway?" As if he hadn't personally pushed test centric policies.

To hear reformed reformsters talk, one would assume that tests simply wandered into schools and took over without any help from policymakers, lobbyists, politicians, and rich private self-appointed school-fixers.

It's not that I want to assign blame. It's that I want these education movers and shakers to think about how they got us here so they don't keep doing it. As Daniel Koretz says about Common Core in The Testing Charade:

It's not just the Common Core that has been dropped into schools wholesale before we gathered any evidence about impact; this has been true of almost the entire edifice of test-based reform, time and time again. I'll argue later that putting a stop to this disdain for evidence-- this arrogant assumption that we know so much that we don't have to bother evaluating our ideas before imposing them on teachers and students-- is one of the most important changes we have to make.

Reformsters need some humility. They need to stop assuming that their ideas are so awesome that they don't need to be tested or even, in some cases, explained-- just implemented, quickly and without time for discussion (remember "our children can't afford to wait for us to change things"). And the number one thing that reformsters have aggressively refused to do...?

2) Listen To Teachers, Dammit

There isn't a thing that reformsters are figuring out today that professional educators haven't been saying for twenty years. But reformsters have remained steadfast in their belief that they have nothing to learn by listening to classroom teachers. David Coleman was just one of many reformsters who believed that his lack of teaching and education background made him more qualified than professionals who had devoted their adult lives to education.

As reformster after reformster has written a "Hey, I just figured out this policy doesn't work" piece, I've written over and over some variation of "also, water is wet" or "no shit, Sherlock" or "we've been trying to tell you this for years."

Yet this lesson-- listen to teachers-- hasn't penetrated much at all.

Those Who Don't Learn From the Past...

Just watch. So many reformsters are lined up behind some version of Personalized [sic] Learning, even though it has no basis in real research, has produced no positive results where tried, and has a chorus of teachers hollering "this is a really bad idea." But instead of saying, "Hey, you know, the last time the stars lined up like this, it didn't work out for us," reformsters are saying, "Oh, shut up. We've totally got this."

It's great that we're learning that the BS Test scores are bunk. It would be greater if folks pursued undoing these policies with the same zeal with which they pursued installing them (one more reason that accepting responsibility would be nice). But still, baby steps in the right direction.

It would also be greater if reformsters learned some of the other lessons that come with failed reform strategies. Just go sit the corner for a few minutes and think about what you've done before you head out to do something else.

3 comments:

  1. There seems to be A LOT of conflation going on here. Let's try to sort things out. Most of the piece focuses on the rejection of standardized tests (any standardized tests, by the way) as a measure of learning. And Greene extols the fact that a "reformer" like DeAngelis is now questioning the value of tests as an end in themselves. In the past, Greene has also highlighted (strangely) the fact that standardized test scores are not improving - presumably a rejection of education reform.

    But then he sneaks in a different idea. "This meta-study comes from an interesting angle-- why is it that school choice hasn't caused giant leaps forward in achievement? The answer-- because choice based on the premise that people would choose the best school, but "best school" would be defined by "best test scores" and what happens if that's actually not a great measure of a good school?"

    School choice and testing are two very different things - with very different scopes. Standardized tests are now used in almost every state and required by ESSA. But only a small percentage of students are able to choose their school. There's just 7,000 charters out of 90,000+ public schools.

    Testing is like taking your temperature when you are sick and go to the doctor. Who in their right mind would think that just putting a thermometer in your mouth would cause your temperature to decline ? Sure, it tells you whether or not you have a fever - but it's not the thermometer that cures you. To continue the analogy, if we provided medicine (which does cure you) to only 7,000 patients, why would we think this would cure the 90,000 who did not receive medicine - but did have their temperature measured ?

    As far as his second argument (trust the teachers), is it at all possible that teachers interests and student interests are not aligned ? Here's a hypothetical. Let's say that we discover that education reform (choice AND testing) DOES lead to better life outcomes - especially for disadvantaged students. (There's already some data that supports this.) Can you see any teacher saying, "Wow ! I had no idea ! You know what ? Forget my pension and tenure and all that. I can now see that education reform (testing, choice) DOES produce better life outcomes for children. I'm on the wrong side of this. Let's give the money that was supposed to go to my unionized school instead to whichever school the family chooses - especially if that school produces better life outcomes." Nope. I can't see ANY teacher EVER saying that regardless of what is better for kids.

    ReplyDelete
  2. Excellent, as always. This year, as a Florida math teacher, I have tried to take a 'deeper dive' into my data. To avoid boring your readers, I'll keep it short. I looked for correlations between my instructional activities as recorded in my gradebook and test performance. Couldn't find any. But maybe that's on me, my grading system isn't really capturing student achievement. So I tried days absent vs. test performance. That's not subjective: the student was either in the classroom or was not. No correlation. Hmm, so it doesn't seem to matter on a high school state end-of-course exam whether a kid bothers coming to school. It's time to conclude that the tests are worthless. (And for all those about to attack me? My pass rate is twenty percentage points above the state average.)

    ReplyDelete
  3. Thinking and hoping that the rats will abandon the ship soon. More than 1000 colleges and universities not using SAT, AP or ACT for admission (this includes U of Chicago!). Harvard being sued over it's use of SAT scores may push them to dump the SAT, AP, ACT charade. And just this week, 8 prestigious High schools in the DC/MD/VA area have decided to dump AP from their course selections. PARCC is down to 6 states and DC and SBAC participation is declining. Even though these testing companies are selling and morphing, the public is already aware and ready to connect the dots to the next charade. Facebook and Microsoft can try to sell their wares, but they are being exposed, too. CBE will soon be on the chopping block. The rephormers are trying to do a CYA.

    ReplyDelete