Tuesday, March 20, 2018

AEI: Voiding the Choice Warrantee

The American Enterprise Institute has a new report  that calls into question one of the foundational fallacies of the entire reform movement. Think of it as the latest entry in the Reformster Apostasy movement.

Do Impacts on Test Scores Even Matter? Lessons from Long-Run Outcomes in School Choice Research asks some important questions. We know they are important questions because some of us have been asking and answering them for twenty years.

Here are the key points as AEI lists them:

For the past 20 years, almost every major education reform has rested on a common assumption: Standardized test scores are an accurate and appropriate measure of success and failure.

This study is a meta-analysis on the effect that school choice has on educational attainment and shows that, at least for school choice programs, there is a weak relationship between impacts on test scores and later attainment outcomes.

Policymakers need to be much more humble in what they believe that test scores tell them about the performance of schools of choice: Test scores should not automatically occupy a privileged place over parental demand and satisfaction as short-term measures of school choice success or failure.

Yup. That's just about it. The entire reformster movement is based on the premise that Big Standardized Test results are a reliable proxy for educational achievement. They are not. They never have been, and some of us have been saying so all along. Read Daniel Koretz's book The Testing Charade: Pretending To Make Schools Better for a detailed look at how this has all gone wrong, but the short answer is that when you use narrow unvalidated badly designed tests to measure things they were never meant to measure, you end up with junk.

AEI is not the first reform outfit to question the BS Tests' value. Jay Greene was beating this drum a year and a half ago:

But what if changing test scores does not regularly correspond with changing life outcomes?  What if schools can do things to change scores without actually changing lives?  What evidence do we actually have to support the assumption that changing test scores is a reliable indicator of changing later life outcomes?

Greene concluded that tests had no real connection to student later-in-life outcomes and were therefor not a useful tool for policy direction. Again, he was saying what teachers and other education professionals had been saying since the invention of dirt, but to no avail.

In fact, if you are of a Certain Age, you may well remember the authentic assessment movement, which declared that the only way to measure any student knowledge and skill was by having the student demonstrate something as close to the actual skill in question. IOW, if you want to see if the student can write an essay, have her write an essay. Authentic assessment frowned on multiple choice testing, because it involves a task that is not anything like any real skill we're trying to teach. But ed reform and the cult of testing swept the authentic assessment movement away.

Really, AEI's third paragraph of findings is weak sauce. "Policymakers should be much more humble" about test scores? No, they should be apologetic and remorseful that they ever foisted this tool on education and demanded it be attached to stern consequences, because in doing so the wrought a great deal of damage on US education. "Test scores should not automatically occupy a privileged place..."? No, test scores should automatically occupy a highly unprivileged place. They should be treated as junk unless and until someone can convincingly argue otherwise.

But I am reading into this report a wholesale rejection of the BS Test as a measure of student, teacher, or school success, and that's not really what AEI is here to do. This paper is focused on school choice programs, and it sets out to void the warrantee on school choice as a policy.

Choice fans, up to and including education secretary Betsy DeVos, have pitched choice in terms of its positive effects on educational achievement. As DeVos claimed, the presence of choice will not even create choice schools that outperform public schools, but the public schools themselves will have their performance elevated. The reality, of course, is that it simply doesn't happen.The research continues to mount that vouchers, choice, charters-- none of them significantly move the needle on school achievement. And "educational achievement" and "school achievement" all really only mean one thing-- test scores.

Choice was going to guarantee higher test scores. They have had years and years to raise test scores. They have failed. If charters and choice were going to usher in an era of test score awesomeness, we'd be there by now. We aren't.

So what's a reformster to do?

Simple. Announce that test scores don't really matter. That's this report.

There are several ways to read this report, depending on your level of cynicism. Take your pick.

Hardly cynical at all. Reformsters have finally realized what education professionals have known all along-- that the BS Tests are a lousy measure of educational achievement. They, like others before them,  may be late to enlightenment, but at least they got there, so let's welcome them and their newly-illuminated light epiphanic light bulbs.

Kind of Cynical. Reformsters are realizing that the BS Tests are hurting the efforts to market choice, and so they are trying to shed the test as a measure of choice success because it clearly isn't working and they need reduce the damage to the choice brand being done.

Supremely Cynical. Reformsters always knew that the BS Test was a sham and a fraud, but it was useful for a while, just as Common Core was in its day. But just as Common Core was jettisoned as a strategic argument when it was no longer useful, the BS Test will now be tossed aside like a used-up Handi Wipe. The goal of free market corporate reformsters has always been to crack open the vast funding egg of public education and make it accessible to free marketeers with their education-flavored business models. Reformsters would have said that choice clears up your complexion and gives you a free pony if they thought it would sell the market based business model of schooling, and they'll continue to say-- or stop saying-- anything as long as it helps break up public ed and makes the pieces available for corporate use.

Bottom line. Having failed to raise BS Test scores, some reformsters would now like to promote the entirely correct idea that BS Tests are terrible measures of school success, and so, hey, let's judge choice programs some other way. I would add, hey, let's judge ALL schools some other way, because BS Testing is the single most toxic legacy of modern ed reform.


  1. "Having failed to raise BS Test scores, some reformsters would now like to promote the entirely correct idea that BS Tests are terrible measures of school success, and so, hey, let's judge choice programs some other way. I would add, hey, let's judge ALL schools some other way, because BS Testing is the single most toxic legacy of modern ed reform."

    Oh, but you've already written many times about the new way to judge all schools. BS Testing is so last year. Now we're into all testing all the time "Competency" or "Outcome" based "Personalized" learning. Surely choice schools will rock that - they've already had a helluva headstart!

  2. Do we dare tell them that so-called school/student "success" has little to do with academic achievement. Witness the old Letterman gag in which a line up of Ivy League graduates could not accurately explain why summers are hot and winters are cold in North America. Of course they learned it and then like most of what we teach that has no practical use for our long term memory banks, we soon forget. What we don't forget is how to show up on time, every day, work cooperative, defer to authority, play by the rules, how to cheat, get along with others, make friends, find young love, how to fight and argue and defend oneself, how to thicken one's skin, the importance of a collective cultural experience to look back on and laugh at or smile at or be thankful its over. And we don't forget that the doors of opportunity that opened up over the years lead us to vocational programs, the college experience, theatre programs, bands and orchestras, the playing fields and courts, to jobs, careers, marriages, families and a sense of community. And the real answer to the question, "Hey Mr. Greene, when am I ever going to use this?" was probably never, unless you are talking about your brain and all the important ways it developed during the K to 12 experience.

    1. I completely agree with this. Some of this is now labeled "soft skills", but it really should be called "how to be human". That's why we should have art and music in schools, because we want to live with better and kind human beings. It all matters, but not because of a test.

  3. Count me as supremely cynical. As soon as the evidence against testing is too much, the reformers say, well, testing must be a bad way to evaluate schools because we all know that school choice is awesome. School choice, and the eventual destruction of public education, is the goal. Testing was just a means to that goal. Now that it is under criticism, they will move on to other means.

  4. PART TWO:

    Check out the next question and Duncan’s non-answer:

    01:25 – 01:41

    REPORTER: “Some people have been critical all along of No Child Left Behind and the testing portions of this. Umm, how fair is that criticism?”

    DUNCAN: “Well, we want to fix the No Child Left Behind Law. That’s a much longer conversation, and we’re working very hard in Congress to do… to do that now.”


    Again… W-T-F? His response is that he wants to “fix” NCLB. Well, exactly WHAT about NCLB do you want to “fix”? For Duncan’s answer to be responsive to the question, he must then address criticism of “the testing portions” that the reporters’ question references… the “portions” that create the breeding ground for cheating scandals like the ones in Atlanta, Washington, D.C. and elsewhere…. and Duncan ain’t doing that.

    The reporter is pushing Duncan to admit that all this test-based evaluating/punishing/rewarding is harmful, but he responds with pointless blather about how “we’re working very hard in Congress to do that now.”

    Really?… “to do WHAT now”? You meant that politicians and education officials should “fix the testing portions” that are harming education and harming kids?

    Again, no answer.

    The reporter then questions whether, in urban areas with so many challenging factors teachers have to fight, that demanding “unrealistic” results led to the cheating problem, that when asked to do the impossible, teachers who are threatened with firing for not achieving the impossible, will then be driven to cheat. (which is the conclusion one gets from reading Rachel Aviv’s NEW YORKER article.)

    This is another great question, by the way. Kudos to the reporter!

    Again, Duncan totally ducks this query. He challenges those doubters who think that the NCLB benchmarks were “unrealistic” that they are the ones in the wrong, that he is “seeing students learn every single year”

    This is his version of the Michelle Rhee diversionary response to evidence of cheating: “You must be racist to think that poor, minority kids can’t learn.”


    01:41 – 02:14

    REPORTER: “But, but the whole idea of unrealistic measurements… something for urban districts, et cetera… Is that – ?”

    ARNE DUNCAN: “I don’t think there is anything ‘unrealistic’ about seeing students learn every single year, and you have in many urban areas tremendous progress being made. The sad fact is that I actually think in Atlanta there’s probably tremendous progress being made… fairly… and unfortunately, this, this… scandal is going to cloud that… ummm…. but this does not in any way take away from shouldn’t take away from the hard work, and the accomplishments, and the improved graduation rates that we’re seeing in many urban districts around the country.”

    Let’s move on to the next question, about the idea that Atlanta school district’s monetary incentives helped create the problem. This is the closest he gets to being responsive to the question being asked.

    He says that monetary incentives ARE NOT ONLY GOOD for education, but that we should have started doing them long before now.
    Oh really?

    The only problem with Arne’s claim is…. the overwhelming evidence shows that…




    All the decades of evidences show that not only do they not improve education; they actually do grave harm to it.

    But hey, Arne thinks we should keep trying anyway, so we’re just going to have to be stuck with more of it. At the end of his spiel, he vomits up the idea that using monetary incentives is “not a hard thing to do”, that you just “have to do it with integrity.”

    Really? “Not a hard thing to do”?

    Then how come it has NEVER worked, that historically, doing so has an utter and total failure rate?

    Duncan thinks we should “spotlight” and “celebrate” good teachers and principals… with monetary rewards (the next question BEL0W).

    Duncan’s assumption is that prior to, or without those rewards to push them, teachers will or are holding back their “A Game”, and not giving it their best effort… and that with monetary rewards, they’ll get off their duff and do the job they should have been doing all along.

    This comes from a man who has never taught a day in his life, or worked as a principal a day in his life, for if he had, he’d know that this is all total garbage.

    02:14 – 03:02

    REPORTER: “Should… a lot of this is about money, I think, you know, that both teachers and principals are evaluated by their test scores of their students, and there’s a lot of money involved in this. Should that be decoupled from student learning?”

    ARNE DUNCAN: “Well, I think rewarding teacher excellence is important. I think I would argue the opposite, that far too often in our country, we haven’t celebrated great teachers, we haven’t celebrated great principals who are making a huge difference in students’ lives. You just want to make sure that they’re doing it honestly, and again, the vast vast majority of teachers are doing an amazing job, often in very difficult circumstances, in helping students beat the odds every single day.

    “I think we need to do a better job of spotlighting that, and incentivizing that, and encouraging that, and learning from that. In education, we’ve been far too reluctant to talk about success. We need to do that. We just need to make sure that we’re doing it with integrity…. not that hard to do.”

    The reporter finishes with a questions about one of the intangible ways that this harms education and society as a whole. She gets personal and talks about how this cheating scandal has taken away her “last heroes”, the teachers, and on and on.

    I’m sick and tired of transcribing this words of this vile person (Duncan, not the reporter, whom I admire)… so, if you want to, you can watch her ask this last question, and the entire video here: