Another Flawed Study Praises Charters

"Study Finds Link Between 'No Excuses' Policies and Success" trumpets the headline at Education Dive, a site that.... well, here's their mission:

Our mission is to provide busy professionals like you with a bird's-eye-view of the education industry in 60 seconds.

So take that for what it's worth. We know that press coverage of research is often faulty, so let's go to the actual paper and see what we have here. Is this the long-dreamed-of proof that No Excuse charters do it better?

"Charter Schools and the Achievement Gap" was published in The Future of Children, a Princeton-Brookings publication. The author is Sarah Cohodes, an assistant professor at Teachers College, Columbia University whose specialty is exactly this sort of achievement gap stuff. Her article was reviewed and critiqued by Lisa Barrow at the Federal reserve Bank of Chicago because when you want an extra set of eyes on an education article, of course you call a banker. Her specialty is apparently education, because economists are all education experts, somehow. But I digress.

Cohodes opens with a quick recap of charter history, then lays out the problem with measuring charter effects-- selection bias because charter students have chosen the charter. But good news-- the selection bias problem is completely solved by charter school lotteries. Except (she acknowledges) not everybody chooses to enter the lottery. And the lottery only applies when schools are over-subscribed. But maybe we can find comparable groups of non-charter students to compare charter students to. Which is hard. Cohodes seems to conclude this kind of research is really hard to design well. So she used some lottery studies and some observational studies in her research. And, having scanned the research, she drops this right in this intro section:

The best estimates find that attending a charter school has no impact compared to attending a traditional public school.

And she spends several pages recapping lots of studies of charter effects. She's a pretty good explainer-- if you ordinarily dread research papery stuff and meta-studies even more so, this is pretty penetrable for the layperson. Her critiques of some of these bits of research are pretty gentle, but I am not going to wade into those case by case here.

Instead, let's go to the headline material. Essentially, she finds that No Excuses charters set up in neighborhood served by very struggling public schools show a big gain in test scores. But here I will get into specifics, because she cites in particular the KIPP schools and the charters of Boston. Yet Boston charters have been found to come up very short in sending students on to complete college.

The No Excuses practices that Cohodes zeros in on are " intensive teacher observation and training, data-driven instruction, increased instructional time, intensive tutoring, and a culture of high expectations." Not being able to narrow the list down is a problem-- if I tell you that my athletic program gets great results by having athletes exercise for two hours daily, drink high protein shakes, breathe air regularly, and sacrifice toads under a full moon, it will be easy to follow my "research" to some unwarranted conclusions. Cohodes' list is likewise a hugely mixed bag.

Longer school day and school year is obvious. More time in school = getting more schooling done. A culture of high expectations is meaningless argle bargle. And the teacher training and "data-driven" instruction boils down to the same old news-- if you spend a lot of time on test prep, your test results get better.

Cohodes also notes that the worse the "fallback" school results, the greater the charter "improvement." In other words, the lower you set the baseline, the more your results will surpass it.

She doubles back to look at how charters relate to the surrounding public schools, again kicking the tires on the research to test reliability.

She notes that there are two ways for lottery charters to cream the best students from the community. One is to manipulate the lottery, which she doesn't think happens (for what it's worth, neither do I, mostly because it's not necessary). The second is to push out the students the school doesn't want. But she is missing two more-- make the lottery system prohibitively challenging, so that only the most motivated families can navigate it. And advertising allows charters to send a clear message about which students are welcome at their school.  And nobody works those creaming tricks like No Excuses schools, with their highly regimented and oppressive treatment of students.

Cohodes acknowledges many of the problems with the research she is using, but ultimately she says things like this:

If we wished to use charter schools to reduce US achievement gaps, one obvious way to spread charter school success would be to replicate the most successful schools.

Um, no. If we wished to use charter schools to reduce US achievement gaps, we might first want to determine whether or not they could actually do that job. And then there's this:

With strong interest in charter schools among policymakers and with school lotteries available to form the basis of high-quality research, charter schools are a relatively well-studied educational reform.

I'm not sure we've yet established that lotteries do, in fact, form the basis of high quality research.

And the criticism that I found myself leveling at very page finally surfaces here:

Given that the overall distribution of charter school effects is very similar to that of traditional public schools, expanding charter schools without regard to their effectiveness at increasing test scores would do little to narrow achievement gaps in the United States. But expanding successful, urban, high-quality charter schools—or using some of their practices in traditional schools—may be a way to do so.

Emphasis mine. If you think that closing the achievement gap is nothing but raising test scores, you are wasting my time. It's almost two decades into this reform swamp, and still I don't believe there's a person anywhere sayin, "I was able to escape poverty because I got a high PARCC score." Using the Big Standardized Test score as a proxy for student achievement is still an unproven slice of baloney, the policy equivalent of the drunk who looks for his car keys under the lights, not because he lost them there, but because it's easier to look there.

It's really not that hard to raise test scores-- just devote every moment of the day to intensive test preparation. What's hard is to raise test scores while pretending that you're really doing something else.

Let’s consider a thought experiment in which further expansion focuses on high-quality charters. What would happen to the achievement gap in the United States if all of those new charter schools were opened in urban areas serving low-income children, had no excuses policies, and had large impacts on test scores like Boston, New York, Denver, and KIPP charters?

Yes, I want to say, and let's consider a thought experiment in which pigs fly out of my butt. However, she continues

Expanding charters in this way certainly could transform the educational trajectories of the students who attend. But if we consider the US achievement gap as a whole, it would have a negligible effect.


What she wants to see is an expansion of charter practices expanded to public schools, and she sees ESSA as a policy tool to do it. But what practices? Expanded school time? That would take too much money for policy makers to support. Relentless test prep at the expense of broader education? No thanks. High expectations in the form of heavy regimentation, speak-only-when-spoken-to, treatment? Pretend that student socio-economic background and the opportunity gap are not really factors? That seems just foolishly wrong. Besides the questionable morality of such an approach, a vast number of parents simply wouldn't stand for it. And how would we replace the mission of public education-- to educate all students-- with the mission of No Excuse charters-- to educate only those students who are a "good fit."

But after more nuanced looks at the research, Cohodes arrives decidedly unnuanced, unsupported conclusions like this

Attending an urban, high-quality charter school can have transformative effects on individual students’ lives. Three years attending one of these high performing charter schools produces test-score gains about the size of the black-white test-score gap. The best evidence we have so far suggests that these test-score gains will translate into beneficial effects on outcomes like college-going, teen pregnancy, and incarceration.

The "transformative effect" is not established. Test scores might be raised, but so what? And "the best evidence we have so far" is a weaselly way of saying "we have no good evidence."

After pages of what appear to be thoughtful considerations of the research, complete with acknowledgement of that research's limitations, Cohodes arrives at a finish that reads like most charter fan PR cheerleading. And it's that part that folks like Education Dive will focus on with headlines that define "success" as "raises test scores some."