Do No Excuses Affect Academics?

Last week at the Fordham blogsite, Kevin Mahnken touted some meta-research about "No Excuses" schools and their affect on the math and language scores on the Common Core Big Standardized Test. Well, actually they claimed to be researching “’No Excuses’ Charter Schools: A Meta-Analysis of the Experimental Evidence on Student Achievement,” but when we're looking at this kind of research, it's important to always remember that "student achievement" just means "test scores on that one high stakes test that narrowly covers a standardized test version of math and reading."

I took a swipe at the research paper itself, because you know I would do almost anything for you guys. But it is tough going for those of us not schooled in the subtle art of meta-research. But I did manage to pull out a few bits.

First, although the meta-researchers started with what I believe is technically known as "a whole buttload" of research papers, through a long and laborious process, they narrowed those down to ten papers. Of those ten, four were only about no excuses schools, two were about types of charters including no excuses, and four weren't about no excuses at all. So, six paper in the meta-analysis? I'm no meta-analyst, but that seems meta-thin.

What conclusions did they reach?

No excuse schools do better than other charters at raising math and reading test scores. They are better at improving math scores than reading scores, which stands to reason as standardized math tests are a little more prepable than reading tests which still, for instance, throw in random straight-up vocabulary questions. They also suggest that no excuses schools do better at raising scores in high schools than in elementary, though they admit that the research on no excuses elementary schools is pretty thin for even meta-slicing.

Some of the conclusions are transparently bizarre even to an untrained meta-observer:

According to Hill et al.’s (2007) standards, attending a No Excuses charter schools for one year closes approximately 25% of the Black-White math achievement gap and approximately 20% of the Black-white literacy achievement gap. A straightforward extrapolation of these results suggests that attending a No Excuses charter school for four to five years could eliminate the achievement gap.

I checked for any qualifier to this astonishing statement, but could not find anything remotely like "That is, any any student who wanted to go the school, was accepted by the school, and who was not shoved out of the school for being disobedient and non-compliant. And of course these results would not apply to any students with any sort of special needs." Surely this finding is not meant to suggest that any Black student could be plugged into a no excuses school and achieve startling success.

In fact the meta-researchers note that they believe they have insured that the sample is randomized by making sure to include research about schools that are oversubscribed and therefor had to use a lottery. But as I am not the first to observe, the process of choosing to respond to no excise charter marketing and navigating the application and lottery process has already insured that the school does not have anything like a randomly sampled student body. And that sampling gets even less random as the years go by and students flee or are pushed out the door of these highly regimented schools.

The writers meta-acknowledge that the research is narrow in its focus and that there's not much out there studying non-cognitive and other effects of charters, though they make sure to note the study that finds charter students are more likely to graduate (well, sure-- because charter students who aren't likely to graduate don't stay charter students for very long and are rarely replaced).

Mahnken is positively meta-giddy with excitement over this report:

We might have expected some optimism after witnessing the stupefying results at world-beating charter networks like KIPP and Success Academy. But it’s still nice that high-performing charters have both passed the eye test of policy commentators and are consistently feted by researchers as well. Now the only question is how the little guys grew up so fast.

But for anyone curious about no excuses education in action, you might consider tales of students going without lunch or bathroom breaks or high suspension rates or teaching young black students the importance of being submissive and compliant.

All of this continues to be justified in the name of student achievement-- except that all that means is higher scores on a single standardized math and reading test. Fans keep touting this result as proof that no excuses schools know the secret of raising students up out of poverty. As soon as I hear about the hundreds of poor students who have found well-paying middle class jobs because they are such good test-takers, I will start taking those claims seriously.