Cheating the NAEP

We've had our giant round of reaction to the NAEP test results, and their woeful failure to show American school children being propelled forward into a wonderland of learning by over a decade of reformster policies. I'm not sure there's any reason to get excited about NAEP results at all, but test-loving folks do, and there's really no denying that this round of NAEP results were Just Not Good.

Oh, but what if it turned out that they were actually even worse?

RaShawn Biddle may not be familiar to you; Biddle runs a one-man media empire parked firmly in reformsterland. But his blog Dropout Nation ran some interesting analysis of some NAEP numbers.

To understand what he's about, you need to know that NAEP allows states to opt out up to 15% of their student special population-- typically students with special needs and English Language Learners. But Biddle is a good reformster, and so he believes in the simple two-step proposition:

1) Public schools are failing. We just need to prove it so we can get support for dismantling them.
2) Making students take Big Standardized Tests who can't possible pass them-- that will help with #1.

Biddle likes to talk about "special ed ghettos" and he's a huge supporter of having all the students there fail BS Tests so we can prove that their school districts suck. But he's not wrong when he points out that some states and cities are gaming their NAEP stats by controlling who actually takes the test. Biddle has assembled two Dishonor Rolls.

On the state level, the big loser is Georgia.

The Peach State was the worst in the nation in excluding fourth- and eighth-grade kids in special ed, keeping 25 percent of each group of students from taking NAEP this year. Although the levels of exclusion declined by, respectively, six and seven percentage points from levels two years ago, Georgia has done far less than either Maryland or Department of Defense to reduce its test-cheating. 

Different states use different exclusion approaches to special needs and ELL. Here are the exclusion leaders when it comes to 4th grade special ed

When we look at ELL; Kentucky leaps into the lead:

Meanwhile, the NAEP is working on trial assessments of big urban districts, so Biddle needs a whole other Dishonor Roll for those Big Cheaters on the City Scale (by his count, fourteen of the twenty-three cities cheated).

Washington DC's heralded NAEP improvement? They excluded almost half of their ELL students from the test. Dallas opted out 44% of their fourth grade students with special needs, and 29% of the eighth grade. Philly and Miami-Dade managed to exclude students all across the board-- both groups, both grades. Baltimore, Houston, and Detroit also excluded huge numbers of students, making their NAEP results somewhere between "suspicious" and "invalid."

It's interesting to see so many states and cities doing their best to support the Opt Out movement. Michigan is not really a fan of Opt Out, but I guess Detroit wants to run its own Opt Out program.

And it's also worth noting that we've got one more example here of how putting stakes on BS Tests leads to people looking for ways to game the system-- even people who are the supposed official guardians of correctness and fair play.

But there's another issue here-- what exactly qualifies as "fair" or "not cheating" in this situation? Following the rules is only fair if the rules are fair to begin with, and I can't find anything to suggest that the 15% opt-out allowance is anything but an arbitrary number arrived at as a political compromise between hard-nosed test love (we must test everyone) and inconvenient reality (given students a test far above their abilities is a pointless, punishing exercise).

I'm also going to invoke my made-up Law of Bad Assessment-- the more inauthentic the assessment and the more removed from what is actually being assessed, the easier it is to cheat. And its corollary-- when an assessment is so inauthentic that its demands can't be met by authentic skills, cheating is not only probable, but necessary.

That law is only amplified when the inauthentic assessment is used for no legitimate purpose. Did we need to have all the ELL and SWSN tested so that we could identify the ones who would fail? The states and cities already identified them in order to exempt them, and Biddle's critique of the states and cities assumes they are correct (he's not saying, "Boy, they would have had better results if only they'd let everyone be tested."). We all already know which students can't make this particular grade. Biddle claims to want the benefits of testing for all students. That's silly. There are no benefits to making a student take a test that she, her teachers and her parents all know is beyond her current level of ability and knowledge.

Did we need to be able to identify failing states or cities, or at least stack rank them? Why? What policy goal is aided by that information? We know where the challenged students are, we know where their schools are, and we know what they need (time, resources, and teachers). What else do you think we need to know, and how will making more students fail the NAEP help gather that information?

So what's the beef? The beef is that by cheating, the states and cities avoid being publicly caught failing and suffering the beatdown that reformsters want them to get. The beef is also that by cheating, the states and cities hid the full extent to which reformsters have failed to achieve the gains they promised us we'd see after years of their policies. And the fact that we are all unhappy is a sign of just how large a clusterfarfigneugen the whole business is.

Can we really talking about gaming the system when the system is just a big game?