When Data Are Not Data

My favorite story of the last week ran in Valerie Strauss's "The Answer Sheet" on the Washington Post site and dealt with Arne Duncan's reaction to the increasingly loud chorus of VAM debunkers.

The research showing VAM as ineffective is piling up. Research by Polikoff and Porter eviscerated, finding no real predictive power and suggesting that what VAM measures is not what we mean when we say "good teaching." That study was particularly telling because it had large piles of Gates dollars piled up behind it.

Perhaps my favorite VAM study was performed by Marianne P. Bitler who showed that VAM predicts a teacher's effect on student test score growth about as well as it predicts the teacher's effect on the student's height.

People who play with this type of statistical analysis have been warning that VAM was not solid for five years now, but now in addition to the professional opinions of people who do this sort of thing for a living, we have actual studies by actual scientists with actual-- what is that stuff called? Oh yeah-- DATA!

Don't we love data now? Isn't the whole foundation of the VAM system the idea that it's cold, hard data, with like numbers and stuff, so we know it has to be right. So wouldn't the US DOE want to make use of all this handy data that's rolling in about VAM? Here's the response that Strauss reports from the DOE:

Including measures of how well students are learning as part of multiple indicators of educator effectiveness is part of a set of long-needed changes that will improve classroom learning for kids. Growth measures are a significant improvement over the system that existed before, which failed to produce useful distinctions in teacher performance. Growth measures — including value-added measures — focus attention on student learning and show progress. While these measures are better than what existed before, educators will continue to improve them, and sharp, critical attention from the research community can help.

Strauss is a real journalist, so she simply reports this quote with little comment. I am not a real journalist, so I can go ahead and say this-- what a load of bullshit.

It is a distilled version of the Reformster narrative. We know (because, you know, we just know) that there are a whole ton of terrible teachers out there. The old evaluation system doesn't show the existence of these gazillion horrible educators, so the old system must not work. We will look for a new system that does work, and we will have proof that it works when it confirms our belief that a huge number of American school teachers suck. VAM may not work any better than tea leaves or reading the bumps on a frog, but it tells us that many teachers suck, and that's good enough for us and certainly an improvement of the old system.

The spokesperson also indicated that the US DOE keeps track of the research.

So bottom line-- Duncan's office knows that VAM is crap. They just don't particularly care. Because data are only data when they say what you want them to say.