The Limits of Data

There was a moment during a presentation at last weeks' Professional Learning Communities training (institute? gathering? big thingy?) that really illustrated, I think a bit unintentionally, the nuts-and-bolts problems with using data to "analyze" teacher effectiveness.

A chart of data from three classes broken down by three skills was on the screen, presented in student by student format. First, we looked at properly parsing the scores-- count the number of students who don't make the cut score rather than looking at averages for the class. Looked at that was, it was clear that one class had excellent results, one class had middlin' results, and one had lousy results.

And then  Dick DuFour started anticipating the explanations.

The classes might have different compositions of students. The classes might include students with learning disabilities. The classes might be at different times of day. Every possible reason you or I might give. And each one, for our example, was explained away. I honestly don't remember whether this was a real case study or a hypothetical example, but the classes were, for all intents and purposes, identical in composition.

The progression of his example was clear. After you have eliminated all other factors as an explanation, only one factor remains. The teacher.

After you have eliminated all other factors.

To make his point, he had to explain away all other variables. And this remains one of the huge limitations of student data. It's basic experimental design-- you have to control for all variables. Otherwise your data tells you nothing. If we design an experiment where plants growing in every different climate of the globe with every possible variation in light exposure, soil types, and types of plants, and then we treat each plant with a different fertilizer, the data we develop will tell us virtually nothing useful about the efficacy of the various fertilizers.

Reformsters have tried to manage the variables in several ways. They insist, for instance, that poverty, language barriers, and learning disabilities are not meaningful variables, that they make no more difference than the color of the wrapper on the fertilizer. They have tried to insist that what we think of as differences between students are not significant differences at all.

The various versions of VAM claim to be able to correct mathematically for the variables. We supposedly know that Level 3.2 squared of poverty affects student achievement by a degree of X sigma over the sine of Y and a half. My question has always been, if we know that precisely what the effects of poverty (or other factors) on student achievement, why can't we design instructional techniques to compensate for that factor. But it doesn't matter-- after we run all students through the VAMinator, they will come out the other side equalized, exactly the same.

PLCs can deal with the data gap simply, because given a good administration, the only people who have to sort out the data are the teachers in the PLC. First, they get to design the data instruments themselves, so they know what the data is supposed to mean-- its not badly written Mystery Crap in a Box. They they get to be the people who crunch the data. They have the power, authority and responsibility to say, "What we have here are apples and oranges, but we know these kids, and we can factor in their differences using our best professional judgment. We know there is more going on here than just pedagogical differences between the four of us."

But on the state and national scale, this insistence that we can explain away all differences between students becomes a large-scale farce handled by people far removed from the actual teachers and the actual students. Under the PLC model, the teachers are the data gatherers, and they are accountable to each other. You don't look your co-worker in the eye and try to sell her some made-up baloney to her face. Under the reformster model, teachers are removed from every single part of the process except the Getting Blamed For Everything part. They get to force-feed their baloney without looking anybody, anywhere in the eye.