Poverty Matters (Ep. 2,364,339)

Over at Education Next, Matthew P. Steinberg and Lauren Sartain went poking through data from Chicago's EITP teacher evaluation pilot. They worked through some interesting data-- and then leaped to a rather...um, odd... conclusion in answer to the question, "Does Better Observation Make Better Teachers?"

Steinberg and Sartain lead with some boilerplate about teachers being the most important factor in a classroom blah blah blah teacher eval systems broken blah blah blah TNTP found that systems weren't getting enough bad teachers fired blah blah blah rise of VAM controversial. Having prepared the field, we are now ready to operate.

The Excellence in Teaching Project was piloted in Chicago in 2008, one of many to turn Danielson into a household name. It was the result of a team-up beginning in 2006 between the school system (under CEO Arne Duncan) and the teachers union, a partnership that fell apart just before launch over disagreement about how to use the resulting ratings (you will be shocked to learn that the district wanted to use results for accountability stuff like tenure decisions).

That's all fertile stuff, but I'm going to skip over it because I have my eye on something else. Likewise, I'm going to skip over the nuts and bolts of implementation and the observation model itself. Short form (as those of us now living with a similar model now know) was rather than drive by and throw some numbers on a sheet, the observations required detailed data and some form of feedback to show the teacher what to change.

What's important to me today is that they generated data for observations from 2005-06 through 2010-11, and they crunched that together with student standardized test results. They monkeyed around with those numbers and then draped a lovely sheet of fancy language over it, but basically they are answering the question, "Does this observation model get test scores to go up?"

Now, let me pause to acknowledge that this whole research is fraught with Dumb Parts. Most especially, it wants to pretend that a couple of standardized tests that cover two subjects constitutes a measure of student achievement. But we can set that aside for a moment. The EITP system was designed to add "instructional coach" to the principal's job description, so that they would provide "targeted instructional guidance."

The first Big Result was that EITP raised reading scores by a statistically significant amount, but it did not do the same for math. The next Big Result was that it only made a difference in the first year (that effect they explain rather simply as "after the first year, CSP dropped support and nobody ever again got the kind of training the first-year cohort did).

But that's not the Really Big News. Take a look at the chart

Yes, the higher the level of poverty, the less effect the new teacher training-via-observation system worked. Or, in the authors' words:

Our results indicate that while the pilot evaluation system led to large short-term, positive effects on school reading performance, these effects were concentrated in schools that, on average, served higher-achieving and less-disadvantaged students. For high-poverty schools, the effect of the pilot is basically zero.

What could possibly explain such a thing? Could it be that high-poverty students face more obstacles than a simple tweaking of teaching techniques could overcome? Is it possible that all that noise about poverty being a serious obstacle to the efficacy of traditional techniques is actually true? Or that students living in poverty have more trouble dealing with pointless standardized tests? Could it be that poverty really is a big part of the explanation for why students in certain schools don't achieve at the same level as more comfortable, less-poor students?

Nope. That's not it at all.

We suspect that this finding is the result of the unequal allocation of principals and teachers across schools as well as additional demands placed on teachers and principals in more disadvantaged schools, which may impede their abilities to implement these types of reforms. For example, if higher-quality principals and teachers are concentrated in higher-achieving, lower-poverty schools, it should not be surprising that a program that relies on high-quality principals and teachers has larger effects in these schools. 

Staring this data directly in the eye, Steinberg and Sartain conclude that high-poverty schools have crappier teachers and "less-able" principals. 

In the absence of any data to support a theory of qualitative difference between teachers in poor schools compared to teachers in not-poor schools, Steinberg and Sartain conclude that the level of success in the program has nothing to do with poverty, but is all the fault of teachers and administrators.

Even though the data points to poverty as the big flashing neon sign of "Hey, here it is!" Steinberg and Sartain walk right past the blinking brightness to select again the teachers and principals as the cause. This is not so much mis-reading data as simply ignoring it. I'm not sure why they bothered with the big long article. They could have just typed, one more time, "Poor students do worse on standardized tests, therefor we conclude that the only possible explanation is that all the bad teachers in the world teach in high-poverty schools." Also, I've noticed that whenever a building is on fire, there are firefighters there with big red trucks, so if you never want your building to burn down, keep firefighters and big red trucks away.