Why What Works Doesn't Work

One of the dreams of reformerdom has been to identify classroom practices that are solid, successful, even foolproof, and to send them out into the world so that every teacher can use them in her own classroom. Students learn, angels sing, and education is one step closer to being neat, scientific and efficient, and one step further away from being a big higgledy piggledy mess.

This may strike you as a pretty picture, or it may not-- it doesn't really matter, because it is never going to happen.

Mike Petrilli (Fordham Institute) has been working on a series of essays about shifting reformy attention from policies to practices, and today's entry is about What Works, and it almost perfectly encapsulates what folks get wrong about this whole business. But he is searching for an answer to the puzzle.

There are many debates in education policy that will never be settled by science because they mostly involve values, priorities, and tradeoffs...Instructional practices, on the other hand, are different. Or should be. Consider elementary schools, those magical places where we work to turn pre-literate, pre-numerate kindergarteners into avid readers, writers, and problem solvers, ready to tackle the Great American Novel in middle school, capable of writing a clear five-paragraph essay, and possessing a mastery of math facts and an early understanding of algebraic reasoning.

The stock photo for the piece even features a microscope, to underline how scientific this process should be.

Petrilli rattles off a long list of practice type questions-- how do we do small groups? how do teachers best manage classrooms? how can students be taught to write effectively? And so on. It's a long list and rightly so, but it leads him to this:

The best part about these questions is that their answers are knowable.

Ummmm.... Yes, and no. He outlines a process: Teachers develop key instructional questions to get empirical answers to (see above list. Scientists design studies and develop hypotheses to find answers to those questions. Professional educators sift through the results and decide on the strength of the evidence what should be widely adopted as preferred practice in guidelines for educators.

Why not, he says. That's what other fields do, like, most notably, medicine.

He acknowledges that some people won't like that analogy, and he's correct. I'm one of them. It's not just the "teaching is part, not all science" business. It's that teaching is way more complicated than medicine.

The scientific foolproof everyone just follow the policies and procedures approach will not work. The simplest evidence can be provided by evert single secondary subject area teacher who can tell you a version of the same story:

"I used this lesson with my second period class and it was awesome-- the kids were completely hooked, I was on fire, everyone totally got it and got into it. Then sixth period I used the exact same lesson and not bombed. The kids hated it, nobody learned anything, it was a disaster."

This has happened to every single teacher in the history of ever. You simply can't set a particular classroom practice in stone and count on it to work every time. Certainly not for every teacher. The learning in a classroom occurs at the intersection of all the humans in the room, plus all the long-term and short-term baggage they brought with them, plus the material itself, plus the time of day, plus the lunch menu, plus plus plus plus. Yes, it would be just like medicine, if doctors treated thirty people at a time and every disease presented differently and responded to different treatment every time it appeared.

Petrilli offers as an alternative Dan Willingham's comparison to architecture, where there is a great deal of variety and creative difference between works, but certain physical rules that everyone has to follow. That's better, but still...

This nice collections of What Works that various government agencies mostly gather dust not because teachers wonder about the evidential base. They aren't asking "I wonder if this works." They're asking, "I wonder if this will work for me, with my particular students." The selling strength, rightly or wrongly, of the teacher-selling-lessons site is that the answer, "Yes, I have used this with actual human students in a classroom somewhat like your many times."

Clearly there are good and less good and really not very good practices. But if I wanted to measure and chart such things, I would skip architecture and turn to chaos theory and the idea of strange attractors. If we could chart all the practices that mostly work most of the time, our graph would not show a point, but a cloud, clustered around, but not strictly attached to a shape. There would be outliers-- points on the outskirts of the cloud, and some far away from it (as I have heard, only, from three different sources in the last three days, "Everything works in some places, but nothing works in all places").

It's a fair criticism that some of what we do know about brain science doesn't carry as much weight in some classrooms as we might like. But there is a real danger in trying to make this Hard Science model fit educational practices-- and we already know about it because the exact same problem infected the policy ideas that Petrilli wants to step away from.

Petrrilli earlier held up the five paragraph essay on a list of basic skills that is "uncontroversial," that everyone agrees students need to have. Except we don't. While I was in the classroom, I would have been happy to have the power to kill the five paragraph essay dead-- especially in the elementary classrooms. But some teachers and administrators like the five-paragraph beast because it's easy to evaluate. Assessing writing is hard. Really, really hard. Assessing whether or not somebody followed a five-paragraph template is easy. And using "scientific" methods to determine the best way to get students to follow the template is easy, too. So once again, here we are scoring bullseyes on a target we shouldn't even be shooting at.

Campbell's Law infects classroom practices just as easily as it infects policies about test-centered accountability. Telling ourselves that we're okay because we've got hard science and data is exactly how education ended up so deep in the Big Muddy over testing. It is the parable of the drunk and the car keys, the guy who is searching under the lamppost instead of where he dropped the keys because the light under the lamppost is better.

Since the beginning of the modern era, reformsters have been notable for their aversion to mess. Teaching is too messy. Democratically elected school boards are too messy. Human beings are too messy. Sorry. That's life. Reform's enthusiasm for eradicating mess is dangerous and destructive, like the person who loves Latin but forgets that all that was required for Latin to become perfect was for every native speaker of the language to die.

You can no more create a binder full of classroom practices that are scientifically proven to work than you can create a scientifically proven courtship and marriage manual. Yes, some practices are more likely to work more often for more practitioners than other practices are, and yes, some practices are probably not going to work most of the time, mostly. But there will always be outliers and exceptions, and there will never be guarantees. Never.

Petrilli's desire to collect better evidence for more practices is a worthy endeavor, and it certainly would help to have more mechanism to separate tested medicine from snake oil. But teaching is always going to be fuzzy and messy and any system set up to talk about "what works" needs to not simply acknowledge that, but embrace it. You can hope for a nice tight blueprint, but you're really looking at Jackson Pollock.