Thursday, September 8, 2016

Can Evidence Improve America's Schools?

That's the question asked by Mike Petrilli (Fordham Institute) asks over at the Fordham blog.

The piece harkens back to Chester Finn's 1986 book titled, unironically, What Works. And the piece also ties into Petrilli's ongoing series about how to drive change in education. After noting that nothing from Finn's work has ever really gotten traction, Petrilli bemoans our benighted edu-state.

Education remains a field in which habit, intuition, and incumbency continue to play at least as large a role as research and data analysis.

Petrilli has, as is his wont, stacked the deck from the first framing of the issue. He could as easily have written that education is a field in which experienced and trained professionals continue to rely on their own judgment rather than the kibbitzing of non-professionals outside the field. But absent the framing, I'll allow him this-- not a lot of outside research makes much of an impression on how schools function.

So the first question is why not? And Petrilli has a list of answers from Vivian Tseng (W.T.Grant Foundation), Tom Kane (Harvard Center for Reformy Education Policy Research) and Michael Barber (Overlord of Pearson's World-Conquering Data-Hungry Corporate Monstrosity).  The list is interesting, albeit not exactly on point.

Limited supply. Maybe there's more research than there used to be, but there are still many educational questions for which there are no research-based answers.

Too much junk. "Education is awash in a deluge of reports, journal articles, emails, tweets, and news stories" that all make claims to research-based pronouncements. And yet, much of it is junk.These two supply issues already raise some points for debate. Petrilli says that v"rigorous studies have made a big impact on teacher evaluation (for better or worse)" and I'd be inclined to argue just how rigorous the studies were that gave us VAM. There's also a certain irony in this observation coming from a guy at Harvard Ed Policy, arguably an advocacy group masquerading as a think tank masquerading as a university department. That kind of multi-mask mummery has given us an unending supply of marketing baloney masquerading as research.

Add to this the growing recognition of a problem in the sciences in general and education research in particular-- the paucity of research replication. Beloved long-held research findings are dropping like flies, to the point that some science is being called broken.  Bottom line-- there may be even more junk than we realize.

Poor Dissemination. School administrators don't check the What Works Clearing House very often. Research outlets do a bad job of pushing out their works of blinding genius.

Ideology. Damn ed college professors. "They’re fundamentally opposed to the reform agenda, measuring schools via student outcomes, and hard-nosed quantitative analyses." Petrelli's tongue is in his cheek here, but the list does touch on something important-- part of the reformy stance has been not so much "here's a technique for better education" as it has been "your entire map of the world is wrong." Most of us remain unconvinced.

Habits and Practices in schools and districts. So maybe we're just not open to new research. Maybe we're tired of the "reform of the month." Maybe (and this is me talking now) reformsters sound like that guy in the old joke-- "Are you going to believe me, or your own eyes?"

Because what's going on daily in schools is not just habits and practices. It's research and data collection. I start a new lesson. I scan the faces of my students, looking for signs of reactions, in particular reactions that signal understanding or confusion, engagement or cutting out. I ask some questions. I gauge the answers. I hand out a practice sheet. I watch  them work at it. We correct it, and I collect to check for particular patterns of understanding or confusion. That's ten minutes of my class, and I've already collected a ton of data and tested a mountain of techniques. I do that all day, every day. When some guy shows up to say, "I've never done your job, and I've never watched you do your job, and I don't know your kids, or your school or your community or you, but I would like to tell you how you should totally change what you do based on one flawed piece of research with ten bits of data, and by the way, I'm hoping to make a ton of money by selling this to you--" Well, that's just not a pitch I'm buying. I'm an education researcher every day of my professional life.

So the list of reasons is a little incomplete. What does Petrilli propose as possible solutions?

Book it. Basically, publish your research in book form so that people who read books will find it. "Develop evidence-based book that might have an impact." And I was almost nodding along with this one, but then he tossed out Doug Lemov's odious Teach Like a Champion as an example, which is kind of like using Art of the Deal as an example of a way to teach economics.

Get together. Again, he's onto something in noting that most teachers get techniques and ideas from people that they personally know and trust. But from there he jumps to the notion that folks with ed ideas to plug should "do a better job partnering" with groups that already exist, and that's not false, but the problem there is also not knew-- it is hard to "partner" with people when your idea of "partner" is "I'll talk and you'll listen. I'll sell and you'll buy. You'll scratch my back, and I'll help you scratch my back." He also likes the ideas of maybe forming new networks which-- come on. Reformsters have been trying to form new grass roots networks from the top down for the past several decades, from the Center for Education Reform to StudentsFirst. Surely we already have enough astroturf.

Go small. Again, not an entirely bad idea. Petrelli suggests that instead of broad global big scale studies, why not small-scale studies that give us information about very specific, very focused situations. Which-- let's be honest-- is what many education researchers have been producing without admitting it. After all, how much education research is there that applies only to small groups of sophomore from one specific university. A great deal of research does only apply to small, narrow situations-- researchers just haven't been willing to admit that they haven't found the solution for everyone everywhere.

Look, as I said above, evidence improves America's school every single day. Every half-decent classroom runs on evidence that the teacher-researcher gathers and processes constantly, often with several replications of the study in a single day. We try new things and we immediately find out how well they work. We share what our research has uncovered (at least, we do in schools where our job security and don't depend on "beating" our colleagues) and we pick up ideas from our larger web of connections (our Personal Learning Networks, as the current jargon puts it).

The problem with much of what Petrilli and Finn before him call "evidence-based" is that the evidence sucks. We are stuck right now in a period when a bunch of influential people actually imagine that the scores from a narrow, badly-constructed test taken by disinterested students is somehow evidence of anything at all. I can't say this enough times. The scores from the Big Standardized Tests are not evidence of a single damned thing.

Meanwhile, the evidence I collect in my classroom is immediate, actionable, and directly impacts instruction and my students' achievement. The kind of research and evidence and growth and development that teachers do every day is the lifeblood of pretty much every school where the process hasn't been choked to death by some lousy administrator or a dopey CMO that that already thinks it knows everything and need not listen to teachers. The real question Petrilli is asking is, "Can we tap into that, inject our own stuff into the bloodstream, and somehow get the process to move in the direction we want it to?" But if we're going to inject it into the heart and soul of the school, it had better be something other than lumpy, clotted junk.


  1. I know you tend to be negative about a lot of things, but what's your beef with 'Teach Like A Champion"?. I ask because I've heard overwhelmingly good things from teachers. I know at least two dozen teachers who've said how it made them more effective. It's even used by master teachers in north NJ districts. I've even found several of the techniques quite useful in my adult ed sessions.

    What I hear most often from public school educators is that Champion techniques are useful for keeping kids focused and engaged..."on their toes" is how one put it to me. So what makes it odious?

    1. I read the book, and one or two techniques are very useful (e.g. don't give the answer right away to questions) but I thought if a school followed all,of the techniques it would no longer be a school. It would be a rigid, boot camp. And I was pretty appalled when he used identifying parts of speech as an example of critical thinking.

    2. The technique of not giving the answer right away isn't exactly something new and original now is it? If I recall there was a fellow named Socrates who did things like that. We could just read Plato's dialogues instead of this book and learn a lot more about knowledge, learning, and teaching. Isn't it ironic that people, like the author of this book, who profess themselves as experts on teaching know so little about intellectual history.

    3. I too have to defend Teach Like a Champion. It was very useful for training my dog.

  2. What is knowledge? What does it mean to learn? How does learning take place? What are the causal factors involved in learning? What is teaching? What is good teaching? These are actually very difficult questions to answer in detail. Also, the social sciences don't really have a great track record of solving problems. How many economists predicted the housing bubble bursting? How many psychologists really understand how the mind/brain works?

    I remember reading recently that only one-third of psychology experiments can be replicated. This is a huge problem. I would think that our social sciences are going to have to improve quite a bit before we can use them to significantly improve learning and teaching.

  3. Hi. Making sure my name isn't posted but want to reply.

  4. Your article makes me remember a third year teacher who LOVED CBE because "otherwise, how would I know they are learning?" I think maybe she is in the wrong field.

  5. Detachment. This endless parade of reform ideas, arguments, proposals, quick-fixes, technological "solutions" all share a common core (ha! A new use for the term): they presuppose that people are detached and should be treated in such a way. Is the human element really so banal that it deserves either no attention or is treated like it can be "captured" in an algorithm?

    Let's face it: if every person who is 5 to 18 in this country is required by law to be educated, how can opportunists do anything but find a way to profit off the guaranteed customer?

    How to overcome this new cultural norm of detachment combined with an addiction to data? Is it hopeless?

  6. Right on point Peter, and good reference links. And the best point, which you've made before: that good teachers (and most of them are) use their judgment and observation skills every day in the classroom to refine their craft.

  7. So, he thinks that starting on a small scale is a good idea but he's fine with Common Core which is pretty much as large a scale for an experiment as you can find in education?

    And he thinks that books are an OK substitute for actual peer-reviewed research?

    I could be wrong ("And I am never wrong," said Prince Humperdinck), but I doubt he's remotely familiar with any of the relevant research on teaching and learning that inform what teachers do in their classrooms daily, or with the more recent research covering what teachers wish they could get rid of (*cough cough* emphasis on tech *cough cough*).

  8. On the "get together":

    Many, if not most public school teachers (and private school teachers) in the USA are so burdened with teaching 5-7 periods per day with inadequate time to plan and keep up with grading & other paperwork. THERE IS NOT MUCH TIME TO SHARE or reflect or to seek out so-called 'best practices' or to read the latest in ed "research" (an oxymoron), etc, etc. In addition, almost all of the male teachers that I have worked with have one or more side-hustles to make ends meet.

  9. And, naturally, Petrilli neglects to mention the substantial censorship and information suppression activities of the club to which he belongs (Fordham / Koret Task Force / Education Next / Harvard PEPG / Hoover Institution, etc).

  10. See also: The Thomas B. Fordham Foundation and Institute: Influence for Hire