Saturday, August 16, 2014

A Curmudgucation Birthday

I made my first post here one year ago today (this is post #485). Several, in fact. Do us both a favor and don't go read them; it was a month or so before I started to figure out what the hell I was doing. Still working on it.

I was fortunate to find some audience fairly quickly. I joined BATs back when there were around 1600 members, and the responses I garnered there were encouraging. I was fortunate to be welcomed into that group. Anthony Cody was the first A-lister to ask to reprint me. Then Diane Ravitch saw something worthwhile in my work and has connected me with a much larger audience. I have been fortunate to have her backing and support. More recently, Bryan Maygers accepted me into the stable of Huffington Post bloggers, which has also let me reach even more folks. Here on the mother ship, this post has pulled about 5,300 views. On the HuffPo, it's been translated into three other languages and pulled about 450,000 likes. That would be my personal best. Soon, I'll be starting work to reach yet another audience by blogging at Education Week.

I have fans, bizarre as that seems to me, many of whom make up the small army of proofreaders who help me avoid looking too wrong for too long. I'm fast, but not always precise. I appreciate all of them as well, and that they care enough to help me do better.

I would be lying if I said it wasn't personally gratifying to have an audience. The only thing I ever wanted to be as much as I wanted to be a teacher was a writer (what tipped the scale was that I also wanted to be a Person Who Eats).

But I've been writing for a variety of audiences for decades. I've never had the material pouring with the volume it does here, nor felt so fired up when I work. That's not because I'm cranked up with dreams of fame and fortune (which is just as well)-- it's because I'm saying things that I believe are important directly to the people who want or need to hear them. The two most gratifying pieces of feedback I get are "You made me think about this differently" and "You put into words what I wanted to say but had trouble expressing."

I've never met any of the people in person that I have "met" on line. As an English teacher, I find it absolutely gutbustingly awesome that we live in an age where so much can be done just through the written word. I haven't found an audience because of my sparkling charm or a multi-million dollar PR campaign. I have a small gift for stringing words together, and I was fortunate enough to connect with people who connect with it. I never cease to find it unutterably cool that we live in a world in which such things can happen.

I'm proud to have connected with so many people, to have written words that help inform some people's opinions and understanding of their work. I'm also a little flabbergasted; I'm at about 550,000 hits on the blog-- how the heck did that happen.

But I'm not what's important here. What's important is the cause, the resistance, the continuing marathon that is the work of making American public education live up to everything it can be. What's important is to find our way as teachers, as parents, as students, in a world that has somehow twisted itself into an attack on its own best hopes and aspirations.What's important is education and teaching and doing it the best way we know how, and doing it every day better than we did it the day before. That's what's important. The attention that my posts get is not because I'm anybody-- it's because people care deeply, passionately about the issues that I write about.

I'm using the blog birthday to spend some time thinking about where I really want to focus, and what I issues I have failed to highlight effectively and see if I can't do better moving forward. I owe a huge debt to public education; I want to keep trying to pay it back.

I'm also resolved to do a better job of promoting other voices. When I was starting out, I felt awkward about hooking onto other people's stuff-- it felt like trying to grab coattails to draw an audience. But I am increasingly aware of the enormous value in amplifying all the voices of the resistance. There are a lot of good, thoughtful writers, each with his or her own strengths to bring to the table. At this point, I have an audience of my own, and I want to direct them to some of the other great stuff that's out there. If I am showing them people they already know, that's okay, too. Linkage and referrals raise a writer's profile on line, and as we've seen, it is good for public education when its advocates cannot be dismissed as a small splinter cell, but must be seen for what we are-- a large and diverse crowd of many and varied voices. Just as my students are better served if I don't close the door and ignore my colleagues, the cause of public education is better served if I don't just sit in my own corner and post nothing but my own thoughts from my own head. Dialogue, discussion, sharing, sass-- those are the building blocks of good edublogging.

This post, like pretty much everything I post, is about getting my thoughts organized and out of my head, to give me a place I can go read a note to myself to refocus and renew (without getting ridiculous). Every year I try to do everything a little better-- better teacher, better father, better trombone player, better husband, better non-jackass. This year I'll also try to be a better blogger.




Can We Enter Phase Three?

    Reading Paul Thomas always makes me feel smarter (and yet we have also shared some great cyber-conversation about comics). A recent post on his blog is both smart and challenging. In it, he addresses the phases of the current education debates.

Phase I goes back to the accountability fever of the 1980s and takes us right up through NCLB and its steroidified sibling, Race to the Top (and waiver-driven Race to the Top Lite).

Most of those accountability years, I would classify as Phase 1, a period characterized by a political monopoly on both public discourse and policy addressing primarily public K-12 education.

We are now in Phase 2, a time in which (in many ways aided by the rise in social media—Twitter, blogging, Facebook—and the alternative press—AlterNet and Truthout) teachers, professors, and educational scholars have begun to create a resistance to the political, media, and public commitments to recycling false charges of educational failure in order to continue the same failed approaches to education reform again and again.
Thomas argues that while we are still in Phase 2, it's time to start on Phase 3, moving (as he puts it) from adolescence to adulthood. And he proposes some changes in strategy.

First, we need to consider our tone. As we expand our audience, we need adopt a tone that does not undermine our mission (Thomas allows that tone-based attacks from reformsters are still not legit-- I'm foreshortening his arguments considerably).

Believe it or not, I do think about tone when I write. Earlier in my blogging career, I was a lot more passed off, and that clearly copies through. Still, I have generally avoided personal attacks (I kn ow the difference between "Chris is stupid" and "Chris just said a stupid thing" or "Chris is promoting a stupid policy." But my goal has always been to speak plainly, and sometimes that creates a bluntness in tone that some people find off-putting.

There's a real challenge in engaging someone who is being condescending and dismissive, and an even larger challenge when engaging someone who is actually attacking you. This is further complicated when the person sincerely fails to grasp that they are attacking. I think there are many reformsters who really don't understand just how seriously they are attacking public school teachers and are genuinely surprised at some of the pushback they get.

The other problem is the problem of being "uppity." As Thomas (and many others) note, teachers are traditionally good team players. We keep quiet, stay in place, follow orders, and never rock the boat. With that background, even a simple, "Excuse me, sir, but I'd rather not" is perceived, by both listener and speaker, as pretty feisty and uppity.

That said, I've always argued that screaming at people and venting the rage-fueled assumption that all who disagree with us must be either stupid or evil is just plain ineffective. Leading reformsters actually corrected themselves on that score.

Also, when considering tone, context matters. I don't just mean the context of content, but the location of the conversation. The same tactics that may be appropriate in Chicago or New York will do no good at all in my quiet small town of 7,000. And vice versa.

Second, we Ned to stop putting out fires.  Thomas says we need to stop simply responding to the Ridiculous Celebrity Comment Du Jour. This is tricky as hell, for a couple of reasons.

One is the audience. One reason I call She Who Will Not Be Named (and refuse to name her) the Kim Kardashian of education is because she is reliable clickbait. I'm under no obligation to write about her, but I know that the crowd will be running off to some other place that did. Getting the audience to ignore someone is a real challenge.

The other is focus, and Thomas is right on point here. The best way attack a person's credibility is to attack their argument. We do not need, and it is not useful, to try to prove that reformsters are terrible people. We need to be talking about their terrible ideas.

They have made this difficult because they rarely present their terrible ideas with evidence or support. Most often, they simply assert linkages that don't exist. We still haven't seen a shred of specific evidence to support, say, the idea that the Core will make a student ready for college, or that the standards are "tougher" or "higher." Evidence and soundness of reformster ideas have never been part of the conversation.

It's a tough stance to counter. When people are hearing about happy puppies and fluffy bunnies and unicorns that poop rainbows, they don't want to be brought down by boring bummer facts.

We will always be at a marketing disadvantage, because ed reform since the dawn of time is always about a Really Cool Thing that will Change Everything and bring Instant Fixes, whereas actual education and schooling is about long hard unglamorous day-by-day work. Guess which is easier to market. But we can't get to where we want to by trying to sell a better brand of snake oil.

I think one answer is to come up with a batch of questions and to keep asking them, again and again  and again, just as many times as reformsters repeat unsubstantiated claims. "How do you explain [foundationless baloney], exactly?" Or, "How do you explain [actual fact]?"

The other answer, at least for me, has been to follow the arguments, not the people. It does get to be like playing whack-a-mole, but I want to point out that in many cases we've been successful. For instance, reformsters have stopped floating the claim that teachers wrote the Common Core. It's true that we need to stop simply reacting, but it's also true that by reacting relentlessly, we have made the conversation drift a little closer to reality.

That's our big advantage-- we have actual facts on our side. Reformsters just have to keep making shit up. Time is on our side. But (as we all know from managing our own classrooms) it's far more productive to focus on what we want to see happen and not on what we want to NOT see happen.

Third, we need more faces and voices.

The mainstream media have reduced the resistance to Ravitch in much the same way that the media have reduced climate change to Bill Nye. The resistance is and must be promoted as a rich and varied body of professionals, both unified and driven by the tensions of our field. Race, gender, sexuality, ideology—the rainbow of our resistance must be prominent and we cannot allow it to be reduced, oversimplified, or marginalized.

Amen. Now, to some degree, this is out of our control. Unless you've got a zillion dollars to spend, it's hard to manufacture media attention.

But all of us can help by amplifying all of our voices. Diane Ravitch had a huge platform essentially fall in her lap-- she was the perfect media-ready person to represent a point of view. And once she had a spotlight, she started aiming it at other people. None of us have that kind of platform, but all of us have a platform of some sort. We can use it to pass on what resonates. Tweet links. Share on Facebook. Email. If you feel as if you don't have the words, share someone else's.

As you read and share, look for two sets of people. People who are just like you and who are saying just what you want to say, and people who are different from you from whose work you learned something about another point of view. Both sets are important to developing a rich and varied chorus that both reflects your concerns and connects you to the larger community of people working to strengthen all that is best and valuable in American public education.

I think Thomas has written a piece well worth thinking about. It's possible that he's simply describing what is about to happen naturally and organically, which would suit me-- I'm not a fan of super-organization. But I do believe the time has come to be just a hair more mindful of what we're saying and doing.

Teachers in Thunderdome

One of the dreams of reformsters is a school system in which teacher employment is shaped by neither tenure nor seniority. When the time comes for cutting staff, administrators will just grab their Big Spreadsheet of Teacher Effectiveness Data, look down at the bottom of list, point at the name next to the lowest rating number and declare, "Okay-- that's who's getting laid off."

We've talked about the huge problems with the data generation methods that would go into such a list. But let's talk about the effect that such a system would have on teaching staffs.

The system would turn shrinking school districts into education Thunderdomes (only "two teachers enter, one teacher leaves" will have a slightly different meaning).

Some of the best educational ideas out there are pushing teamwork and collaboration, built on the idea that all of our students belong to all of us teachers. Not my kids in my room, and your kids in your room. Unfortunately, most VAM-based systems don't see it that way. My kids determine my fate, and your kids determine yours.

So do I really want to help you with your kids when that means making my own job less secure?

Look, I think the overwhelming majority of teachers are good people who went into teaching for the right reason, and I think they would have a hard time saying, "No, I won't give you any help in figuring out how to teach that skill to your class." But how does anyone overlook the fact that she has a family to help support, kids to feed and put through college, a mortgage to pay off-- how does anyone look at that and say, "Yes, I am going to actively work to make my employment less secure."

Our current system depends on both official and unofficial mentoring of new teachers. How many pieces of advice, handy lesson tricks, moments of moral and educational support can you bring yourself to share when every bit of help you give to someone else is a bit of hurt for yourself?

Who really, truly imagines that a teacher beaten and carried out of Thunderdome will go home and cheerily announce, "Sorry, kids, but no new clothes and no new vacation for you. But i'm sure you'll be glad to know that I lost my job to someone with better numbers."

How will Thunderdome affect hiring? After all, some districts include classroom teachers in the interview process. Will teachers sit in interviews and think, "Yes, I want to find someone good enough to take my job!" Or will it become a tricky business of finding someone who's good enough not to be a big chunk of dead wood, but not so good that they're a threat? And will administrations figure this out and shut teachers out of the hiring process entirely?

In Thunderdome, teaching assignments will be critical. In a shrinking district, student and class distribution will become a matter of professional life and death. With so much riding on it, what do you suppose the odds are that the process will become twisted, driven by concerns other than what's best for the children? How hugely important will it be to smooch your principal's tuchus? How ugly will it be when certain students are turned into human hot potatoes?

How will Thunderdome affect the collegiality, the collaboration, the success of all students? Will it promote the learning of all students, or will it exacerbate the problem of teachers huddling in their own classrooms, keeping for just twenty-five students the educational assistance that could have helped 100?

Yes, teachers are professionals, and caring people, and usually naturally inclined to help and support each other. But Thunderdome raises the stakes. Helping a teacher become as good as, or better than, you would not just be a blow to your ego or trigger some sort of existential crisis-- it would mean your job. We already have evidence (as if we needed any), via multiple test cheating scandals, that when you pit people's devotion to philosophical purity against their desire to feed their children, purity often loses.

Teacher Thunderdome is a dumb idea that would do huge, irreparable harm to schools and create obstacles to student success. There's an old saying about how, when you're in a pack of people trying to escape a bear, you only have to be faster than the slowest person in the pack. That is not a scenario conducive to teamwork. There may be fields where the road to excellence is a one-lane footpath that must is best traveled alone, but in teaching, excellence always depends on the support and assistance of a larger team. To create a system that cuts that team apart, that makes teachers compete with the people they should be sharing with, is just dumb.



Friday, August 15, 2014

Marketing and Mystification for CCSS

Well, we knew this was coming.

Launching in winter of 2014, EdReports, a new non-partisan non-profit, will provide "Consumer Reports-style reviews will highlight those instructional materials that are aligned to the higher standards states have adopted so that teachers, principals and district and state officials charged with purchasing materials can make more informed choices."

Politico calls it a "Consumer Reports for the Common Core." The organization will bring in some teachers and other educationistas to rate materials from various publishers. They'll be starting with "Pearson’s enVision Math, McGraw-Hill’s Everyday Math, Houghton Mifflin’s Go Math and more than a dozen other widely used curricula."

If you are thinking, "Oh, good. Some independent experts will rate these materials and give us an impartial view of which materials are the best," then I have a bridge I'd like to sell you. EdReports.org is brought to you by the usual suspects.

Per Politico, a cool three mill in funding is coming from the Gates Foundation and the Helmsley Trust. Education First, a thinky tank/consulting firm that has teamed up with the Fordham Institute to promote the Core, is "incubating" them. The executive director is Eric Hirsch, previously a big wig at the New Teacher Center (they sell teacher induction) and the Center for Teaching Quality (spoiler alert-- quality comes with the Core). The board chair of EdReports is Dr. Maria Klawe. Dr. Klawe's day job is mathematician and president of Harvey Mudd College, a sort of high-powered STEM school. You might also be interested in one of her side gigs-- one of ten members of the board of Microsoft Corporation.

So what's happening here? My guess is two things-- one obvious, and one not quite so.

First, it's just good marketing.

Common Core has always been in large part about branding and marketing. A nationalized education system where textbook companies don't have to market fifty different flavors of the same product, but can just hawk the same material coast-to-coast --- that kind large scale sales had to get Pearson et al salivating from day one.

But an unregulated CCSS marketplace meant that just anybody could slap a sticker on a book and start cashing in on the new wave. That's not good. For one thing, competition is a Good Thing if only the Right People are allowed to compete. Little fish have to be squeezed out. For another thing, what good are standards if you don't have standards for standardizing the standards. Folks like the Brookings guys have been saying all along that we have the need, the need to weed, as in weeding out the crap that is CCSS is cover sticker only.

An independent-looking verifier of  your product's excellence is super-duper marketing.

Second, the mystification factor.

Here's a quote from Thomas Newkirk, from "Speaking Back to the Common Core."


We are already seeing at work a process I call “mystification”—taking a practice that was once viewed as within the normal competence of a teacher and making it seem so technical and advanced that a new commercial product (or form of consultation) is necessary.

In other words, in the brave new Common Core world, teachers are not capable of choosing textbooks on their own. Their professional judgment is not sufficient to the task-- we need an entire non-profit organization of consultants and experts to truly discern if this math series or those language textbooks are okay to use.

In the old days, a committee of teachers could work as a committee, look through various texts, hear the pitch from salespeople, and make a choice based on what they thought would work best for their program in their schools. Now here comes EdReports to say, "Step aside, little ladies. We wouldn't want you to hurt your pretty little heads doing all this hard pedagogical thinky stuff. Let us just tell you what you want to pick."

Teachers used to be educational experts. Now, apparently, we're not.

Bonus factor: Common Core boosting

One of the things teachers would probably get wrong is considering textbooks and materials based on what would, in their professional opinions, provide the best education for their particular student population while fitting the strengths of their building and district. They might look at all sorts of technical things like the sequencing of concepts and the examples and exercises used to support instruction.

EdReports is here to remind you that what most matters when selecting a textbook is how well it lines up with the Common Core. All those myriad of questions that you ask your self when reflecting on your practices and instruction-- you should only be asking one question. Does this line up with the Core. Because nothing else matters except how well you and your students adapt yourselves to the one size that all must fit.

As I mentioned above, this delightful service doesn't launch for a few months yet (just in time for textbook shopping season). Be sure to alert your district administration so that you can avoid the mistake of letting teachers make up their own minds about materials.

Teachers in South Korea

Remember a few years ago there was a repeated talking point about how teachers are treated in other countries? Heck, the 2011 State of the Union address included this bit of cheerleading-

Let’s also remember that after parents, the biggest impact on a child’s success comes from the man or woman at the front of the classroom.  In South Korea, teachers are known as “nation builders.”  Here in America, it’s time we treated the people who educate our children with the same level of respect.

Well, maybe not.


On the one hand, South Korean teachers are paid well compared to the US-- extremely well as their career continues. Turns out that teachers in South Korea are free to be treated with respect as long as they stay in place and behave themselves.

Korea's top teacher association, the Korean Federation of Teachers' Associations, traces its routes back to 1947; however, nobody seems to think of them as an actual union.  But in 1989, teachers formed a new, feistier union, the Korean Teachers and Education Workers Union (KTU). The government didn't much like the idea then, and they've kept right on not liking it ever since. The union has spoken out on educational issues, including a call to end standardized testing. In response, the government has looked for reasons to shut the union down.

There are about 360,000 teachers in South Korea. About 94,000 of those belonged to this union at one point, but in recent years membership has dropped considerably. Government has kept up pressure, taking actions such as occasionally publishing the list of all the union-affiliated teachers.

The South Korean government has found a variety of creative ways to squeeze the union, including demanding that the full time staffers of the union all get back to the classroom. Union offices were raided and the KTU was accused of making anti-government statements (yes, if you imagine that South Korea is some sort of free and open democratic paradise, I have more bad news for you). And last fall the labor Ministry made a more aggressive move. South Korean law says that trade union membership is restricted to workers actually working in the workplace. The KTU was found to include nine fired members still on their logs.

KTU leadership points to their opposition as politically motivated.

“When she [President Park Geun-hye-- yes, South Korea's president is a woman] was a lawmaker seven or eight years ago she said one harmful insect makes the Korean Peninsula red. It is exactly what she has in mind about the teacher’s union,” says Hwang Hyun-Su, the KTU international secretary. “She thinks that the teacher’s union members are just followers of North Korea or something. It is very old fashioned thinking that exists among Korean conservatives.”

In other words, the union is just a bunch of North Korea-loving commies. Korea's teachers are supposed to be politically neutral. It's not hard to see why the KTU might not be beloved by conservatives.


In the KTU manifesto it says that they believe the Korean education system “cultivates students who are selfish and obedient; we do not teach them to be independent human beings who live collaborating and fulfilling lives.” It also calls for teachers to develop students that can “carry out democratization and destroy all vestiges of the decades of military dictatorships, and who can achieve the reunification of Korea.”

The KTU was still protesting its illegitimate states this summer, and they have successfully overturned a government-imposed shutdown in the past. In the meantime, the nation builders of South Korea are being encouraged to sit down, shut up, and do as they're told.








Classroom Lessons for Cops

Are the Saint Louis County police feeling stupid this morning? I hope so. They should be. The difference between Wednesday and Thursday nights in Ferguson was the difference between chaos and community, between war and peace.The difference certainly wasn't in the crowd; Ferguson's population didn't change over night and they had, if anything, more reason to be enraged after Wednesday's mess.

Some of the press has framed this contrast in terms of different behavior from the crowd or a different attitude or behavior from the crowd, but that's baloney. There is one clear and obvious difference between the two evenings that easily accounts for the different outcomes. This is not rocket surgery-- if you send police determined to preserve and protect, you get a different result than if you send an army to control and command.

Any good classroom teacher can see the lessons of exercising authority (what we usually file under the heading of "classroom management") in the last few days of chaos and unrest in Ferguson.

Own your mistakes. We can hope that the focus on the unrest and subsequent return to peace in Ferguson does not distract people from the event that sparked all of this-- a police officer shot down an unarmed teenager. I have no doubt that as the story continues to unroll, we'll find additional details, conflicting stories, much more information about both Michael Brown and the man who killed him. Like much of America, I'll pay attention to all of that.

But here's the thing. Nothing you can tell me could possibly change the fact that Michael Brown's death should not have happened. Nothing you could possibly tell me would make me go, "Okay, well, I can now see that maybe the police officer did the right thing." Nothing. Michael Brown has been implicated in a "strongarm" robbery. So what. The penalty in this country for boosting baby cigars is not death.

Classroom teachers know-- when you make an inexcusable, obvious mistake, you don't try to stonewall and exert your authority to demand that your students view Something Clearly Wrong as Something Right. I know the impulse-- if I admit I screwed up, it will lessen my standing as an authority figure. But the opposite is true. By insisting that up is down, blue is red, and hot is cold, you only make yourself look ridiculous and, worse, untrustworthy.

All authority is earned. You don't earn authority by being untrustworthy. "I messed up. That never should have happened," may feel very vulnerable, but you can't say anything else and maintain integrity or leadership.

And most importantly-- owning it is the first step in fixing it. And when something is way out of whack (like, say, shooting down an unarmed black teen), then it desperately needs to be addressed, and the causes need to be changed.

Don't bring a tank to a tennis match. Over-reaction is deadly to authority. If I ask a student to sit down and scold him when he balks, that's a proportional response, and he'll feel like he received a just response to his action. If I ask him to sit down and two seconds later start screaming at him and write him up for twelve detentions and tear up his test paper, he no longer sees any connection between his behavior and mine. He will simply see me as an attacker.

When you bring a SWAT team, armored vehicles, and a sniper rifle to a peaceful demonstration, that's not a response. It's an attack.

Don't confuse your enemies and your purpose. One of the striking difference between the police stances of Wednesday and Thursday was that Thursday's police were there to make sure the protesters could exercise their rights.

The assault of the journalists in the McDonalds contains a detail that may seem minor, but I think it's right on point. One of the journalists reported that he was directed first to one door and then to another, and then assaulted when he couldn't quickly sort out the confusing directions. This is exactly what happens when you are not trying to help the person succeed, but have already assumed that he is failing.

We've all met teachers at the point in their career where they haven't worked this out. They complain about a class, complain "How am I supposed to teach that bunch of ignorant barbarians?" Most go on to figure out the answer which is, of course, that you must first stop treating them like ignorant barbarians.

As a person with authority, you have a group of people who are your charges, your responsibility. Your whole purpose is to help them. When you start to view them as your enemy, an obstacle to your work, then you have lost sight of your purpose and you need to check yourself.

It is true that some of our clients are less than ideal. Doesn't matter. You will always deal with people who are different, who have different values, different ways of expressing themselves, different cultural background, different families of origin, and most of all, different levels of ease in dealing with those who wield authority. The job is still the job, and a good wielder of authority recognizes that a person in trouble is a person who needs help.

In short, the Saint Louis County police could have upped their game simply by consulting any experienced first grade teacher. Here's hoping more capable heads prevail in the time ahead.

Thursday, August 14, 2014

A Big Problem with Ed Research

I've always taken a skeptical view of education research. I was in college in the seventies, and I have memories of repeatedly discovering that the Gortshwingle study of How Students Learn was actually a study of how twenty male college sophomores at a small Midwestern college performed a particular task. Educational research seemed to suffer a experimental subject of opportunity problem. And like much research involving human beings and the psychological and intellectual intangibles that drive them, educational research also seemed prone to bias. "I was completely surprised by what the data revealed" seemed not to come up very often. On top of that, designing an experiment that really captures life as it happens in an actual classroom (or, in some cases, on planet earth). Put it all together and I've always found plenty of reasons to view educational research with a very critical eye.

A recently released study suggests that educational research has another huge problem. In "Facts Are More Important Than Novelty: Replication in the Educational Sciences," Matthew A. Makel (Duke University) and Jonathan A. Plucker (University of Connecticut) suggest that there is gaping hole in educational research through which one could drive a fleet of school buses.*

The authors open with a Carl Sagan quote, and then get straight to the central problem:

The desire to differentiate "truth from nonsense" has been a constant struggle within science, and the education sciences are no exception.

Makel and Plucker show us the newly raised stakes-- the US DOE's Institution of Educational Sciences (IES) has been set up as a central clearing house for "real" scientific education research, disseminated through avenues such as the What Works Clearinghouse and the Doing What Works website. But is that research truth or nonsense?

Makel and Plucker walk us through the various ways in which the Randomized Control Trials and Meta-Analyses that make up much of this research can be less-than-solid. Bias, bad design, dumb ideas, poor execution, stupid bosses-- they have a whole list of sourced Ways Things Can Go Wrong in research. The authors are working us around to the major manner in which Real Science corrects for those problems.

Replication.

Since the first primitive lab assistant said, "Woah, that was cool! Can you make it do that again? Can I make it do that again?" or the first proto-scientist said, "Hey, take a look at this and tell me what you see," the backbone of science has replicatable results. Turns out that educational science is more of a spineless jellyfish.

The authors' pored over five years worth of 100 journals to see how often replication had actually happened (they explain their technique in the paper; feel free to check it out yourself). The results were not stunning.

The present study analyzed the complete publication history of the current top 100 education journals ranked by 5-year impact factor and found that only 0.13% of education articles were replications. 

It gets worse. The majority of replicants did in fact confirm the original research. Well, at least, they did that if the replication involved one or more of the researchers who did the original work. If the replication was done by actual third parties who had no stake in proving the original research correct, successful replication was "significantly less likely." The success rate for the original authors in the original journal was 87%. For completely different authors in a new journal, the success rate was 54%.

Not that I'd pay too much attention to that portion, because the sampling is small. The authors looked at a total of 164,589 articles published in the journals. Of those, 461 claimed to be replicants, but the authors determined that only 221 actually were.

So what does this mean? It means that very likely a great deal of what's passed off as research-based knowledge is information that has never been checked, the result of just one piece of research. Imagine if you were seriously ill and your doctor said, "Well, there's this one treatment that only one guy did only this one time, and he thought it turned out well." Would you consider that a hopworthy bandwagon?

The authors maintain a scientific tone as they say "Well, we guess the good news is, hey-- lots of room for improvement." There are lots of ways to address "the rampant problem of underpowered studies in the social studies that allow underpowered studies in the social sciences that allow large, but imprecise, effects sizes to be reported." So this is Not Good, but it is also Not That Hard To Fix.

In the meantime, when confronted with education research, remember to ask a few simple questions. In addition to my own personal favorites ("If this involved studying live humans, what live humans were used? What was the research design?") we should also add "Has anyone ever replicated this research, and can we get a look at that, please?"

In short, just because someone flings  the words "science" and "research" at you, don't assume that you're about to be hit with The Truth.


*Hat tip to Joy Resmovits from HuffPo for pulling this obscure little piece of wonkery into the cold light of twitter.