We've been hearing about the joys of evidence-based curriculum for years, so why isn't it happening? Here are some thoughts.
Over at Education Next, the website run by Harvard Kennedy School's Education Policy and Governance Program, Morgan Polikoff and Jake Rowley address "The Myth of the Evidence-Based Curriculum," and I'm not going to walk you through the whole thing, I do want to highlight a couple of pieces.
First, we should remember what "evidence-based" means in the Every Student Succeeds Act, because it does not mean what you think it means. There are four tiers:
Tier I: Strong Evidence. At least one good paper, not contradicted by other studies, showing the "intervention" will raise their test scores.
Tier 2: Moderate Evidence. One "quasi-experimental study" that may show some correlation between the "intervention" and good results.
Tier 3: Promising Evidence. A correlation study. Which is a really weak approach (see this website for many Spurious Correlation examples).
Tier 4: Demonstrates a Rationale. "We've read a few papers and we have what we think is a pretty good theory." All you need is a logic model and a good argument. Maybe a plan to do a study.
To the layperson, tiers 3 and 4 may look an awful lot like "no actual evidence," and I would agree. The original language of ESSA has been periodically re-interpreted, and Polikoff and Rowley wrap them in some pretty academic language, but it has always seemed to me that the only "evidence" that ESSA rules out is "shit you just made up." And I'm not sure you couldn't get that in if you dressed it up enough.
The idea under ESSA was always that the feds and the states would start gathering up all sorts of Education Ideas, running them through the evidence-inator, and creating a collection of the good ones that teachers could then dip into. That hasn't happened. Instead, the authors note, publishers are doing the work, which is suboptimal. The vision would be lovely, they note
But many obstacles impede that vision and underscore the need for more independent, centralized curriculum evaluations, ideally led by states or foundations.
So why can't we have a centralized curriculum evaluation bank? They have a list, and I have some thoughts about it:
Unrealistic timelines. A real, serious evaluation of materials would take years, including years that schools were using the materials without any evidence about their effectiveness. So publishers measure over just one year, which doesn't really mean squat. The authors' timeline issue is also, really, another issue-- it's not okay to experiment on students whose educational years cannot be replaced if it turns out they've been wasted testing junk.
Logistics. Another version of that same problem-- no district really wants to volunteer its students as test subjects. Nor are schools interested in volunteering for a randomized test where their curriculum is upended so that different materials can be assigned randomly. My head hurts to imagine teaching in a department where every teacher is working from different textbooks.
Questions of Validity. The limits covered in those first two items mean that studies are often performed on non-typical groups. A school that would allow its students to be guinea pigs or its curriculum scrambled for randomization is probably not a typical district. So even if a study is completed, there's no reason to feel assured that it produces a result that's true for the general population of students.
Measuring student outcomes. As I may have mentioned a few gazillion times, the use of the Big Standardized Test as a proxy for or measure of student achievement is bunk (and for the many states that require the BS Test to be taken on computer, double bunk). Daniel Koretz's The Testing Charade came out in 2018, and nobody in the pro-testing crowd has yet come up with a convincing response. This is the fatal flaw in a mountain of educational research-- the research is based on what they think they can measure instead of what they need to measure for any sort of useful result. This one problem overshadows all the others.
The thorny issue of implementation. You can develop a textbook and curriculum materials all you like, but after 39 years in the classroom, I can confidently assure you that the number of teachers implementing your stuff exactly the way you want them to is pretty much zero. If you are measuring how the curriculum is written and not how it is actually implemented, you are just measuring unicorn horns.
The replication crisis. I don't know if I'd call it a "crisis," other than it represents a crisis to researchers who want to be taken seriously. But multiple studies have now shown that the majority of research done in some fields are not replicated. A piece of research that can't be replicated is a piece of research that doesn't show what its authors claim it shows. If you want me to adopt your curriculum, show me three or four studies that support it. Just one study is no better than zero studies.
Cost barriers. Polikoff and Rowley are talking about costs for the publishers. Research is expensive. Creating new materials is expensive. On the other hand, I remember decades ago when Harcourt-Brace had a hugely popular literature series for high schools and within a few years, every other publisher had "developed" something similar. Cost barriers get passed along. For example, the transition to digitized curricular materials was marked by publishers declaring, "Well, if we sell these digital resources for less than hard copy books, we will lose money, so let's just make the digital stuff just as expensive." In other words, part of the cost of digital resources was the lost income-- so that cost was just passed along. And then they discovered licensing fees! It's expensive to do all this research in developing materials, especially if you can't make the money back by charging more for your better materials. So why bother?
So how could anyone ever have thought that the massive, centralized evaluation of curriculum materials was ever going to happen. Independent third-party evaluators were going to do the job because.... they felt a moral obligation? The state didn't want to pay them, so the publishers did. And then the research can be conducted without disrupting schools, using students as test subjects, or settling for inadequate measures? Nope, nope, and nope. If pigs had wings, they could fly, but until they start feathering up, we should stop wasting money and time on pig airports.





