Your district is out there looking for a program for your school, but you want one that will actually work. So you go to a education clearinghouse to sort through what's out there, to look for one that's effective, that is "evidence-based." This turns out to be a challenge.
Sarah Sparks, writing for Education Week, asked the question: "What Does 'Evidence-Based' Mean?" She had a particular reason for asking, and we'll get to that in a second, but first...
The federal government has an actual answer to that question, and as one might expect when it comes to the feds, the definition turns out to be not very useful.
We can find it buried in Title VIII (General Provisions) under Sec. 8101, bottom of page 129, we find a definition:
Evidence-based.--
``(A) In general.--Except as provided in
subparagraph (B), the term `evidence-based', when used
with respect to a State, local educational agency, or
school activity, means an activity, strategy, or
intervention that--
``(i) demonstrates a statistically significant
effect on improving student outcomes or other
relevant outcomes based on--
``(I) strong evidence from at least
1 well-designed and well-implemented
experimental study;
``(II) moderate evidence from at
least 1 well-designed and well-
implemented quasi-experimental study; or
``(III) promising evidence from at
least 1 well-designed and well-
implemented correlational study with
statistical controls for selection bias;
or
``(ii)(I) demonstrates a rationale based on
high-quality research findings or positive
evaluation that such activity, strategy, or
intervention is likely to improve student outcomes
or other relevant outcomes; and
``(II) includes ongoing efforts to
examine the effects of such activity,
strategy, or intervention.
``(B) Definition for specific activities funded
under this act.--When used with respect to interventions
or improvement activities or strategies funded under
section 1003, the term `evidence-based' means a State,
local educational agency, or school activity, strategy,
or intervention that meets the requirements of subclause
(I), (II), or (III) of subparagraph (A)(i).To break that down and render it in plain English, there are three definitions that are good for federal funding, and two more that... just exist?
Evidence-based means:
1) Shows a statistically significant effect on student outcomes via strong evidence from at least one well-designed and well-done study. You've got at least one study, and it seems like a decent study, and it gives solid evidence.
2) Shows a statistically significant effect on student outcomes via moderate evidence from at least one well-designed and well-done study. Your decent study shows some meh evidence.
3) Shows a statistically significant effect on student outcomes via promising evidence from at least one well-designed and well-done study. Your study evidence is not great, but it can be massaged into looking like maybe better things are coming.
It can also mean (ii) something you kind of think probably could work. Maybe a shade better than an educated guess.
So if you were thinking that "evidence-based" means "we have some solid proof that this actually works," well, no. Just one study that provides any sort of results that don't actually clearly disprove your idea--that's close enough for government work.
Which may be why we end up with the study that gave Sparks her question-- a paper from February of 2023 by Mansi Wadhwa, Jingwen Zheng, and Thomas D Cook entitled "How Consistent Are Meanings of “Evidence-Based”? A Comparative Review of 12 Clearinghouses that Rate the Effectiveness of Educational Programs."
I can't access the paper (well, I could, if I wanted to spend $37.50 to peruse it for 24 hours--one more clue to the Mystery of Why Educators Don't Spend More Time Studying Education Research). But the abstract tells us plenty. The researchers looked at 12 education clearinghouses to see how they measured effectiveness and to see how consistently the measures were applied.
They looked at 1359 programs. Of the programs rated by more than one clearinghouse, only about 30% got similar ratings. In other words, the answer to "Is this program effective," depends pretty much on who you're asking.
As Bill Dagget, founder of the clearinghouse Successful Practices Network, told Sparks:
“If you’re trying to define ‘evidence-based,’ it’s very difficult to incorporate any of the skills that are harder to measure,” like critical thinking, collaboration, or social-emotional development, Dagget said.
Right. You need one good study. And many, many, many, many, many aspects of education are very hard to design decent research for. Particularly when your measure of "success" is nothing more than "did it raise student test scores."
"Student outcomes" is, as always, doing a lot of work. What outcomes? Test scores? Employment? Deeper understanding and comprehension? Happy life? Spoiler alert: way too many researchers go with "test scores" because that's a simple, easy measure.
But if we're going to try to find programs that are evidence based--well, what kind of evidence? Evidence of deeper learning? Evidence of long-lasting comprehension? Evidence of improved skills? Or are we just going to go with test scores again because they make nice numbery data? Do we just end up in the educational version of this conversation?
District attorney: You've been on this case for a while. Do you have anything for me?
Detective: You betcha. I have evidence. Boxes full of evidence.
DA: Great! Evidence of what?
Detective: You know! Evidence! A whole bunch! Loads of it!
DA: Of what?! Of what??!!
Detective: Soooooo muuchhhh evidence!
"Evidence-based" as defined by the law is so broad that it could mean almost anything. But then when we start looking closer, it becomes clear that sometimes it means nothing at all.
No comments:
Post a Comment