How Khan Academy (And Others) Fudged Their Research

Computer tutoring is the hot thing, and the big players have all sorts of sexy research numbers to back them up. Are the numbers bunk? They sure are. 

I'll warn you--this is spun from an article by Laurence Holt, a guy who has worked with NewSchools Venture Fund, Amplify, and, currently, XQ. But most of my readers don't also read Education Next, where the piece appeared in April. But his point is too important to ignore. 

Thanks to COVID, computer-delivered instruction has experienced a boost, from microschool to catch-up interventions. Programs include Khan Academy, i-Ready, Dreambox--but here's the question--

Do they work? In August 2022, three researchers at Khan Academy, a popular math practice website, published the results of a massive, 99-district study of students. It showed an effect size of 0.26 standard deviations (SD)—equivalent to several months of additional schooling—for students who used the program as recommended.

A 2016 Harvard study of DreamBox, a competing mathematics platform, though without the benefit of Sal Khan’s satin voiceover, found an effect size of 0.20 SD for students who used the program as recommended. A 2019 study of i-Ready, a similar program, reported an effect size in math of 0.22 SD—again for students who used the program as recommended. And in 2023 IXL, yet another online mathematics program, reported an effect size of 0.14 SD for students who used the program as designed.

Did you notice a key phrase?

"For students who used the program as recommended."

So how many students is that. Well, Holt checked the footnotes on the Khan Academy study and found the answer--

4.7%

Not a typo. The study threw out over 95% of the results. Holt says that the other programs report similar numbers. 

I suppose the takeaway could be that folks should be trying harder to follow the program as recommended. Of course, it could also be that students who rea motivated to follow the program as recommended are the most ready-to-learn ones. 

But if you hand me a tool that has been made so difficult or unappealing to use that 95% of the "users" say, "No, thanks," I'm going to blame your tool design. 

It's a problem eerily similar to that of ed tech itself, where the pitch to teachers is so often, "If you just change what you do and how you try to do it, this tool will be awesome." When the main problem with your piece of education technology is that it's not designed in such a way that your end users find it actually useful, that is on you. 

In the meantime, schools might want to be a little more careful about how they select these programs. Ed tech companies are interested in marketing, in selling units, and if they have to massage the data to do it--well, the free market. As I've said many times before, the free market does not foster superior quality; the free market fosters superior marketing. And nothing markets in the ed sector like Scientific Evidence Supported by Hard Data.

Always check the data. Always.