It's Not A Tutor

There's a new ten-week study from the UK looking at AI as--well, the study people writing about it (like Greg Toppo at The 74) insist on calling it AI tutoring. There's also a recent paper studying the use of Khanmigo which is also characterized as an "AI tutor." We'll look at the results in a moment, but first, I want to look at that term--

"AI Tutor" is yet another marketing term. "Tutor" tries to connect the software to the (somewhat dubious) data showing that tutoring does awesome things for learning! (Two sigma!!) But the use of "tutor" also evokes the idea of a human tutor, a knowledgeable human expert who sits down with the student to provide face-to-face individual support and assistance for learning. That's such a more charming image than, say, a solitary student hunched over a desk and working her way through a ten-inch stack of worksheets.

In fact, if we're thinking about a student ploughing through worksheets or simply buried in a book about the topic at hand, nobody says that students is being tutored. They're just studying. The worksheets are a study aid. The book is just a study aid. And any sort of software-- that's just a study aid being viewed through a computer screen. 

The advantage of actual tutoring (if any) is working with a human who can enter into dialogue about what the student does or does not understand, drilling down to find the mental error the student is tripping over. The tutor also has the advantage of the ability to put human-to-human pressure on the student to perform-- to actually try and keep at it, as opposed to just checking out and/or wandering off. 

GenAI promises that kind of interaction, or at least markets itself on that notion. But the Oreopoulus and Low Khanmigo study found, as did an earlier study this year, that students don't actual interact in that way. Students are supposed to be able to message Khanmigo to discuss issues, but only 17% followed up on a mistake. And many didn't do anything at all. As Khan himself told Matt Barnum of the program's tutoring efficacy for students, “They just didn’t use it much.”

The new study is a trial with ten weeks worth of lessons for 1,179 UK students with Eedi's product, built on Gemini, Google's fine-tuned for teaching genAI bot. Google Deepmind did the analysis and discovered that the AI didn't make much difference compared to "static content." There are differences between girls and boys when it came to AI vs. human tutors, but generally not much difference. Eedi proudly notes that the AI generated hardly any errors. But in terms of advantages, it offered just one-- it's probably cheaper than an actual human tutor. 

What about the 5% problem-- the finding that these programs work great for students who use the program as intended, but that's only about 5% of all students. And there's plenty of reason to think that the 5% students are the ones least in need of help or at the very least would have done just fine with any sort of study aid. Eedi's students engaged more regularly, however there is zero mystery to that finding-- the Eedi sessions were built into regular class time. And 1 in 5 students still ignored the software when a "do you need help" message popped up. Clippy can relate.

AI "tutors" still lack one other element of the human approach-- motivation. Bibi Groot, Eedi's chief impact officer, told Toppo that AI tutoring is “great to support students who are already motivated, but we need to figure out how to motivate students who aren’t motivated in the first place.”

Look, it's not a tutor. It just isn't. It's one more study tool, and for the right student in the right situation with the right material, it might even be a useful one. But a study tool is all it is, and the rush to make it a useful one could only be aided by dropping the charade that it is a replacement for a (costly) human tutor and instead focusing on how it could be a more useful study tool.

But the AI industry isn't going to do that, because it would involve going back to what AI "tutoring" really is-- an extension of the old ed tech Personalized Adaptive Learning model, which was mostly just an endless series of worksheets chosen by an algorithm and fired off at some beleaguered student until they were allowed to unclamp themselves from a screen. But an infinite pile of worksheets was never very compelling-- not even if you shout "But they're coming at you through a computer screen!" 

AI ramps up some of the tech capabilities ("Now, instead of selecting from a library of thousands of worksheets, we can create all new worksheet based on our library of thousands of worksheets!") and the ability to mimic human speech (but not the human thought or insight behind it-- being able to generate a question doesn't make you Socrates). But a study guide, however zippy, is not a tutor.