I have spent plenty of time over the years tracking the fortunes of a high-capacity clown car of computer software that promises to grade those student essays. Labeling the variously inadequate programs AI just adds a new level of marketability to this unholy monstrosity, a piece of Schrodinger's software that is simultaneously totally as great as humans and also just about to be perfected within the next two years (going on twenty years).
I just came across a new model that promises speed, efficiency, validity and reliability. Welcome to No More Marking. Or maybe don't welcome it.
The company serves all the English-speaking world after spawning in the UK, headed up by Daisy Christdoulou. And her Clever Idea is called "Comparative Judgement."
The idea is deceptively and seductively simple. It's hard, goes the argument, to make absolute judgments. If someone walks into your room, can you judge whether they are tall or not? But comparative judgements are easier for human brains-- if two persons walk into your room, you can tell pretty quickly which one is taller.
So what if, instead of reading a student essay and trying to decide whether it was a 94 or 88 or 91 or whatever, you looked at two essays and decided which one was better. Wouldn't that be quick and simple? And if you had multiple teachers working through the same essays in the same way, wouldn't you have tons of data?
This sounds not bad until the first five seconds you spend thinking about it. Oh, but then...
If I used this technique on student height, I could probably generate a pretty good arrangement of students from tallest to shortest. But I still wouldn't know squat about how tall they actually were. Any kind of non-generalized collection of students (a kindergarten class, a group of pro basketball players) would give me particularly unhelpful results. And if the students are very similar in height, suddenly the judgment isn't so easy, and the results are nearly meaningless.
Part of their solution is a "powerful statistical model" involving some fancy maths that generate raw scores that are turned into other scores. Is the result valid? Well, the site tells us "Human Comparative Judgement is the gold standard of human decision-making. It is supported by an extensive research literature." So, you know, there you go. The other part of the solution appears to be a large sample size.
This is also another one of those tech labor transfer systems, because before any of this can start, someone has to feed all of the essays into the computer program. That can include scanning handwritten copies. You'll have to be sitting at a screen to use this. And of course results don't come back until at least one other human scorer runs through the essays, but while you're waiting, perhaps you can go ahead and be a second reader for someone else's essay stack. Are we saving time yet?
The company insists you are. 30 essays would take you two hours the traditional way with a rubric, but with human comparative judgement, you can cut that to an hour. Why are you so much faster reading every essay without a rubric? I don't know.
Don't worry, because we can save time another way, and you knew this was coming. What if some--or even all-- of the "readers" were AI programs? The company suggests going 90/10-- 90% AI and 10% human. 100/0 is of course an option.
Sigh. Okay, the premise of the whole CJ biz is that it's easier and faster for a human to judge which essay is better than it is to evaluate an essay. But that's human beings. It's not clear if the AI in the loop is doing comparative judgements or just offering the usual crappy robograding assessments; the language hints that it's the former, but it's not really clear. If it's the latter, that's bad news because bots are bad at assessing writing, but trying to figure out which of two essays is "better" seems like a whole other level of judgment that AI is not equipped to perform.
If you want to give students the impression that their teacher actually read the work she assigned to them, then you voice-deliver some comments and the AI will spruce that up and attach it.
For the American market, the company offers three national writing assessments. You can throw in a multiple-choice grammar test. The company says they are also teamed up with The Writing Revolution, which isn't encouraging.
The company insists that they are valid and reliable and, hey, the program lets you see where the humans and the AIO disagreed. Christodoulou has a substack, but after digging through the company website I was too grumpy to dig any more. Okay, I looked at one post that made the argument that if an AI comes up with results similar to a human, it must be valid. I've heard this a zillion times, and to me it is an indictment of the degree to which human teachers have been herded into mechanical rubric-centered assessment. All you're telling me is that robots are pretty good at imitating humans who have been trained to imitate robots.
Christodoulou asks some good questions (will knowing they're writing for an AI affect how students write), and she clearly knows that some buzzy items like Bloom's 2 sigma study is bunk. Christodoulou also acknowledges elsewhere that students really care about what their teachers think, and the simple "final product" of a grade is not enough. They've been at this model for a decade or so, so I'm going to assume good intentions. But the site doesn't offer any insights into what standards or training the AI is programmed with, nor the question of how the company deals with the inevitable AI bias and lying about what it has "read."
CJ is an interesting approach, or at least more interesting than the typical "AI so smart grade your essays quick just like human teacher" pitch. But I remain unconvinced.
No comments:
Post a Comment