The Search For Computerized Essay Grading Continues

It is the dream that will not die. For some reason, there are still people who think the world would be a better place if student essays could be evaluated by software, because reasons. The problem has remained the same--for decades companies have searched for a software algorithm that can do the job, but other than deciding to call the algorithms "AI," progress has been slim to none.

And yet, the dream will not die. So now we get a competition, mounted by Georgia State University has teamed up with The Learning Agency Lab (a "sister organization" with The Learning Agency).

The Feedback Prize is a coding competition being run through Kaggle, in which competitors are asked to root through a database of just under 26K student argumentative essays that have been previously scored by "experts" as part of state standardized assessments between 2010 and 2020 (which raises a whole other set of issues, but let's skip that for now). The goal is to have your algorithm come close to the human scoring results. Why? Well, they open their case with a sentence that deserves its own award for understatement.

There are currently numerous automated writing feedback tools, but they all have limitations. 

Well, yes. Primarily they are limited because they don't work very well. The contest says the current automated feedback programs is that "many often fail to identify writing structures" like thesis statements of support for claims. Well, yes, because--and I cannot say this hard enough--computer algorithms do not understand anything in the sense that we mean the word. Computer language processing is just weather forecasting--looking at some bank of previous language examples and checking to see if the sample they're examining has superficial characteristics that match what the bank of samples would lead one to expect. But no computer algorithm can, for instance, understand whether or not your supporting evidence provides good, er even accurate, support.

The competition also notes that most current software is proprietary so that A) you don't even know what it's trying to do, or how and B) you can't afford it for your school, particularly if your school is resource-strapped, meaning that poor kids have to depend on regular old humans to grade their writing.

For extra juice, they note that according to NAEP, only a third of students are proficient (without noting that "proficient" on NAEP is a high bar). They do not cite any data showing that automated essay grading helps students write better, because they can't. 

But if you enter this competition, you get access to a large dataset of student writing "in order to test your skills in natural language processing, a fast-growing area of data science."

If successful, you'll make it easier for students to receive feedback on their writing and increase opportunities to improve writing outcomes. Virtual writing tutors and automated writing systems can leverage these algorithms while teachers may use them to reduce grading time. The open-sourced algorithms you come up with will allow any educational organization to better help young writers develop.

902 teams have already entered; you can actually check their current status on a public leader board. There are lots of fun team names like Feedforward, Pomegranate, Zoltan and Fork is all you need. Plus many that are not in English. Poking through the site, you can see how much the writing samples are referred to ad discussed as data rather than writing; many of these folks are conceptualizing the whole process as analyzing data rather than assessing writing, and in fact there don't seem to be any actual writing or teaching experts in sight, which is pretty symptomatic of the whole field of automated essay evaluation. 

Who is in sight?

Well, you'll be unsurprised to find that the competition thanks The Gates Foundation, Schmidt Futures, and the Chan-Zuckerberg Initiative for their support. Schmidt Futures, the name you might not recognize here, was founded by Eric Schmidt, former Google CEO, to technologize the future.

And if we look at the Learning Agency and the Learning Agency Lab, it's more of the same. The Agency is "part consultancy, part service provider," so a consulting outfit that works to "improve education delivery systems." They tout a team of "former academics, technologists, journalists and teachers." Sure. We'll see.

The outfit was founded by Ulrich Boser in 2017, and they partner with the Gates Foundation, Schmidt Futures, Georgia State University, and the Center for American Progress, where Boser is a senior fellow. He has also been an advisor to the Gates Foundation, Hillary Clinton's Presidential Campaign, and the Charles Butt Foundation--so a fine list of reform-minded left-leaning outfits. Their team involves former government wonks, non-profit managers, comms people and one woman who used toi teach English at a private K-12 school. The Lab is more of the same; there are more "data scientists" in this outfit than actual teachers.

I'm going out on a limb to predict that this competition, due to wrap up in a couple of months, is not going to revolutionize writing assessment in any way. But the dream won't die, particularly as long as some folks believe that data crunching machines can uplift young humans.