How Dr. Perelman Helped Save Australia

(Note: If you're not in the mood to read the whole piece, skip down to the bold quote at the end-- it'll be worth it.)

Longtime readers know that Les Perelman is one of my heroes. Retired in 2012 from teaching and administration at MIT, he has continued his work in the world of education, most notably repeatedly poking holes in the balonified field that is robo-grading. Software that can assess writing is the Great White Whale of testing companies; it would let them offer measuring instruments beyond crappy multiple-choice items without the cost of actual human beings to score writing samples. Not a year goes by without some ed tech firm announcing that they have a super-duper piece of software (nowadays infused with magical AI powder) that can score a piece of student writing. They have been wrong every single time, and likely will continue to be wrong for a long, long time to come.

I have followed many of Perelman's exploits, but somehow I missed his adventure in Australia. And even if you don't think about Australia very often, it's worth noting this adventure because Australia is buffeted by some of the same winds of reformster baloney that blow across US schools.

Australia has had, since 2008, something called the National Assessment Program--Literacy And Numeracy aka NAPLAN. It's a national Big Standardized Test, what PARCC dreamt of becoming when it grew up.

NAPLAN had been sniffing around at the idea of scoring with software since 2012; in 2017, the New South Wales Teachers Federation commissioned Perelman to take a look. The results were not pretty.

The Australian Curriculum, Assessment and Reporting Authority (ACARA) was sad; they had done a report back in 2015 asserting, among other things, that the automated essay scoring "met or surpassed the quality of human markers." I don't know how you figure "surpassed," but I do know how you get software to equal the performance of humans-- you make the humans follow the same dumb rules that the software follows. As Perelman pointed out, the ACARA report "completely ignores" any research that is critical of robot scoring. ACARA's general manager of assessment said that Perelman and the NSWTF are "known critics" of robo-grading, reinforcing the notion that somehow any contrary voices should be ignored. This is not a good way to reach sound conclusions.

Perelman's conclusions were pretty clear:

It would be extremely foolish and possibly damaging to student learning to institute machine grading of the NAPLAN essay, including dual grading by a machine and a human marker.

In January of 2018, the government scrapped the robot scoring plans,  but by then, Perelman had taken a good enough look at NAPLAN's writing assessment that he could see how bad the writing assessment was, and he spoke up about that.

It's the worst one of the 10 or 12 of the international tests that I've studied in depth. It's by far the most absurd and the least valid of any test that I've seen.

The word "bizarre" also turned up. Ouchies. And just so the average layperson could grasp what he was talking about, Perelman released a Guide To A Top Scoring NAPLAN Essay. It's got fifteen whole steps, but it's too good n ot to share, because it wouldn't be entirely useless for certain US writing tests. I'm going to condense them for length here; go look at the original:

Learn a bunch of big spelling words, and throw them in. Don't worry ab out meaning, but do worry about spelling them correctly. Repeat the ideas in the prompt often.

Five paragraph essay all the way. Every paragraph should be four sentences; don't worry about repeating yourself to get there. Start the last paragraph with "In conclusion," then repeat your thesis from graph #1. Somewhere work in a sentence with the structure "Although x (sentence), y (sentence). (Perelman's example-- Although these instructions are stupoid, they will produce a high mark on the NAPLAN essay.)

Use "you" and ask questions. Use connectors like "moreover" or "however."  Start sentences with "In my opinion" or "I believe that" (not for the first or last time, Strunk and White are spinning in their graves). Repeat words and phrases often, and throw in passive voice (whirrrrr). Throw in one or two adjectives next to nouns.

For narrative essays, just steal a story from a movie or tv show-- markers are exoplicitly instructed to ignore that they recognize a story.

And the final and most important rule-- never write like this except for essay tests like the NAPLAN.

Perelman also took a look at the marking criteria (that's where "bizarre" comes in) and lays out the issues there in detail, noting its focus on low-level skills. ACARA's chief said "We will take Dr. Perelman's advice on board," which I assume is Australian for "Bless your heart" or "Go get stuffed."

The sequel to all of this was an award for Perelman from the NSWTF, naming him a Champion of Public Education, an award given not annually, but whenever there seems a call for it. Perelman's acceptance speech has somehow been viewed only about ten times on youtube (and a bunch of those are me) and it's worth a view, but here are some highlights:

Free public education is the cornerstone of a stable democratic and free society.

The main problem with edu-business [for profit entities in education] is that the most important products of education, such as critical thinking and analysis, are both the least tangible and the least profitable. They are expensive both in staffing and in assessment. Edu-business wants to MacDonald-ize education, make it cheap to produce and distribute, highly profitable and with little nutritive value. It wants, like Dickens' Gradgrind, to focus on relatively unimportant facts and rules that can literally be mechanically taught and mechanically counted. Edu-business values psychometricians over practitioners, testers over teachers, reliability over validity.

It's a little long for a t-shirt, but it might be worth the effort.