ICYMI: Canadian Smokehouse Edition (7/19)

We didn't get it as bad as places like Detroit, but we've been at a good, solid 200+ for a couple of days, and if that number doesn't mean anything to you, congratulations on being Not Downwind of the Canadian forest fires. What a fun summer. My town band canceled a concert for air quality issues, which is the first time that has happened in 170 years. 

It's been a busy time, but I still have a list for you. Remember to share.

Barns, Go-Karts, and Strip Malls: the Wild West of Private Schools that Collect Taxpayer Dollars

This Private School Had Students Scrub Floors and Attack a Fellow Classmate. The State Still Funds It.

How To Research Private Schools Like an Investigative Reporter

If you read just one piece on the list this week, let it be these. ProPublica took a look at the world of taxpayer-funded vouchers and the schools that profit from them. It's thorough, alarming, and hugely informative. Outstanding work from the team.

"Venmo government," administered by ClassWallet

Want to know more about ClassWallet, one of the companies cashing in on voucher administration? Steve Nuzum has the story.

Linda McMahon & How the Pledge of Allegiance Represents a Worse America

Jed Bookout at Frazzled explains how the administration's patriotism push is bad news for the country and education. 

She's So Sure

Audrey Watters with another explanation of what AI does not and can not do. 

Tennessee, Exemplar for Meritocratic Rot

James Horn breaks down some numbers to show that Tennessee is using testing to reinforce inequity in the name of merit.

Teachers save time with AI. Their students may pay the price

I have my doubts about the time saving, but Jill Barshay at Hechinger gets the rest right. 

Counter-Narratives: Mississippi Reading Reform

Paul Thomas explains some different, better ways to understand the narrative of the Mississippi reading miracle.

Found Him in the Front Hallway

At EduHonesty, a look through old discipline referral forms provides some insights into the sort of issues that disrupt a classroom.

Dismantling the Department of Education: Ten Bills That Ask Zero Questions About What Happens to Kids

Bruce Lesley takes a look at those ten bills being used to dismantle the Department of Education, and what they don't say about the students.

Dangers Are Embedded in Federal Waiver Proposal Being Advanced by Ohio Department of Education and the Workforce

Jan Resseger looks at a proposed waiver that would allow Ohio to turn federal money into block grants, part of a troubling trend.

Dear Alpha School: I Hope You’re Right

I'm still baffled by the continued fawning coverage of Alpha School, but this week Robert Pondiscio provided a useful critique from the reformy side of the aisle.

Let's try something a little more soothing this week. Huntertones are proof that you can still start a band in college and keep it going into your alleged adulthood. 

If you sign up, you can get all my stuff in your inbox without any effort. And it's free, now and forever.

Communication Communication Communication

We've been having a couple of weeks of medical adventures here with my 92-year-old mother, and it has reminded me of some of the things we get wrong in public education.

Here is Northwestern PA, most medical care comes one way or another through the folks of Unimaginable Piles of Money Collected. There are advantages to dealing with such an institutional behemoth, most notably a digitized medical record that is up-to-date and available to the patient and the family. It is routine for my sister to see and share test results before the doctor can make it around to the room to talk to us about them, and this is invaluable in terms of coming up with useful questions to ask. 

Hospitals deal with people who are extremely vulnerable, and that makes communication important. Our hospital quality check folks who sometimes amble into the room to ask questions about how well your health care professionals are communicating with you (best question asked-- "Do you know what you are waiting for?")

But this time we went through some real communications whiplash. We knew what they were doing, what they expected, what the goals were-- and then one morning we were moving--rapidly--in a whole other direction. Not because of any new change in the patient's condition; just in how the hospital was responding to it. No real communication about the what or why, catching both us and the facility on the other end of these choices flatfooted. It was an odd moment-- for about 24 hours, while nothing really changed for Mom, it still felt like we were handling an emergency situation.

It sucks enormously to be in a vulnerable medical situation and have nothing communicated to you beyond "We know what we're doing, so just sit there and trust us." It's frustrating and alarming, and if you have a family member who's good at interrogating the functionaries involved, you may make some progress.

The education world should take all of this to heart.

While it may not be a life and death situation, it feels hugely vulnerable to hand your child over to the care of others-- especially if you don't know those others personally. And it surely doesn't help if the school's answer to "What are you going to do with my child" is a hand wave and vague, "You know, school stuff." 

Many districts have an administration-level policy that opposes communication. Not that they phrase it quite that way, but the basic principle is "If we tell people about this, I will get phone calls, so let's just not tell them." This trick never works. Years ago my district contemplated closing an elementary school and decided to deal with the anticipated fallout by keeping the whole business quiet as long as possible. All they got for their trouble was members of the public who were angry about the proposed closure AND the lack of communication.

And it doesn't count as communicating if what you are really trying to do is use a carefully crafted line of baloney to "manage" the public. That is the opposite of communication, and its shiftiness is not hidden if you wrap it in shiny AI slop.

After roughly 100 very lonely parent open house evenings, I absolutely get that attempting communication from the school side can feel like shouting into the void. And we have made progresses-- the pressure to keep that digital gradebook up to date may be annoying some days, but from the parent side, it is a huge blessing. Digital gradebooks, e-mail, even those damned learning platforms-- it is easier than ever for parents to hunt down the information they want.

But none of that means that school districts, school buildings, and classroom teachers cannot do better at communicating to parents. Asking parents to add yet another app to their phones doesn't count. Information passed through channels that are already available and easy does count. For districts and buildings, this may mean spending some money. It is astonishing to me how many school websites do NOT have current information, but I know why it happens--because website maintenance has been assigned to someone who is expected to do updating work in spare minutes between the demands of their actual real job. I know how much time it takes to keep a website current; many school administrators have not a clue.

It is easy, living the dailiness inside the bubble, to imagine that the whole world is awash in the information of what's going on, what people are doing, what is the Usual Stuff of the school. When I passed out of that bubble at retirement, even I was surprised at how little information makes it outside of that bubble. (I wrote to my old district's leadership and board about the issue. "Do you want to do the job," they snarked back. No, because one, I'm retired and two, it needs to be done by someone inside the bubble.)

If you are a district administrator, a building administrator, or a classroom teacher, and you are not reaching outside the bubble at least once a week, you aren't doing enough. Talking to your students doesn't count-- they are inside the bubble with you. For the love of God, do not have an AI agent do it, because the resulting bloated bland and boring product is just noise. You have to let people know what you are doing.

Yes, I know-- lord, I know-- that you don't have time for this. But I am telling you that you can't afford not to take the time. It is how you build trust and support and partnership in your community. It is how you reduce the total amount of unnecessary fear and uncertainty and anxiety in your community (and really, these days, we all need less of that). Will you sometimes get pushback? But the correct response to pushback is to A) examine your premises and practices, B) make your case, and C) listen to the cause of the pushback and decide if it might be worth considering.

If we were better at this in education, reformsters over the last forty years would have heard a lot more, "I don't know what you're going on about. I know what my school does and your claim that they are a disasterific mess are just wrong!"

It's a big ask. It's one more damned thing. But right now I'm living through the difference between navigating with clear complete communication and being left to fumble through opaque and spotty communication, and the difference just feels really huge and the world, including the education portions of it, would be a much better place with less of the latter and more of the former. Inside the bubble, it may not feel like a big deal, but outside the bubble, it's huge. 

NY: District Will Install Teacherbot For the Fall. Yikes.

It is, of course, a rural district, a district located on the Seneca Nation reservation in Western NY.  The Salamanca City Central School District enrolls roughly 1400 students K-12, roughly a third of whom are Native American. 

And they're getting a "humanoid robot" teacher. 

There are so many red flags flying around this story. Take this description from Melissa Manno at New York Focus who talked to Andrew Siguel, CEO of Toronto-based Realbotix (more about them in a bit):

The female robot, named Sally, will have a “lifelike appearance” with silicone skin and long brown hair, Kiguel said in an interview with New York Focus. It will be stationary in a seated position but have a wide range of upper-body movements and facial expressions.

The "female" robot? Robots do not have gender, as far as I know. Students will use a unique id code to announce themselves to the bot, so the bot can id with whom it is speaking. Place your bets now on how many nanoseconds until students decide there is fun to be had by switching id codes. 

The robot will be tied into the school's tech curriculum, which was developed by Steve Sozniak "to prepare students for high-demand tech jobs," and I sure hope they aren't referring to all those coding jobs that are now being handed over to AI.

But hey-- nothing like waving jobs and technology at rural, marginalized districts (the district has roughly 79% low-income students) in an attempt to get them to sign up to be some tech bro's guinea pig.

And it won't just help students, who might, for instance, "upload photos of homework for feedback, ask the avatar to generate lessons on topics that interest them, or receive real-time translations in over 100 languages." You know. Cheat. But teachers benefit, too-

If a teacher loses their place during a lesson or needs a prompt on what comes next, Kiguel said, they can ask the robot for guidance because it has been loaded with the district’s curriculum.

Good lord in heaven. How the hell does a teacher "lose their place" in a lesson, unless it's in a district that is so tightly scripted that teachers are required to memorize the script? Maybe she turned two pages at once? And what the heck does it do to student respect for the teacher if they get to watch the teacher be redirected and corrected by a bot??  

Teacherbot will cost $57,590, but it will not come with some of the advanced features of Realbotix's other products, and if you're thinking "What features" or "How do this company think it's ready to make a roboteacher, anyway," well, hold onto your hat.

Realbotix is already in the humanoid robot business. Specifically, the humanoid sex robot business. 

Realbotix used to be Tokens.com, a crypto company (which let you rent real estate in Zuckerberg's Metaverse) that bought Simulcra, the company behind RealDoll, in 2024, into That was merged with RealBotix with the intent to expand above and beyond the sex robot biz into AI and other robots. That move was funded in part by investor Arthur Hayes, a crypto-bro with legal problems of his own (to add to the confusion, Bed Bath and Beyond just bought the Tokens.com domain). Realbotix also wants you to know that they make companion robots, intended to fight the loneliness epidemic (but not, apparently, with sex). 

The "how did this even happen" part is its own little story. Manna reports that Salamanca's superintendent Mark Beehler says that the "partnership with Realbotix began after a former colleague met an investor at a dinner and discussed the possibility of bringing the company’s robots into the education sector." 

The company provided a demo of the teacherbot "counseling" a child who has been bullied, with basically, "I'm sorry that happened to you. You should go tell a trusted adult." 

And you may be wondering about the wisdom of getting teaching support from AI and its tendency to just make shit up. Realbotix thinks it has that whole "hallucination" thing under control. 

To avoid AI-generated inaccuracies, Beehler said Realbotix trained the robot and avatar to say, “I don’t know,” instead of generating fabricated or misleading responses known as “AI hallucinations.”

That seems both unlikely and inadequate.

Look, I would not be particularly concerned if my child's teacher was a former sex worker, but I would sure as hell be concerned if it weren't even human. And I would be doubly concerned if that non-human was given a bunch of synthetic humanoid features designed to trick my children into thinking of it as a person rather than a plagiarism-fueled next-token-predicting machine. 

It would also be super if all these Frankenstein wannabe's stopped unleashing their ideas on poor, rural families who lack the clout to push back. Behhler says he'll assess the success of this nightmare fuel through "qualitative feedback" from teachers and students. And Ryan Schaaf of Notre Dame told Manna he's optimistic, but that it will require "thoughtful implementation" (yikes) and lots of teacher monitoring and guiding student interaction with the bot (in other words, another new tech making more work). Here's hoping this doesn't waste a whole year of student's learning.                                   

Chatbots Can't Read Your Essay (Part 1,299,437)

This is one more example of how a chatbot will NOT read your writing and provide useful insights into what you have created. I referenced this piece of mini-research before, but it deserves its own post (partly so I can find it easily when I want to reference it again).

Adam Kucharski is, according to his bio, a mathematician/epidemiologist. He wrote Proof: The Art and Science of Certainty, and his work has appeared in Wired, Guardian, and New Scientist, among others. 

Back in May, he published an account of a simple experiment. It started with a simple question-- how good is Copilot at finding insights in a data file. 

To test it out, I asked Copilot to look at differences in how people in US and UK expressed emotions in an Excel dataset that contained thousands of survey responses.

Copilot found all sorts of deep insights. The responses differ "mainly in tone, intensity, and wording style," the bot typed. And it went on to provide deeper analysis. "US responses were typically more direct and emotionally amplified, while UK responses were more restrained." or how about "UK responses use more poetic, metaphor-rich phrasing; US responses favor plain, action-oriented wording." And even "UK wording reflects understatement and nuance; US wording emphasizes clarity and emphasis."

Wow, that is some deep and nuanced analysis. 

Except that the dataset wasn't real-- and it wasn't different. Kucharski explains his technique:

First, I’d created 2000 free-text responses and labelled them ‘UK’. Then I copied and pasted the exact same 2000 responses but labelled these ‘US’. Finally, I combined them to create a dataset of 4000 total responses, and jumbled them up.

So Copiloty looked at two identical data sets--not made by a human in any country-- and found all these nuanced differences between them. It's almost as if the chatbot is just making shit up.

So he tried it again:

This time, I got an LLM to simulate 200 statements about career aspirations. Then I duplicated the dataset five times, labelling each one ‘US’, ‘UK’, ‘France’, ‘Germany’, ‘Italy’.

Again, the bot found a variety of differences and nuanced distinctions between the five different countries. It even generated a chart that shows the likelihood of people in a particular country pursuing particular careers-- In France, they are twice as likely as the UK to pursue a career in the arts. US citizens are most likely to pursue a career in business, and way more likely to pursue one in medicine. 

Again-- all this analysis is based on identical datasets generated by a bot that is not a citizen of any country. 

Kucharski's post gets into some specifics and offers links to his datasets. Plus some updated follow-ups by other folks who took a stab at the same experiment. And it addresses a bit the question of "Well, if you used a different process..." You know who doesn't have knowledge of the best ways to trick the bot into doing the thing for real? The kind of folks who think the computer is smart and will help them with their writing.

Any time someone tries to tell you that a chatbot can be a useful tool for analyzing your writing, you should be thinking of this (and many other) actual demonstrations of how badly bots do at this task. 

One of the most useful checks I've seen came from anonymous writer online who suggested that the best way to think of a prompt is that you're not asking "Do an X for me" but instead "What would an example of X look like?" The bot is not going to do an actual analysis-- it's just going to extrude an example of what an analysis would look like based one whatever stolen analyses it was trained on. 

It is not reading your work. It is not analyzing your work. It is most definitely not your writing partner or editor.

ICYMI: Under Water Edition (7/12)

It has been a week. Maybe we'll talk about it some other time. But I still have a reading list for you.

I Asked My Students About AI Again

Marcus Luther talked to his students about AI. Maybe that tidal wave isn't coming.

The Real Cost of Fifty Years of Policy:

Greg Wyman puts the last several decades of education reform in context.

What Schools Can’t Do

David Larabee reprints a speech from 2009, and it is taking perhaps a broader view than classroom practice, but boy dose some of this really hit. Like:

we pass off on schools social problems that we are unwilling to accomplish through the political process, where the capability for success actually resides. Instead of addressing these problems directly through political action, we foist them off on schools and then blame them for continually falling short of the desired goal.

And also:

Reformers are loath to give up their aims in the service of making the reform acceptable to teachers, so they tend to plow ahead in search of ways to get around the obstacles. If they can’t make change in cooperation with teachers, then they will have to so in spite of them. They see a crying need to fix a problem through school reform, and they have developed a theory for how to do this, which looks just great on paper. Standing in the state capital or the university, they are far from the practical realities of the classroom, and they tend to be impatient with demands that they should respect the complexity of the settings in which they are trying to intervene.

There's plenty more where that came from.

AI: The Trust-Breaker

Matt Brady examines the effects of AI use on the trust necessary for schools to work.

Where's the Science?: Not in Think Tank Reports or Reading Legislation

Nobody has critiqued the Science of Reading movement more relentlessly that Paul Thomas. Here's his reaction to a new "report" from the National Council pn Teacher Quality.

‘Wasteful and extravagant’: Here’s what Utah auditors found digging into the first year of school voucher spending

Utah's new Arizona-style voucher program seems riddled with Arizona-style abuse and fraud. Reported by Carmen Nesbitt for the Salt Lake Tribune.

The GOP-created university civics centers aren't popular. They want to require attendance.

Ohio set up university centers to combat liberal bias, but students weren't interested, so now the GOP, champions of the free market, will just force everyone to attend. Morgan Trau reports for News 5 Cleveland.

Let AI Burn

If you like your anti-AI arguments brutal and uncompromising, Ed Zitron is your guy.

Looping. It’s Not New. It’s Not a Panacea.

Nancy Flanagan on the recycling of old education ideas.

Arizona students with disabilities fuel rapid growth in billion-dollar ESA voucher program

What do you suppose would happen if you set up your taxpayer-funded vouchers system to incentivize a diagnosis of special needs? Craig Harris reports for 12 News.

Families hoard ESA funds as Arizona public schools face low funding, records show

Arizona's taxpayer-funded vouchers are also piling up big money stockpiles for some parents. Craig Harris again.

Teachers in Michigan school district can pick from 32 phrases or face legal review over classroom posters

How stupid can the attempts to police teacher posters get? Pretty stupid. Lily Altavena at Chalkbeat.

Big Changes to Federal Grants Are Coming: What They Could Mean for Schools

Much of the boring routine funding from the feds is about to be disrupted. Marl Lieberman at EdWeek reports.

“One Big Beautiful Bill” Has Already Begun Damaging Children’s Well-Being: It Will Only Get Worse

Jan Resseger looks at the worst side-effects of that thing.

AI Data Centers: Greed Rather Than Need.

The indispensable Mercedes Schneider looks at the threats coming from data centers.

Teens Arrested for Breaking Into High School With a Homemade Plasma Cannon

Are you worried that Kids These Days just aren't as ambitious or creative as they used to be? This is a story for you. Luis Prida at Vice covers a North Carolina story.

This week at Forbes.com, I took a look at how Arizona is one step closer to imposing actual accountability on its taxpayer-funded voucher program. 

A video artifact-- a medley from the cast of the then-current musical Hair, on Ed Sullivan. Asks your parents (or grandparents) what's going on here. 

The Federal Religion Commission on Education

The federal Religious Liberty Commission just coughed up its 224-page report, and I'm not about to shovel through the whole thing here, but it does devote a whole chapter to education, and that's worth a look for what it tells us about the MAGA view of education.

Much of the report is taken up with some anecdotes from students who allege various versions of being picked on in/by schools for their faith. One fifth grade student was "forced" to read a book about transgenderism to his kindergarten buddy, and this qualified for testimony because somehow being anti-trans is a Christian belief. One student was denied the right to wear a covid mask with "Jesus Loves Me." One student was denied the right to thank Jesus in his graduation speech. One girl's attempt to sing a Christian song in the school talent show. One girl wanted to pray with friends in a corner of the cafeteria and was denied. And one more student was subject to antisemitic bullying.

The stories are dramatically embellished (when the graduating student went ahead and thanked Jesus anyway, everyone clapped, teachers embraced him, and the principal cried). Sure. And we'll never know just how quiet the cafeteria prayer really was. But while the trans book story is complicated, the other stories, if accurate, are stories of students whose rights were, in fact, violated by their school. 

They are anecdotes. The commission might have included others, like the story of the kindergartners who wore hijabs to graduation and were held up for abuse by the President of the United States. There are undoubtedly other anecdotes that bend the other way; take my own former superintendent who routinely opened elementary graduation ceremonies with a Jesus prayer, or my former student whose fourth grade teacher who pressured her to convert from Judaism to Christianity. 

The commission deals with the anecdotal nature by offering a blanket observation that the "story is not an isolated event." 

That's fine-- providing a compelling narrative is how folks do the commission testimony thing. But what's more notable is the principles outlined to provide context for the narratives.

The commission offers the now-familiar tale of how institutions have been captured by the Bad Guys. For a century, they assert, teacher training institutions "have increasingly framed education not as the transmission of intrinsic moral truths, but as a vehicle for ideology-driven social transformation." 

Well, now. "Social transformation" has arguably always been the point, as in transforming society by trying to make everyone numerate, literate, and knowledgeable. "Ideology-driven" only makes sense as analysis if you remember, as implied by this construction, that "ideology" only refers only to ideology followed by Those Guys, while our ideology is actually "truth." Either way, it's a stretch. Virtually every teacher I've ever known or worked with mostly had a classroom ideology of Can We Please Learn This Content. Though I'll bet you can find a few ideologues of every flavor in classrooms here and there about the country (there are over 4 million teachers, guaranteeing at least one of everything you can imagine). 

But what really strikes me is the commission's idea of what schools are supposed to do--

The transmission of intrinsic moral truths.

That's a phrase doing a lot of heavy lifting, but dovetails with the notion that the world is all absolute truths, probably already figured out by some dead Greek and Roman guys. Once grasped, the truths allow for no negotiation, discussion, and certainly no revision (see also the argument that we don't need to respect all religions because everything that's not Christian is just wrong). 

They go after John Dewey (though I'll bet you that there aren't 25 teachers working in 2026 who could name two things Dewey believed) and accuse him of this kind of stuff:

He proposed that schools create an “embryonic community life,” where children would be taught primarily not how to think or explore truth, but to develop habits useful to the state, such as cooperation, shared purpose, and democratic citizenship.

Dewey gets charged with re-characterizing schools as "a vehicle for ideologically driven social progress." Again, I think his offense here is not going after "social progress" so much as letting the incorrect ideology have a say. 

Their example is sex ed. After the sexual revolution of the sixties (the decade when everything went to shit for these folks) schools started teaching sex ed and exposing young kids to discussions of sex (bad, because otherwise young humans would not have ever heard about sex). Also, they taught birth control instead of abstinence, which is bad, because schools should prioritize politically correct over actually works.

The commission argues that "ideological current has too often left little room for students who hold traditional religious convictions, such as the belief that God created man and woman, that moral truth is real, or that America—though imperfect—is not fundamentally evil." Maybe not so much "left little room" as "didn't allow these beliefs to occupy a central place over all others." And I'm not sure how beliefs about America qualify as traditional religious convictions (even if, as they apparently mean, only certain religious traditions count here). But humans have a historic tendency to confuse their own personal beliefs with the Will Of God.

The report positions all this debate of the First Amendment as a battle over culture:

It is part of a larger cultural struggle over whether schools will form children in the moral inheritance of Western civilization or in a newer creed that often treats historic religious belief as something backward, suspect, and even harmful.

"Form children." Yikes. (I refer you to Russell Barkley's contention that we are shepherds, not engineers.) And historic religious belief has, at times, been backward and harmful, and remembering that ought to inform any discussion of the First Amendment and what the founders had in mind.

But from there we're on to the argument that some sort of hostile secular religion displaced "the 2,000-year canon of Western moral and religious tradition" (the writers are careful not to explicitly argue for Christianity by name, but they might as well have). 

This is a bogus argument and always has been. Public schools have no official position on vegetarianism. Their only mandate is to offer an assortment of healthy choices and the freedom to pack a lunch from home. Likewise, the First Amendment mandates that religion not be on the menu of what schools offer, leaving students free to practice as they wish. There is no such thing as "secular religion," rather, "secular" is what is left when you remove all the religious parts. 

We can find three issues in the commission's complaint.

One is that some schools, over-zealous (or afraid) about church-state separation take steps to squelch student religious expression when they should not. School officials should not lead a religious prayer at graduation, but if student speakers want to thank Jesus, they should. Admittedly, the line is sometimes tough to spot-- if a student, as part of a school project, chooses to paint a religious image, that's probably okay, but if the painting is on the wall of the school, the issue is fuzzier. 

Second is the part they try to avoid saying out loud, which is the belief that White Christianity should be the dominant culture in America, and that should start in school. 

Third is the notion, repeatedly taken into court, that one cannot fully exercise one's religion unless one is free to inflict it on others. That includes financing that exercise with public tax dollars and discriminating against certain Others. This goes hand in hand with having a dominant position. 

Some folks will absolutely regret it if they ever win these debates. For one, they will find themselves swamped with people who want to claim a religious free-money-and-no-rules card. For another, the inevitable result will be government commandeering and regulation of religion-- you know, like setting up a federal commission to oversee religion in this country. The First Amendment is just as much about protecting the church as it is protecting the state.

The commission offers all sorts of recommendations. Some involve providing more guidance about First Amendments, which isn't a bad idea depending on whether or not the training is related to the actual language. "Know Your Rights" posters for school. Sure; posters really work in school. Set up a portal so that people can report naughtiness on line. Sic the DOJ on those naughty schools. Get behind courtroom support for the transparently unConstitutional laws requiring Ten Commandments displays. Also, school choice. They don't bother to explain why they want it; they just want it.

Nothing here you wouldn't expect from a commission headed up by Dan Patrick and including Ben Carson, Dr. Phil, Franklin Graham, Cardinal Tim Dolan, Paula White, and Todd Blanche. Can't wait to see what they come up with next.

A New Bot-enhanced Assessment Approach

I have spent plenty of time over the years tracking the fortunes of a high-capacity clown car of computer software that promises to grade those student essays. Labeling the variously inadequate programs AI just adds a new level of marketability to this unholy monstrosity, a piece of Schrodinger's software that is simultaneously totally as great as humans and also just about to be perfected within the next two years (going on twenty years).

I just came across a new model that promises speed, efficiency, validity and reliability.  Welcome to No More Marking.  Or maybe don't welcome it.

The company serves all the English-speaking world after spawning in the UK, headed up by Daisy Christdoulou. And her Clever Idea is called "Comparative Judgement."

The idea is deceptively and seductively simple. It's hard, goes the argument, to make absolute judgments. If someone walks into your room, can you judge whether they are tall or not? But comparative judgements are easier for human brains-- if two persons walk into your room, you can tell pretty quickly which one is taller.

So what if, instead of reading a student essay and trying to decide whether it was a 94 or 88 or 91 or whatever, you looked at two essays and decided which one was better. Wouldn't that be quick and simple? And if you had multiple teachers working through the same essays in the same way, wouldn't you have tons of data?

This sounds not bad until the first five seconds you spend thinking about it. Oh, but then...

If I used this technique on student height, I could probably generate a pretty good arrangement of students from tallest to shortest. But I still wouldn't know squat about how tall they actually were. Any kind of non-generalized collection of students (a kindergarten class, a group of pro basketball players) would give me particularly unhelpful results. And if the students are very similar in height, suddenly the judgment isn't so easy, and the results are nearly meaningless.

Part of their solution is a "powerful statistical model" involving some fancy maths that generate raw scores that are turned into other scores. Is the result valid? Well, the site tells us "Human Comparative Judgement is the gold standard of human decision-making. It is supported by an extensive research literature." So, you know, there you go. The other part of the solution appears to be a large sample size.

This is also another one of those tech labor transfer systems, because before any of this can start, someone has to feed all of the essays into the computer program. That can include scanning handwritten copies. You'll have to be sitting at a screen to use this. And of course results don't come back until at least one other human scorer runs through the essays, but while you're waiting, perhaps you can go ahead and be a second reader for someone else's essay stack. Are we saving time yet?

The company insists you are. 30 essays would take you two hours the traditional way with a rubric, but with human comparative judgement, you can cut that to an hour. Why are you so much faster reading every essay without a rubric? I don't know.

Don't worry, because we can save time another way, and you knew this was coming. What if some--or even all-- of the "readers" were AI programs? The company suggests going 90/10-- 90% AI and 10% human. 100/0 is of course an option.

Sigh. Okay, the premise of the whole CJ biz is that it's easier and faster for a human to judge which essay is better than it is to evaluate an essay. But that's human beings. It's not clear if the AI in the loop is doing comparative judgements or just offering the usual crappy robograding assessments; the language hints that it's the former, but it's not really clear. If it's the latter, that's bad news because bots are bad at assessing writing, but trying to figure out which of two essays is "better" seems like a whole other level of judgment that AI is not equipped to perform.

If you want to give students the impression that their teacher actually read the work she assigned to them, then you voice-deliver some comments and the AI will spruce that up and attach it. 

For the American market, the company offers three national writing assessments. You can throw in a multiple-choice grammar test. The company says they are also teamed up with The Writing Revolution, which isn't encouraging.

The company insists that they are valid and reliable and, hey, the program lets you see where the humans and the AIO disagreed. Christodoulou has a substack, but after digging through the company website I was too grumpy to dig any more. Okay, I looked at one post that made the argument that if an AI comes up with results similar to a human, it must be valid. I've heard this a zillion times, and to me it is an indictment of the degree to which human teachers have been herded into mechanical rubric-centered assessment. All you're telling me is that robots are pretty good at imitating humans who have been trained to imitate robots. 

Christodoulou asks some good questions (will knowing they're writing for an AI affect how students write), and she clearly knows that some buzzy items like Bloom's 2 sigma study is bunk. Christodoulou also acknowledges elsewhere that students really care about what their teachers think, and the simple "final product" of a grade is not enough. They've been at this model for a decade or so, so I'm going to assume good intentions. But the site doesn't offer any insights into what standards or training the AI is programmed with, nor the question of how the company deals with the inevitable AI bias and lying about what it has "read." 

CJ is an interesting approach, or at least more interesting than the typical "AI so smart grade your essays quick just like human teacher" pitch. But I remain unconvinced.