The Search For Computerized Essay Grading Continues

It is the dream that will not die. For some reason, there are still people who think the world would be a better place if student essays could be evaluated by software, because reasons. The problem has remained the same--for decades companies have searched for a software algorithm that can do the job, but other than deciding to call the algorithms "AI," progress has been slim to none.

And yet, the dream will not die. So now we get a competition, mounted by Georgia State University has teamed up with The Learning Agency Lab (a "sister organization" with The Learning Agency).

The Feedback Prize is a coding competition being run through Kaggle, in which competitors are asked to root through a database of just under 26K student argumentative essays that have been previously scored by "experts" as part of state standardized assessments between 2010 and 2020 (which raises a whole other set of issues, but let's skip that for now). The goal is to have your algorithm come close to the human scoring results. Why? Well, they open their case with a sentence that deserves its own award for understatement.

There are currently numerous automated writing feedback tools, but they all have limitations. 

Well, yes. Primarily they are limited because they don't work very well. The contest says the current automated feedback programs is that "many often fail to identify writing structures" like thesis statements of support for claims. Well, yes, because--and I cannot say this hard enough--computer algorithms do not understand anything in the sense that we mean the word. Computer language processing is just weather forecasting--looking at some bank of previous language examples and checking to see if the sample they're examining has superficial characteristics that match what the bank of samples would lead one to expect. But no computer algorithm can, for instance, understand whether or not your supporting evidence provides good, er even accurate, support.

The competition also notes that most current software is proprietary so that A) you don't even know what it's trying to do, or how and B) you can't afford it for your school, particularly if your school is resource-strapped, meaning that poor kids have to depend on regular old humans to grade their writing.

For extra juice, they note that according to NAEP, only a third of students are proficient (without noting that "proficient" on NAEP is a high bar). They do not cite any data showing that automated essay grading helps students write better, because they can't. 

But if you enter this competition, you get access to a large dataset of student writing "in order to test your skills in natural language processing, a fast-growing area of data science."

If successful, you'll make it easier for students to receive feedback on their writing and increase opportunities to improve writing outcomes. Virtual writing tutors and automated writing systems can leverage these algorithms while teachers may use them to reduce grading time. The open-sourced algorithms you come up with will allow any educational organization to better help young writers develop.

902 teams have already entered; you can actually check their current status on a public leader board. There are lots of fun team names like Feedforward, Pomegranate, Zoltan and Fork is all you need. Plus many that are not in English. Poking through the site, you can see how much the writing samples are referred to ad discussed as data rather than writing; many of these folks are conceptualizing the whole process as analyzing data rather than assessing writing, and in fact there don't seem to be any actual writing or teaching experts in sight, which is pretty symptomatic of the whole field of automated essay evaluation. 

Who is in sight?

Well, you'll be unsurprised to find that the competition thanks The Gates Foundation, Schmidt Futures, and the Chan-Zuckerberg Initiative for their support. Schmidt Futures, the name you might not recognize here, was founded by Eric Schmidt, former Google CEO, to technologize the future.

And if we look at the Learning Agency and the Learning Agency Lab, it's more of the same. The Agency is "part consultancy, part service provider," so a consulting outfit that works to "improve education delivery systems." They tout a team of "former academics, technologists, journalists and teachers." Sure. We'll see.

The outfit was founded by Ulrich Boser in 2017, and they partner with the Gates Foundation, Schmidt Futures, Georgia State University, and the Center for American Progress, where Boser is a senior fellow. He has also been an advisor to the Gates Foundation, Hillary Clinton's Presidential Campaign, and the Charles Butt Foundation--so a fine list of reform-minded left-leaning outfits. Their team involves former government wonks, non-profit managers, comms people and one woman who used toi teach English at a private K-12 school. The Lab is more of the same; there are more "data scientists" in this outfit than actual teachers.

I'm going out on a limb to predict that this competition, due to wrap up in a couple of months, is not going to revolutionize writing assessment in any way. But the dream won't die, particularly as long as some folks believe that data crunching machines can uplift young humans. 

The Other Pandemic Unmasking

At first glance, I suppose it seems like a reasonable set of solutions. 

Expand the pool of who can be a substitute teacher. Anyone with a college degree. Anyone who already works in the building or district. Anybody with a high school diploma. 

In Oklahoma, police officers can now step in as substitute teachers (in Moore, they've already done so). In New Mexico, the governor has called in the National Guard to fill the teacher gap. 

It is amazing how quickly some folks have pivoted from "We must insure teacher and educational quality" to "We must get students into a building with the word 'school' in its name no matter what actually happens once we're inside." It turns out that an awful lot of that big talk about educational excellence and quality was insincere posturing and as long as we can get schools open and students stuffed inside with something resembling a probably-responsible adult with a pulse, that's good enough. Oklahoma has been oh so concerned about making sure nobody was in a classroom indoctrinatin' students, but now it turns they mostly just wants someone--anyone--in that classroom so they can keep the building open.

It is the ultimate expression of "anyone can do that job," even, incredibly, dwarfing the old Teach For America line that we can teach an Ivy League grad everything they need to be an awesome teacher in just five weeks. Hell, now we can teach any adult how to be a perfectly adequate teacher in no weeks.

For what other profession would we consider this a solution. "I'm sorry, Mrs. Fleegleman, but Dr. Hergensheimer is not well enough to perform your heart surgery. But don't worry--Sgt. Blinko from the 15th Precinct will handle it." The doctors and nurses are all out sick, so we'll just have the custodians and administrative assistants run the place. The plumber is too ill to fix your clogged sink, but here's a recent high school grad with a piece of wire. The judge is laid up with illness, so we've brought in the kid who delivers the judge's newspaper. In what other profession would we settle for any warm body to step in for the job.

Look, I know that the defining feature of education during the pandemic has been that all available options stink (though with two years to work on it, we should have figured out how to make some, like remote learning, stink much less). I get that all available choices are sub-optimal. And I have respect for those who have struggled to find a path to quality education for students, even when their choices are not the ones I would make.

But at the same time, there's no escaping that when push came to shove, a whole lot of people decided that they were far more worried about making sure school buildings were open than they were about what was going on inside them. Some have revealed that in their list of priorities, teacher safety and teacher quality come in far behind teacher presence and teacher pulse. Education, shmeducation--just get that child car service running again, whatever it takes!

I just want us all to remember this when the day comes for them to start posturing again. Their mask is off, and we can see what's underneath.

The Fallacy In Learning Loss Panic

Back in March of 2021 (roughly a thousand years ago in pandemic time), I made the argument that Learning Loss is educational halitosis; you start with a real thing, dress it up in some faux science, and use the ensuing panic to sell your preferred remedy. 

The tricky thing about Learning Loss panic is that it's not entirely made up--there are certainly some pieces of some sorts of learning that didn't happen when we were all pandemicking around with schools fully closed and half-assed distance learning and all the rest of the pendemess. But as soon as you start claiming that you can measure what has been lost in months or days or liters or cubic centimeters or hectares of learning, you are shoveling fertilizer in hopes of growing an orchard full of money trees. 

But one element of Learning Loss is just plain made up. Let me tell you what it is, and how to respend to people who try to push it on you.

For maximum panic, some folks are claiming that a drop in test scores due to Learning Loss indicates a future loss of earnings for individuals and economic strength for countries (for example, this from one of the leading promoters of test scores = future earnings, Eric Hanushek). All of this is based on a correlation between test score and life outcomes, except that there are problems with using this correlation.

The big one is that it is just a correlation, like noting that kids who wear larger shoes in fourth grade tend to be taller as adults. There is a connection--it's just not cause and effect. Students who come from a wealthier, whiter background tend to do better on tests. Students who come from a wealthier, whiter background tend to do better in life. In fact, let's trot out this old chart:

There has always been a critical piece of proof missing from the test score = life outcomes assertion. Does changing the score change the student's future? In other words, if I make my fourth grader wear larger shoes, will she grow taller as an adult? 

The very reformy Jay Greene, while of the very reformy University of Arkansas Education Reform Department, looked for evidence of that very thing--and found nothing. 

There is no reason to believe that getting Pat to score higher on the Big Standardized Test will earn Pat more money at work and a better life. None. Raising Pat's test score above the score that Pat would have achieved in some other unboosted alternate universe accomplishes nothing except getting Pat a higher score. (Well, unless you impaired the rest of Pat's education to get that score increase). 

By the same token, if Pat gets a lower test score than Pat would have "normally" achieved, there is no reason to believe that Pat will now suffer lower wages and lifetime earnings, an uglier spouse, unhappier children, and a less friendly dog than Pat would otherwise have experienced. The focus should not be on what a score predicts, but what a change in score from the "expected" score predicts.

People like Hanushek (and the sloppy journalists who depend on him) will say, "Research says that people with low test scores have low lifetime earnings."

The response to this is, "What does research say about how getting a lower-than-predicted test score? Is there research to show that lowering the test score lowers lifetime earnings? How does varying from the expected test score affect the student's future?" The answer is that the research says that it doesn't.

Bottom line: there is zero reason to believe that low pandemic-related test scores are indicative of future financial and economic disaster for individuals and countries.  

Research: Yes, Common Core Was Bad

Did Common Core fail so badly that its failure is visible from another continent? Did it have negative effects on education as a whole? Can fancy research prove what teachers knew a decade ago? Will economists ever get tired of pretending to be education experts? And can researchers get all of this right and still draw the wrong conclusion?

Let's look at a new working paper from the Leibniz Institute for Economic Research, written by Benjamin W. Arnold at the Liebniz Institute for Economic Research at the University of Munich, and M. Danish Shakeel at the Program on Education Policy and Governance at Harvard University.  It's entitled "The Unintended Effects of the Common Core State Standards on Non-Targeted Subjects," and as soon as teachers read that title they can already answer that question, but lets see what these scholars come up with.

The early indicators are a little concerning--among the folks thanked for discussions that "greatly benefited" the paper are Eric Hanushek and Paul Peterson, and there is a great deal of baloney in here, like a repetition of the now-much-aged cheese that test scores are "an important predictor of economic outcomes at the individual and societal level." Really? Does Estonia, long eating the world's lunch on the PISA test, have the leading economy on the globe? The authors will also throw in with the notion that the differences in test scores by race (the "achievement gap") "have been shown to account for relevant shares of the racial/ethnic gap in adulthood social and economic outcomes." 

Given all this, you will be unsurprised to discover that the authors leaned heavily on data from Achieve Inc, a group that intended to cash in big on the Common Core revolution. They code states based on whether they adopted the core "permanently" without noting the distinction of states that dis-avowed the Core and replaced it with the exact same standards copied over on a different template. And central to the data set are results from the NAEP, the maybe-not-really gold standard of US testing.

The paper does offer some droll statements like noting "anecdotal evidence that the CCSS presented challenges in teaching and testing to schools" and that "CCSS-based standardized tests were not always suitable." And there is some impressive stats-and-economist gobbledeegook formulae.

Yet, despite all that, the result they arrive at is exactly the one that actual classroom teachers predicted four paragraphs ago. Exposure to the Core resulted in no improvement in reading and math. Exposure to the core lowered student achievement in subjects other than reading and math. That effect was worse for disadvantaged students.

Why? Well, you already know. But using NAEP teacher self-reported info, the researchers gleaned that "the adoption of the CCSS has shifted the instructional focus away from the non-targeted subjects." 

This is not news to anyone who has, for instance, been told by administration that the budget only has funds for things that will be On The Test. It will not be news to anyone who watched schools cut electives and the arts and even recess to make room for more test prep. The Common Core-based test and punish movement turned schools upside down; instead of being there to serve the students need for an education, the CCSS policies sent the message that students are there to serve the school's need for scores on the Big Standardized Test. And of course that hit disadvantaged students the hardest, as school used batteries of practice tests to identify students who needed to dragged across the cut score line by stripping everything from their education but reading and math test prep. 

Millions of teachers have stories. In my own district, the middle school principal pulled students from history and science so they could have double-periods of math and reading every day, which not only gutted their education, but installed a deep hatred for math and reading (and testing). 

Set up a system that judges schools and teachers based on scores from narrow tests focused (poorly) on two subjects, and you send a clear message to schools--your primary job is to get students ready to take a test on these two subjects. The results of that message are predictable, which is why so many of us predicted them and why research keeps revealing those exact results.

And yet, just in case you think reform-minded folks have learned a lesson from all this, I present the opening sentence from the writers' concluding paragraph:

In terms of education policy, our results suggest that the CCSS might have been more beneficial if it had been adopted for all school subjects. 

Yeah, that's it. The problem with Common Core was that there wasn't enough of it! Of course, the authors want to add science and history, but what about arts and phys ed and health and a few other subjects? 

It's almost interesting to contemplate an alternate universe where Coleman and Zimba somehow decided that other subjects mattered and also had some buddies who wanted to whip up some standards for those subjects, and then we'd have to get Big Standardized Tests for all those subjects and new "aligned" materials and a magic formula to combine and balance all the scores and--well, it would have been a might cluster-farfignugen that would have collapsed under its own weight, which in many ways would have been better than what we ended up with. 

When You Open Schools To Religion...

 There has been a push for a while now to open public schools to religion, and it has been pushed a variety of ways, such as the case Good News Club v. Milford Central School.

That suit made it all the way to the Supreme Court in 2001. The Good News Club is a program of the Child Evangelism Fellowship, a group founded in 1937 by Jesse Levin Overholtzer with the express purpose of evangelizing children. They claim 109,828 clubs worldwide. In the 90s, a couple decided to establish one of these after-school clubs in Milford, New York, but the school said no based on the stated intent to have  "a fun time of singing songs, hearing a Bible lesson and memorizing scripture." Deeming the club religious instruction (which it totally was) the district said no, and many lower courts agreed. SCOTUS, however, did not. Justices Thomas, Rehnquist, O'Connor, Scalia and Kennedy were okey-doke with this (Breyer concurred in part). 

And so US school have to provide use of the facilities to after school religious groups.

And so, here comes the next obvious step.

Meet the After School Satan Clubs. 

Religious counter-protesting has a fine history in this country, with such notable groups as the Church of the Flying Spagetti Monster. The Satanic Temple is one such group, with their not very diabolical motto "Empathy. Reason. Advocacy." They've run a number of attention-grabbing activities, including declaring abortion a sacred ritual as a direct challenge to the Texas abortion laws.

After School Satan Clubs ("Educatin' with Satan") also seem to be out to make a point. The club is aimed specifically at schools with Good News Club chapters, and as its mission--

Proselytization is not our goal, and we’re not interested in converting children to Satanism. After School Satan Clubs will focus on free inquiry and rationalism, the scientific basis for which we know what we know about the world around us.

We prefer to give children an appreciation of the natural wonders surrounding them, not a fear of everlasting other-worldly horrors.

Their FAQ include an answer to the question why:

The pre-existing presence of evangelical after school clubs not only established a precedent for which school districts must now accept Satanic groups, but the evangelical after school clubs have created the need for Satanic after school clubs to offer a contrasting balance to student’s extracurricular activities.

They have made their point in a handful of districts, where parents have seen info about After School Satan Club and freaked the hell out. 

One Moline parent, who shared a photo of the club's flyer online, received incredulous responses to her post, including this: "Wait what????? How is this even a thing? Who approved this?"

The word "approval" is exactly on point--breaking down the wall between church and school will inexorably lead us to a world in which folks call for government to approve religions. Historically, this has never ended well. Somehow people keep expecting that religious freedom should only mean freedom for their own religion. But when you break down the wall between church and school, you should not be surprised when the Satanic Temple walks through the gap. 

(Someone popping up to explain that this is a super argument for choice in 3.... 2..... 1...........)

ICYMI: It's That Time Again Edition (1/16)

 By "that time" I mean time to once again see who will win the annual contest to twist some MLK quote into the pretzel form needed to support their particular cause. Turns out, every year, that MLK would have supported virtually everything. Yay. Here's your reading list for the week.

In our alarmingly unequal society, public schools by themselves cannot be the great equalizer

Jan Resseger has a look at another chapter from an upcoming anthology about public education. This one's by Kevin Welner and it's a good one.

Sheriff uses grades and abuse history to label schoolchildren as potential criminals

In Florida, they're using a Minority Report style system to violate privacy in the name of catching future criminals

Florida officials tried to steer education contract to former lawmaker's company

Corruption in Florida? I am shocked. Shocked! The Tampa Bay Times has the story

This vested interest in the children's incompetence

Teacher Tom has a particularly insightful post here about how some grown-ups are not great with kids.

Florida bill would allow cameras and microphones in classroom

CBS news reports. Come for the terrible new ideas, and stay to find out what terrible old ideas are already being used in Florida classrooms.

Stitt's education bro tries desperately to repair image

Oklahoma's ed chief is doing poorly. This week he really put his foot in it, and tried some light damage control.

Kids on the "McDonalds track" are living in a rigged system

Laura Bryce writes an op-ed for the Inquirer about the mess that is PA school funding

Down goes Frayser

Gary Rubinstein has long kept an eye on Tennessee's Achievement School District, the special state takeover turnaround system that has never done anything but fail hard. Here's the latest update on this sad history.

The pastor, the speaker of the house--and a Christian Academy educator

Nancy Flanagan looks at the sad, greasy tale of Lee Chatfield

Indiana SB 167

The latest version of one of those bills that wants to make sure that teachers don't teach anything a certain parent might find uncomfortable. Blue Cereal Education takes three posts to break it down-- I'll let you start with the third and work your way backwards.

Meanwhile, over at Forbes, I looked at the special features of South Carolina's voucher bill.

Should Schools Teach The Success Sequence

You know the "sucess sequence." It's the idea that if Young People just do the right things--finish school, get a job, get married, have a kid--in that order, they are less likely to end up not poor. It has occasionally been oversold ("Follow these three rules and you will join the middle class!") and the "data" used to bolster it is a little suspicious (like claiming that only 2% of people who follow these rules end up poor anyway--2%?! Really?)

But at the same time, it makes a certain sort of sense, and it's hard to argue that dropping out and having out-of-wedlock babies while unemployed are few people's idea of great life choices. But does it follow that we should, as Rick Hess recently suggested, teach the "success sequence" in school?

I have always had issues with the idea of the success sequence, despite the fact that, as Hess would point out, I followed it myself, sort of. My biggest suspicion is that folks are, once again, confusing correlation with causation. I can believe that the sequence and some level of financial stability go together, but I'm not yet convinced that the causation arow runs from sequence to success and not the other way around. 

Nor is history really on the side of the sequence, given that we are only a generation or two away from the days in which a vast number of Americans did not finish school at all, and yet did perfectly fine. There's also a weird disconnect here for women, who were, for many generations, expected NOT to get a job. They may have been expected to land a husband with a good job, but that's not really the same thing. Nor have good jobs always been available to all people; with a stagnant minimum wage, among other factors, we are looking at a large number of working poor in this country--should they simply consign themselves to never getting married or having children because they can't clear the "good job" hurdle? Hess does acknowledge that many parts of the sequence are beyond the individual's control.

That takes us back to the correlation thing again. Currently the average age for a first marriage in the US is close to 30, which means that folks have to be able to get through a decade or so on their own, though that has changed a lot for women, who up through the 1970s had a median first marriage age of around 20, presumably because they did NOT have to follow the sequence (that age has been steadily climbing since). 

How do we move the sequence into schools? I'm trying to imagine what I would have said to my many students for whom working in high school was an economic necessity. I don't want to imagine students going home from school to announce to their family, "I know why we're poor. It's all your fault." 

Hess links to an article by Philip Cohen who makes a case for why the sequence is bad public policy, noting that costly initiatives to sell the redemptive power of marriage have utterly failed. Of the advice to wait, he says

Success sequencers believe it’s hypocritical to hoard this advice and only dispense it to the children of privilege. But you can’t wish away education, career, and marriage uncertainty or impose order on instability by force of will. If we’re not prepared to guarantee all women the same opportunities as those in my classes have, it’s not reasonable to demand the same attachment to the success sequence that those opportunities make feasible. In the absence of that guarantee, you’re simply asking, or requiring, poor people to delay (until “they’re ready,” in Sawhill’s terms, meaning not poor) or forego having children, one of the great joys of life, and something we should consider a human right.

And he points out the connections between the sequence and race and class

Not coincidentally, the history of welfare politics in the United States is intricately bound up with the history of racism against black women, who have been labeled pathological and congenitally dependent. The idea that delaying parenthood until marriage is a choice one makes is highly salient and prized by the white middle class, and the fact that black women often don’t have that choice makes them the objects of scorn for their perceived lax morals. The framing of the success sequence plays into this dynamic. For example, Ron Haskins has argued that welfare reform was needed to “[change] the values and the approach to life of people on welfare that they have to do their part.” The image of the poor welfare “taker” has a race and a gender in America.

Importantly, Cohen also points out that an attempt to sell the sequence assumes that there are a bunch of young folks out there on the fence, thinking, "Hmm, I can't decide whether I want to be an unwed mother or not." 

But beyond all of that, I can tell you why the sequence will never, ever be adopted as a part of formal education in this country.

Birth control.

The clear, logical implication of the sequence is that teenaged girls should be on birth control until they have reached the proper moment in the sequence. Heck, the success sequence is practically a full-on endorsement of the "I'm not ready for a child yet" case for legal abortion. If you are pushing the sequence as a practical plan for success in life, then it only makes sense to allow teenagers the practical tools that will help them postpone having a child until they're at the right point in the sequence.

Yes, many sequencers like to use the idea to sell abstinence, and that tips the hand of the real idea for many sequencers--that the success sequence is not a practical plan to achieve desired outcomes, but a moral test to see who deserves those "success" outcomes. For some it is another way to make the argument that poor folks are poor because of their own lousy choices, and if you don't want to be poor, make better choices.

As a practical, pragmatic plan, the success sequence could be helpful if we were willing to really back it. Free and easily available birth control starting at the onset of puberty. Raise the minimum wage so that anybody who has a real job has an income good enough to get married and start a family. But as a moral test of worthiness, the success sequence is a dead end. 

Hess is not wrong when he suggests that "we need to focus not only on the structures and externalities that can shape students’ lives but also on what students can do to control their own destinies," though I'm betting you'd be hard pressed to find a school in this country that doesn't. But I think he's overly optimistic when he adds "The success sequence is a compelling, evidence-based, broadly appealing way to do both. In our intensely polarized age, it provides a promising path to talk about opportunity, impediments, and responsibility in a way that may help to span some of our bitter divides." I think talking about it will highlight those divides in new ways, but I'm pretty sure we can all predict that the first set of responses to "So how do we help insure that teenaged girls do not have babies before they're ready?" will find us having old familiar arguments.