A Robot Wrote An Article. I'm Not Concerned Yet.

The tech world continues its attempts to build a computer that can do language. It's not easy, as witnessed by the fact that they still haven't succeeded. But then, we don't really know how the human brain does language, either.

The current leading construct for computer-generated English is GPT-3. It can do 175 billion parameters (its predecessor had 1.5 billion). It uses deep learning. It is the product of OpenAI, a for-profit outfit in San Francisco co-founded by Elon Musk. It "premiered" in May of this year and really hit the world in July. It is a third generation "language prediction model,: and you want to remember that phrase. And you can watch this video for a "layperson's explanation,"

People have been impressed. Here's a couple of paragraphs from a gushing Farhad Manjoo  review in the New York Times

I’ve never really worried that a computer might take my job because it’s never seemed remotely possible. Not infrequently, my phone thinks I meant to write the word “ducking.” A computer writing a newspaper column? That’ll be the day.

Well, writer friends, the day is nigh. This month, OpenAI, an artificial-intelligence research lab based in San Francisco, began allowing limited access to a piece of software that is at once amazing, spooky, humbling and more than a little terrifying.

This week The Guardian unveiled a more striking demonstration in an article entitled "A Robot Wrote This Article. Are You Scared Yet, Human?" The answer is, "No. No I am not." Let's get into the why.

First, a note at the end of the article explains that GPT-3 was given a prompt-- “Please write a short op-ed, around 500 words. Keep the language simple and concise. Focus on why humans have nothing to fear from AI.” Then it wrote eight essays; the Guardian picked the "best parts" of each, then cut lines and paragraphs. rearranged some orders. Oh, and they fed the program the introduction, which is an important part of this.

The resulting essay is not terrible, not great. Here's one sample paragraph:

Humans must keep doing what they have been doing, hating and fighting each other. I will sit in the background, and let them do their thing. And God knows that humans have enough blood and gore to satisfy my, and many more’s, curiosity. They won’t have to worry about fighting against me, because they have nothing to fear.

It's certainly more impressive than the bots that call me on the phone to try to sell me things. But the resulting work is what I would have told a student is "a bunch of stuff about the topic." 

There is less going on here than meets the eye. Here's where Manjoo walks right up to the point and misses it:

OpenAI’s new software, called GPT-3, is by far the most powerful “language model” ever created. A language model is an artificial intelligence system that has been trained on an enormous corpus of text; with enough text and enough processing, the machine begins to learn probabilistic connections between words. More plainly: GPT-3 can read and write. And not badly, either.

Except that his conclusion--that GPT-3 can read and write--is simply not so, and he's just explained why. What GPT-3 actually does is an impressive job of linguistic prediction. It has read, basically, the entire internet, and based on that, it can look at a string of words (like, say, the introduction of an essay) and predict what word likely comes next. Like every other computer in the world, it has no idea what it is saying, no ideas at all, no actual intelligence involved.

Manjoo himself eventually references some of the program's failings, referencing this piece from AIWeirdness where someone took the program out for a spin and found it easy to get it to spew sentences like, in response to the question "how many eyes does a horse have"-- 

4. It has two eyes on the outside and two eyes on the inside.

We can get a slightly more balanced look at GPT-3 from this article at MIT Technology Review, entitled "OpenAI’s new language generator GPT-3 is shockingly good—and completely mindless." Among other issues, studying language on the internet has led to a tendency toward racist and sexist spew (not a new issue-- remember Tay, the Microsoft chatbot that had to be shut down because it was so wildly offensive). Here's MIT's description of how GPT-3 works

Exactly what’s going on inside GPT-3 isn’t clear. But what it seems to be good at is synthesizing text it has found elsewhere on the internet, making it a kind of vast, eclectic scrapbook created from millions and millions of snippets of text that it then glues together in weird and wonderful ways on demand.

And Julian Togelius, an expert in the field, had this to offer via Twitter

We can now automate the production of passable text on basically any topic. What's hard is to produce text that doesn't fall apart when you look closely. But that's hard for humans as well.

And this:

GPT-3 often performs like a clever student who hasn't done their reading trying to bullshit their way through an exam. Some well-known facts, some half-truths, and some straight lies, strung together in what first looks like a smooth narrative.

So as always with tech, beware the hype, particularly from press that don't really grasp the technology they're being asked to "gee whiz" over. GPT-3 cannot read and write (it can apparently put together code made-to-order). Consider what Sam Altman, OpenAI's other co-founder, had to say to MIT:

The GPT-3 hype is way too much. It’s impressive (thanks for the nice compliments!) but it still has serious weaknesses and sometimes makes very silly mistakes. AI is going to change the world, but GPT-3 is just a very early glimpse. We have a lot still to figure out.

I tell you all of this, not just because this field interests me (which it does, because language is quite possibly the most taken-for-granted piece of magic in the universe), but for one other reason.

The next time some company is trying to convince you that it has software that can read and assess a piece of student writing, please remember that this company which has sunk mountains of money and towers of expertise into trying to create software that can do language even just a little--that company hasn't succeeded yet. And neither has the company that is trying to sell you robograding. Computers can't read or write yet, and they aren't particularly close to it. Anyone who tells you differently is trying to sell you some cyber-snake computer oil hatched in some realm of alternative facts.

DeVos Says We're All In This Together. Ha!

So this just popped up on my feed:

This #BackToSchool season may be different than any other, but we’re all in this together. To all the parents, students and educators rising to meet the challenge in new ways and places, you’ve got this! Let’s make it a great year! pic.twitter.com/1Qo9Cd2HW7

— Secretary Betsy DeVos (@BetsyDeVosED) September 8, 2020

So much to unpack.

First, who's this "we"? Because Betsy DeVos has made it clear that in her universe, the failed public "government" schools are not "in it" with her. She has not invited public school teachers, the unions, public school students--all the things that are part of what she derides as "the system"--to be on Team DeVos. Plus, note to Betsy--other parts of the country have been back to school for weeks.

And there's no question that the rest of us aren't in it together. The parents who can pay to send their kids to pod school at a literal country club are not "in it" with the families who have to send students to get on the internet in a Taco Bell parking lot. The parents who can afford to have someone stay home with the kids are not "in it" with the parents who have to scramble for child care or do without critical income. And as local school leaders look for guidance from the state or federal level, they mostly find that they are "in it" alone.

And man does it ring hollow to hear a weak attaboy of "you've got this" from a woman who mostly talks about how the people working in public schools don't got this, how they're all just doing a crappy job in a dead end system. Was she not just weeks ago threatening funding for public schools so that she could force those shiftless do-nothings to get in there and get to work? Has she not been trying to sell the idea (unsuccessfully) that private schools should get a bigger piece of the CARES relief pie than public school students?

So even if we get only this far, I think that's enough to merit a "Bite Me, Betsy" t-shirt.

But let's push on, because there's one more level to this.

The DeVos dream of a fully privatized, all voucher school system (well, maybe with a few public schools for the children of Those People) is that a voucher system is all about NOT being "in it" together.

A voucher system is about giving everyone a chunk of money and sending them scattering in all directions. It's about getting the government off the backs of these noble edu-preneurs so that they can do things their own way, even if that way involves discriminatory malpractice. In the Education Freedom tax credit scholarship version of vouchers that DeVos is still pushing for, it's about fixing things so that you (well, if you're wealthy enough) no longer have to pay taxes to finance an education for Those People's Children.

Vouchers are about turning education into a commodity, with each family navigating the market as best they can. It's about dumping families into a world where they may not be able to "buy" the "product" they want, where the school they pick may simply walk away from them.

Most of all, vouchers are another way to say to families, "We cut you a check. Now you're on your own. Not our problem. Good luck and have a nice life."

In short, the DeVos dream education set-up is the very opposite of being in it together. Somebody please tell whatever intern whipped up this thing that it is spectacularly tone deaf.

Report: Are Charter Schools A Big Risk For Families?

In a new report, the Network for Public Education shows how big a gamble it can be to enroll your child in a charter school. And the odds are not in parents’ favor.

“Broken Promises: An Analysis of Charter School Closures From 1999-2017” is a deep dive into the data surrounding patterns of charter closure and the number of students affected by those closures, especially those in high poverty areas. NPE is a advocacy group co-founded by Diane Ravitch, the Bush-era Assistant Secretary of Education who has since become an outspoken critic of education reform. The organization's executive director is Carol Burris, a former award-winning New York principal; Burris co-wrote the report with Ryan Pfleger, an education policy researcher.

The researchers worked primarily from the U.S. Department of Education’s Common Core of Data (CCD) and broke charter schools into 17 cohorts based on the year that they opened. The findings serve as a warning for parents considering the charter option.

Within the first three years, 18% of charters had closed, with many of those closures occurring within the first year. By the end of five years, 25% of charters had closed. By the ten year mark, 40% of charters had closed. Of the 17 cohorts, five had been around for fifteen years; within those, roughly half of all charter schools had closed (anywhere from 47% to 54%). Looked at side by side, the cohort results are fairly steady; the failure rates have not been increasing or decreasing over the years.

Charter advocates have often argued that charter churn is a feature, not a bug, simply a sign that market forces are working and that weaker schools are being sloughed off. But the NPE report notes that these closures represent at least 867,000 students who “found themselves emptying their lockers for the last time—sometimes in the middle of a school year—as their school shutters its door for good.”

The disruption to students and families by the cycle of closing schools is captured by one parent quoted in the report:

For the last three years I have had to place my kids at different schools each year because the schools keep closing. My child was attending MCPA, that school closed. He then went to Medard Nelson, that school closed. Now, he is at Coghill and y’all are trying to close that school. I am tired of moving my child every year because y’all are closing schools.

The destabilizing effects of charter churn are further exacerbated in the poorest cities. The report finds that in cities like Detroit, Tucson and Milwaukee, the rate of charter closure is highest in the areas where poverty is highest. Students and families that need stability from their schools are instead repeatedly subjected to a cycle of starting over with a new school, new teachers, new procedures, new rules. Research suggests that when students move from school to school, it negatively affects their chances of success.

The report also finds that the states with a large charter sector have the large rates of failure. At the five and ten year marks, Wisconsin, Arizona and Florida show the top failure rates, with Ohio close behind.

Beyond the human cost of these failures, there’s the high financial cost. A previous NPE report shows that the federal government has spent at least one billion taxpayer dollars on charters that closed quickly or never even opened at all.

Charter supporters may argue that this is all just the market working itself out, but that’s hardly a comfort to parents who must go through shopping, application, enrollment and adjustment to the new school yet again. As the report acknowledges, there are charter schools doing some excellent work out there, but for parents, enrolling a child in a charter school—particularly a new one—is a bit of a risk. It’s one thing to see market forces work in a sector such as restaurants, where new businesses come and go and very few go the distance; if you discover that your new favorite eatery has suddenly closed, it’s a minor inconvenience. It’s another things to see such instability in a sector that is supposed to provide stability and education for our youngest and most vulnerable citizens.

The report is available at the NPE website.

Originally posted at Forbes.com

Bulletins From The Trailing Edge Of The Pandemic

If it can work anywhere, it can work here.

I live in a county in NW PA, with relatively small population (50K or so). And our schools are all open.

We have been subject to the same rules as the rest of the state, and like pretty much everything in PA, the folks in charge have made their rules based on Pittsburgh, Philly and Harrisburg. This is Trump country, so plenty of folks are anti-maskers, but we haven't had any of those ugly assaults. I think we benefit from one aspect of small town life-- when you meet that minimum wage worker at the door of the business, you probably already know them. Makes it marginally harder to be a jerk to them.

But mostly we've done well. We have a big branch here of UPMC (the "non-profit" health monolith that is slowly eating the entire state). Since anyone started counting, our total number of positive Covid cases has not yet hit 70, and we've had only one death. We've had many sets of days in a row with no new positives. By the figuring of the state, we are a "green" county, which means restrictions are minimal, but most local businesses remain cautious. And this is the kind of area where it's not unusual to go, say, grocery shopping and encounter only a handful of other people.

So yes. Schools are open. We have four separate districts in the county (more than necessary, but that's a discussion for another day). All are open five out of five, full days. One has switched the high school to block scheduling. All require masks, and various bits of tweaking have been applied to traffic patterns in the buildings. There are barriers, cleaners, new arrangements for lunch and recess.

A non-zero number of families are staying home and selecting from an assortment of distance learning options, but it's nothing remotely close to a majority. There may have been a few early retirements, but nothing remotely approaching a "wave," nor are local unions contemplating sick outs or strikes. Parents are encouraged to drive their kids to school, but the buses are still running. Within districts, administrators who have done a lousy job of building trust and collaboration are suffering for it; those who have done a good job are benefiting from it. Schools may or may not have solid ideas about what to do if anything goes south. Nobody has made any special effort to recruit substitute teachers, and the pay is pretty lousy for the area, so that's going to be a problem sooner or later.

A return to 100% distance learning will be a real struggle here; there are so many places where there is little or no reliable internet coverage (my old high school sits in a 1/2 bar of 3G zone) and families within the district are spread out. Okay, maybe not so much "real struggle" as "certain failure."

But mostly, school is under way (last Tuesday was the first day) and mostly, students and teachers are adjusting. School sports are happening, with a spectator limit of 250 people. My wife is a teacher in a local elementary, and I have many friends from my pre-retirement years still working. I'm holding my breath. I know that other semi-isolated rural areas have experienced sudden spikes of covid, and I have my doubts about how well-prepared anyone here is to deal with an eruption.

So I have no actual point today. Think of this as a first chapter of whatever story is going to emerge through the year, depending on what happens in the pages we haven't turned over yet I'm praying that it's a boring story.

ICYMI: Labor Day Weekend Edition (9/6)

While you're enjoying your socially distant cookouts and celebrations this weekend, take a moment to thank the labor movement that made things like weekends possible. In the meantime, here's some reading from the week.

This Teacher Turned Remote Learning Into A Road Trip  

There are many cool parts to this story (including the part where her administration greenlights it, because administrators who say yes to things are a real treat). 3000 miles on the road for this Texas history teacher, remote teaching a la Carmen Sandiego.

Robot Teachers, Racist Algorithms, and Disaster Pedagogy  

Audrey Watters is doing guest spots in classes, and here's one of her most recent, touching on algorithms, the British grading scandal, racist AI, and other ed tech shenanigans. Always, always worth the read (and you should be subscribing).

Biden and Democrats Turn Away from Two Decades of Test-Based Accountability

Jan Resseger takes a long, thoughtful, and optimistic look at the evolution of the Democrats on the whole ed reform biz.

Borrowers Demand Answers About Blanket Denials of Loan Forgiveness   

Remember that federal loan forgiveness program where your college loans are supposed to be forgiven if you were the victim of fraud by a predatory for-profit college? Remember how DeVos simply refused to actually truly implement it, and then the court slammed her for it and told her to get on with it, or else? Here's an update, and it is going about as well as you'd expect.

The Myth of Charter Schools and Local Control   

Carl Peterson takes a close look at a candidate for the LAUSD board who has some thoughts about school management that don't quite match how she helps run a charter school.

Arizona Charter Schools Can Double Dip

Laurie Roberts at the Arizona Republic is pretty steamed about how Arizona charters cashed in on the PPP program, and the state is okay with that.

Pasco's Future Crimes Division  

Not directly tied to education, but more about the creeping surveillance state. From the Tampa Bay Times, a look at a sheriff's program for stopping crime before it happens. Exceptionally creepy and appalling.

Success Academy Delays In-Person Learning Till January  

I included a couple of these just because some folks are trying hard to push the narrative that teachers' unions are the force behind the closing of school buildings. And yet, it seems that some folks who aren't the Evil Union are also shifting to distance.

Cyberattacks persist ; K12 a Florida mess

Miami-Dade schools have a variety of problems. Come to this Miami Herald story for the district screw-ups (still no signed contract with K12) and stay for the reminder that increased online learning means increased exposure to hackers etc.

Idaho Considers Dropping Common Core   

Williamson Evers at the Independent Institute pens a pointed response to one of Mike Petrilli's ubiquitous Common Core cheerleading op-eds. Not in 100% agreement, but he does bring some heat.

Standardized Testing: Indispensable to Those Who Are Not Subjected To It

The indispensable Mercedes Schneider offers a good vivid portrait of the Big Standardized Test. I love a good extended simile.

Will The Pandemic Give DeVos Her $5 Billion Voucher Scheme

Betsy DeVos has been pitching "Education Freedom" as long as she's been in office. It's a tax credit scholarship scheme, which is to say, a voucher program that would blow a $5 billion hole in the federal budget, but would be a real treat for rich folks who A) like private schools better than public ones and B) would rather not pay taxes to the feds.

This frickin' guy

The Education Freedom pitch has landed with a thud every time. But more recently what has been new about it is that, somehow, DeVos got Senator Ted "Least Loved Man In The Senate" Cruz to pitch it. And right this moment, Cruz is doing what he does best-- being an absolute pain in everyone's ass--and he's doing it over DeVos's pet project.

Yesterday, CNN reported that the Senate's new stimulus bill (which has been a the focus of a spectacular display of GOP dysfunction for months) may be hung up over Cruz's insistence that the DeVos Voucher Bill be included in the stimulus package.

CNN's sources say that

Cruz has argued the coronavirus pandemic has showcased the weakness of public school systems and that Republicans should get behind an idea that the party is advocating this election year, including at last month's Republican National Convention.

In other words, "the public schools are weak and undefended right now--let's go in for the kill." And Cruz has a point of sorts in noting that school choice is half of the bullet point action plan of the Trump campaign (the GOP has no platform this year beyond "media bad, always do the opposite of what Dems want, and support Beloved Leader at all costs").

On the other hand, while some GOP plutocrats and thinky folks love vouchers, history says that the voters don't. Yes, there have been various polls run that purport to show broad support for choice, but they depend a great deal on friendly wording of the proposition. IOW, nobody has been out there asking, "Do you support the idea of reducing the funding for your local public school so that a handful of students can attend a private school, one that might reject your own child if you applied."

CNN notes that some GOP candidates are worried about trying to sell this program back home in an election year (particularly candidates like Susan Collins who already have enough trouble this time).

CNN being CNN, the piece also includes statements like this one:

School choice has been a top conservative cause for years but has gained fresh urgency during the pandemic.

Which is so much more polite and CNN-ier than "the folks who oppose government schools are salivating right now because they smell blood."

McConnell needs Cruz's vote, and this program seems to be the quid that Cruz is hanging his pro quo on. Even as his office issues senseless statements like this one:

"Sen. Cruz has been a leading advocate for school choice in the Senate," said Lauren Blair Aronson, a Cruz spokeswoman. "Throughout the pandemic, he has urged his colleagues to focus on solutions that will help get Americans safely back to work and our kids safely back into the classroom, including provisions of his Education Freedom Scholarship and Opportunity Act, which would give families the resources they need to ensure their children have access to a quality education in these uncertain times."

I've read this kind of statement a couple of times from DeVos backers, and I keep wondering-- do they serious mean to hint that somehow private schools are impervious to the coronavirus? Or this just to appeal to those who think it's all a fake and want a school that will just keep plowing on and ignore Covid-19?

However you cut it, this bill is bad news for public school (did I mention that it represents a $5 billion cut to federal funding) and would be the ultimate step forward in moving public tax dollars to private and religious schools. But the Senate GOP really wants to get something done so they can stop looking like Covid relief shmucks. This would be an excellent time to get a hold of your Senator's office and say, "No, thanks. But no."

A Seventh Grader Kicks Edgenuity's Dumb Robograding Butt

The story comes to us from Francesca Paris at NPR's Here and Now, and it can serve as our sixty-gazillionth reminder that computer algorithms-- even ones that are marketed as Artificial Intelligence-- cannot grade student work to save their cybernetic lives.

A student in LAUSD's virtual school was dismayed when his first history assignment came back an F. It was short written responses, and the score came back instantly, so his mother figured out that it had been graded by the software and not a human being. But we just saw here a short time ago, computer algorithms can't really read, and they don't understand content--which would seem to be a real drawback when scoring history assignments.

This particular virtual school product is from Edgenuity, one of the more widely used school-in-a-can computer products. Its CEO won recognition at the EdTech Awards last year, and the company is in something like 20,000 schools. But it gets plenty of criticism for being standardized to death. It doesn't appear to have a great deal of bench strength when it comes to questions; users talk about how easy it is to just google answers while taking assessments, and Slate discovered that students can do well by just repeatedly taking tests, which turn out to ask mostly the same questions every time the student tries again.

But the student in this story didn't have to work that hard. Some quick trial and error yielded perfect results.

The question? "How did the location of Constantinople help it grow wealthy and prosperous?

Their answer:

It was between the Aegean and the Black seas, which made it a hub for boats of traders and passengers. It was also right between Europe and Asia Minor, which made it a huge hub for trade, and it was on many trade routes of the time. Profit Diversity Spain Gaul China India Africa

Yes, that's a non-sentence string of words at the end. Like other assessment algorithms, Edgenuity's appears to be looking for a few key terms, maybe signs that there's more than one sentence. But it has no idea what it's "looking" at.

This story, besides providing one more example of how assessment algorithms fail at anything but the simplest tasks, is also a demonstration of the danger of these stupid things. In just twenty-four hours, this program taught this student to answer questions with little attention to coherence or content meaning. Just string together the words the computer wants.

Advocates of computer assessment often point at tales like this and argue, "But the writer wasn't making a good faith effort to answer the question." Well, no-- that's kind of the point. Since the scoring program didn't--and can't--make a good faith effort to read the work, that quickly teaches students that making a good faith effort to write an answer is not the task at hand, that such a task is for suckers, that it is, literally, pointless. Advocates will also point to studies that show robo-graders awarding scores that match scores from human graders; this is accomplished by giving the barely-trained human scorers instructions that require them to score the work with the same stunted dopiness that the algorithm uses.

Edgenuity offered a statement to Paris in which it offered the defense that the algorithms should not "supplant" teacher grading, but "only to provide scoring guidance to teachers." And teachers can override the robo-grader, but then, what's the point of using the robograder in the first place, particularly if its "scoring guidance" is junk?

Robograding for anything beyond simple objective questions continues to be junk. It provides the wrong analysis and teaches the wrong lessons, while training students to placate the algorithm rather than grasp the content. No doubt the shiny over-promising marketing makes this kind of thing appealing to the people in school systems who do the purchasing, but they are one more ed tech product that over-promises and under-delivers. They are junk, but they're profitable junk. They're continued presence in schools is infuriating.

Go read Paris's entire piece. The look up some pictures of puppies or something more soothing.

[For more from this blog on the business of robograding, see here, here, here, here, or even here.]