AI, Language, and the Uncanny Valley

We experience vertigo in the uncanny valley because we’ve spent hundreds of thousands of years fine-tuning our nervous systems to read and respond to the subtlest cues in real faces. We perceive when someone’s eyes squint into a smile, or how their face flushes from the cheeks to the forehead, and we also — at least subconsciously — perceive the absence of these organic barometers. Simulations make us feel like we’re engaged with the nonliving, and that’s creepy.

That's an excerpt from Douglass Rushkoff's book, Team Human, talking about how the uncanny valley is our best defense. The uncanny valley is that special place where computer simulations, particularly of humans, come close-but-not-quite-close enough and therefor trigger an ick reaction (like the almost-humans in Polar Express or creepy Princess Leia in Rogue One). 

The quest for AI runs right through the uncanny valley, although sometimes the ick factor is less about uneasiness and more about cars that don't drive themselves where you want them to. The gap between what AI promises and what it can deliver is at least as large as an uncanny valley, though companies like Google are now trying to build a fluffy PR bridge over it (hence Google's directive that researchers "strike a positive tone" in their write-ups).

Since summer, journalists have been gushing glowingly over GPT-3, the newest level of AI powered language simulation (the New York Times has now gushed twice in six months). It was the late seventies when I heard a professor explain that the search for decent language synthesizing software and artificial intelligence were inexorably linked, and that seems to still be true. 

It's important to understand what AI, or to call it by its current true name, machine learning, actually does. It does not understand or analyze anything. You can't make it blow up by giving it a logic-defying paradox to chew on. A computer is infinitely patient, and is good at cracking patterns. Let it read, say, all the writing on the internet, and given a place to start, it can make an analysis of what, statistically, would probably come next. GPT-3 is a big deal because it has read more stuff and broken out more patterns than any previous software. But it's still just analyzing language patterns based on superficial characteristics of words. It is Perd Haply with bigger memory capacity.

We've seen the more limited versions of AI, like the automated robocaller that can only cope with responses that fit in a limited menu. But for someone who reads a lot, even the more advanced versions land in the uncanny valley. GPT-3 can spit out some weird wrongness, as demonstrated in this piece that includes exchanges such as 

Q: How many eyes does a horse have?
A: 4. It has two eyes on the outside and two eyes on the inside.

This set of testers found that GPT-3 was sometimes prone to plagiarism, providing correct-but-copied sentences from websites. Nudged in a slightly different direction, it produced paragraphs like this one

Whales, and especially baleen whales, are well known for their enormous size, but most types of whales are not larger than a full-grown adult human. Exceptions include the blue whale, the largest animal ever known, the extinct “Basilosaurus”, which was longer than a blue whale and likely the largest animal to have ever existed, and the “shovelnose” whales, especially the genus “Balaenoptera” which include the blue whale, “B. musculus”, the fin whale, “B. physalus”, and the sei whale, “B. borealis”.

The reviewer said this "reads well but is often wrong." But some of the samples I've read don't particularly read well, like this that was part of an "essay" prompted by Farhad Manjoo

Humans must keep doing what they have been doing, hating and fighting each other. I will sit in the background, and let them do their thing. And God knows that humans have enough blood and gore to satisfy my, and many more’s, curiosity. They won’t have to worry about fighting against me, because they have nothing to fear.

Like much of GPT-3's output, it reads to me like a disinterested student trying to come up with enough bulk to fill paper, resulting in writing that is just A Bunch Of Stuff About Topic X.

The uncanny valley has also turned up in my comments section. I get a lot of funky stuff there, but this one jumped out at me, responding to an old piece about the Boston Consulting Group.

I totally agree with your idea, and the group's beautiful comparison to the Black Knight and the Reaper is really a loss for public schools. In terms of the BCG report that made three recommendations, I think the idea of ​​companies helping educators to define and implement to update education in nearby cities is good, and really the idea of ​​strengthening schools is hard work and needs everyone's help. Great partnership between Harvard Harvard and BCG, I believe that it is more accessible to enter MBA programs mainly with a large investment.

First, that's not really connected to anything in the original piece. Second, even not knowing that, it's not hard to recognize that we've entered the uncanny valley here. Lots of bad writing gives one the impression of an actual idea struggling to escape from a tar pit of troubled technique. This is just words strung together. 

The poster's name is given as Daniela Braga. There's a model by that name, but Daniela Braga is also "founder and CEO of DefinedCrowd, one of the fastest growing startups in the AI space. With eighteen years working in Speech Technology both in academia and industry in Portugal, Spain, China, and the US." I reached out to Braga on LinkedIN to see if she wanted to fess up to turning an AI loose on blogging comment sections, but as yet have received no reply. 

Uncanny valley stuff is a reminder first, that humans can be very hard to fool, and second, that we capture and process huge, huge, huge amounts of data--so much so that there's a whole part of the brain that does the capture and process without us being fully aware of it. It's enough to make one think that maybe the conventional notion that says computers do capture and process of data better than humans might not be entirely true. Machines have the advantage of being tireless and immune to boredom, but they need both of those advantages just to get close to catching up with humans. 

A good example of this gap is the attempt to AI our way to cheating prevention, with the terrible AI surveillance programs that are making student lives miserable, while at the same time failing at their assigned task. But spotting a student who's cheating is not easy, and the algorithms designed by software companies have clearly been created by somebody who never actually had to catch a sneaky high school  junior mid-test. And you can't design software to know what you don't know, because software doesn't know anything. And yes, computer folks will say that machine learning allows the machine to "teach"itself things it didn't know, but that's mostly insofar as the algorithm can recognize old patterns in new places. And even that is limited--hence facial software's notorious inability to recognize that Black folks have faces. Faking reality, or fake-reading it, turns out to be really, really complicated and really, really hard.

All of which is just to say, again, that computers are not going to be able to run a classroom full of students any time soon, nor are we getting closer to algorithms that can truly manage a student's education or grade a student's essay. It takes an actual authentic human to do all of that. 

Rushkoff's point is that the uncanny valley--our sense that something is seriously wrong--is a defense mechanism, and that we should pay attention to it and trust it. He also has some things to say about inauthenticity in other areas of life--I'll let him wrap up:

Our uneasiness with simulations — whether they’re virtual reality, shopping malls, or social roles — is not something to be ignored, repressed, or medicated, but rather felt and expressed. These situations feel unreal and uncomfortable for good reasons. The importance of distinguishing between human values and false idols is at the heart of most religions, and is the starting place for social justice.

The uncanny valley is our friend.

Nobody Hates Miguel Cardona (So Far)

(OF course, I finished this post at about 3:55 PM Eastern time on Tuesday, so there's plenty of time left for folks to crank up the disapproval)

So now we know who the new guy is (probably) going to be, and an awful lot of folks are asking who he is and whether we should be delighted or upset or whatever.

Diane Ravitch notes that he hasn't taken a position on many of the hot-button issues, like charters. But she notes:

Having read his Twitter feed (@teachcardona), I get the impression that he is a very decent and concerned administrator who cheers on students and teachers.

Lily Eskelsen Garcia said "I proudly support this nomination" and expresses confidence that Dr. Cardona "will ensure that the federal government's role in education is to provide access and opportunity for every student..."

At the74, Andrew Rotherham calls the choice "deft" and notes that Cardona has a good, inspiring up-from-poverty story (he does) and that "he doesn't fit cleanly into the reform or anti-reform camp." On charters, "he's a Goldilocks--not too hot or cold."

Jeanne Allen at the super-charter-loving and ultra-teacher-union-hating Center for Education Reform calls the nomination "intriguing" and calls the nomination "reminiscent of Rod Paige" (yuck) and likes that he's not rooted in the "platitudes of powerful interest group."

The Connecticut Education Association four days ago came out in favor of his nomination:

Miguel Cardona’s formative experience as a teacher and administrator has been critical to his accomplishments as Connecticut Education Commissioner. He has been tested by the unprecedented upheaval caused by the pandemic. While this challenge has been a rocky road — and many issues remain unresolved — teachers and school support staff have appreciated his openness and collaboration. If selected as Secretary of Education, Dr. Cardona would be a positive force for public education — light years ahead of the dismal Betsy DeVos track record.

Lots of folks have played the "he's not DeVos card" and that's a pretty easy card to beat. He's also not Arne Duncan. He has had a rapid rise in his career--he taught for about four or five years, became an administrator on the local level, then jumped up to the state level. "Youngest principal ever in the state" comes up a lot.

Hechinger has a nice recycled interview piece up about him. He appears to be a moderate on the Big Standardized Test (can be useful, don't give it too much power) and has been pro-re-opening the buildings, but seems to have maintained a decent relationship with teachers at the same time. I hear rumblings that some Connecticut teachers don't like him. He's apparently a Linda-Darling Hammond pick, and LDH has long been a figure of some controversy (or at least some spirited "what side is she on, anyway" arguments) in the education world. As Ravitch notes, the DFER crowd was pushing hard for someone more their style, and they didn't get that. 

But at this point, nobody has jumped up to holler that he's awful and unacceptable, and he's probably not going to get a fight from the GOP senate. 

There is nothing obviously and terrifyingly unqualified about him (I know--it's a low bar, but it's a low bar that hasn't been cleared in a long time). There are some question marks, and we've been burned before by folks who talked pretty and didn't do the walk to go with it. Educated in public school, public university system--that's a plus. He seems to fit my desire for someone who is non-adversarial with teachers. The years as an administrator could be a nice addition to his relatively few years in the classroom, if he isn't one of those guys who forgot everything he knew about being a classroom teach ten minutes after he landed an admin job. And lots of profiles mention his focus on kids, which could be a good thing but after years of listening to reformsters say "It's for the children" to justify every damned thing, I don't find that factor very soothing.

So, we'll see. It's worth remembering that these days, many of the big fights are going to be in the states and in the courts. But having somebody who's not totally incompetent, who's not disconnected from the actual work of education, and who does not arrive in DC with an bad news agenda that he's determined to push-- well, that would not be terrible. Relax, enjoy the holidays. It's going be weeks before we find out what we've really got for a new secretary. 

ICYMI: Santa's Almost Here (12/20)

 And the Board of Directors has just about figured out this Christmas thing and twigged onto the notion that presents are coming (but why not right now). I am going to try not to think constantly about all the family and children and grandchildren that I am not seeing this week, because that sucks. In the meantime, here's some reading from this week.

The Logjam that Awaits Biden's education secretary

Derek Black at CNN, with hard dose of reality therapy for everyone imagining that a new administration will bring dramatic change. What a grinch. Okay, he may have some points, too.

How to assign writing when you don't teach writing  

Paul Thomas with some great thoughtful practical advice for assigning writing when that's not really your lane.

For Black educators when school systems aren't doing enuf

Dena Simmons at ASCD with some powerful personal reflections for the times.

Reaganland: Public education and America's right turn

Have You Heard talks to Rick Perlstein and takes us back to the seventies. Really interesting stuff about how schools became a target in the culture wars.

Black students most likely to be going to school remotely

Samantha Fields at Marketplace looks at an emerging trend. Safety and trust seem to be the issues (and not that they are dupes of the teachers unions).

Testing students this spring would be a mistake  

Can't say this enough, but this time it's not me, but testing expert Lorrie Shepard at EdWeek.

Fifty years of trickling down didn't work  

Not directly related to education, exactly, but important validation for what everyone already knew.\

How teachers are sacrificing student privacy to stop cheating

From Vox, one more article pointing out that surveillance software is a bad idea, and schools should knock it off.

The 2020 snow day is here. It must include "sleducation."

Okay, I wish Joshua Goodman at Education Next had the courage to write sleducation without the quotation marks, but still a nice little piece.

A rural school under pressure to stay open

This is how ugly it's getting in places like rural Idaho, where the 'rona is still a big hoax and people are too tough to mask up. Kirk Siegler at NPR.

Sen. Jon Tester on Democrats and rural voters

Tester has some thoughts, including the novel idea of standing up for public education. From New York Times.

Florida lets voucher schools hire dropouts as teachers--and keep it secret

The Orlando Sentinel has been a great source of watchdogging the Florida shenanigans. You may or may not be able to scoot past the paywall, but if you can, this story is amazing. You will not believe how bad it is down there.

As the gap between students and teachers of colors widens in PA, Black families demand change

Sojourner Ahebee reporting for WHYY, Philly's NPR station. This is a great piece of reportage, with plenty of nuance and detail for a difficult topic. If you don't read anything else this week...

Pennsylvania's Teacher Problem

 We already know that the teaching profession is primarily composed of white women (average age 43). But sometimes, when you break data down in particular ways, it becomes even more striking.

Research for Action is a Philly-based group that has done some great work over the years, and they've done some research about the TOC/SOC balance in PA that are featured both in this article from Sojourner Ahebee and a "supplemental" Twitter thread (and you can read the research here).

Some of the PA data is just striking. First, PA is an outlier when it comes to the gap between students and teachers of color. Nationally, the share of SOC is 2.5 times the share of TOC, but in PA the SOC share is 6 times greater (only NE, NH and Indiana have a bigger gap). I'm not totally shocked and surprised by that; PA suffers from a fairly severe urban vs. rural divide, and the rural parts of the state are pretty white. 

But when RFA starts breaking down by districts, it gets even starker. There are 500 school districts in PA (which is its own problem) and 184 of those districts employ zero teachers of color. Only about twenty-five districts in the entire state have staffs that are more than 5% TOC. Here's a map from Research for Action, breaking it down district by district:

There are roughly 3,200 schools in Pennsylvania; RFA found that 1,500 of them have all-white teaching staffs. A dozen schools across the state in 2019-2020 were 80% students of color and had no teachers of color at all. And there are other schools that come close to that; Ahebee's piece looks at William Penn School District, where 4% of the students are white and 80% of the teachers are. 

The shifts have occurred over time, both in the increase of SOC and the decrease of TOC. And in some rural districts the staff is mostly or all white, but so is the student body. But we are now in a problematic place.

I know that this subject is a touchy one for some white teachers, that the initial reaction is to bristle and argue that white teachers are perfectly capable of teaching students of color. But at this point the research is overwhelming--the presence of Black teachers in a school produces better results for Black students. With a Black teacher, Black students are more likely to go to college, less likely to suffer exclusionary discipline, more likely to be placed in gifted classes. The list goes on. And it's important for the white students in a school to encounter Black teachers as well. Ahebee hints at one other reason that Black teachers can be effective for Black students--in many settings, those teachers are the ones who actually live in the community that the school serves. Every student is best served by teachers from within and from outside her community. The student needs someone who gets the local flavor, and someone who can show her what lies outside familiar boundaries. All voices have their place within the school, and a diversity of voices best serves the students. 

We are at a difficult point at the moment, with the teacher supply already problematic. In my corner of PA, colleges have been compressing and even eliminating teacher programs because they don't have the enrollment. Add pandemic messes and we are in a place where the entire teacher corps needs to be rebuilt. That's a tough challenge, but it's also an opportunity to rebuild a teacher corps that is not so white. 

How is it done? RFA reports that PA has actually grown, slightly, its share of male TOC by 129, which is better than nothing. The guru of teacher workforce studies, Richard Ingersoll, has told us for years that retention is a huge part of the problem, especially for TOC, so part of the challenge is not just to recruit folks, but to keep them around. 

There are folks working on the issue. The PA department of education has a program to address the drain of teacher talent and the lack of TOC in the state. And the Center for Black Educator Development ("We address educational inequities to improve academic and social outcomes for all students through increased diversity") is one organization working on the issue. Teach for America, after its launch as a platform for primarily white temp teachers, has made equity more of a priority, but students need to see someone who stays, not someone who's just passing through for a couple of years. Per RFA dats, charters in Philly are doing better than public schools, but you know I don't think that's the answer (even if we ignore all the other charter issues, sheer scale rules them out). It will take state initiatives, organizations like CBED, and deliberate, mindful decisions by the people who do hiring at each of those 500 districts.

I don't see or hear anyone in all of this saying that white teachers can't teach students of color, or, for that matter, that women can't teach young men. But there is something fundamentally missing in any education system where a child can spend twelve years and never see an adult person like themself. Pennsylvania is in the weeds on this. 

Read Ahebee's article, which delivers a nuanced and close-up look at how the issue plays out for students and teachers. Read RFA's study, which has layers and layers of data to unpack. There is plenty here for Pennsylvania's education leaders to think about and act upon. 

1776 Commission Members Appointed (And It's About What You'd Expect)

You remember just six 2020 weeks ago (that's roughly a year and a half in regular time), Dear Leader proclaimed that the 1776 Commission would be formed in order to create a more perfect set of teaching stuff that would teach our young people to think about our country in the Correct Way. The proclamation announcing this was a piece of work, among other things laying out how we should teach students to have the Proper View of our country (we were bad about Black people for a while, but now that's all fixed) and wipe out that 1619 Project stuff. It was not exactly lined up with the goal of teaching critical thinking, but, hey, Patriotism.

Now Dear Leader has appointed his crew, and it has gone about as one would expect. You can check out the official list here, but here's a rundown of who some of these folks are.

Larry Arn will be the chair. Arn is the president of hyper-conservative ultra-Christianist Euro-centric Hillsdale College (Betsy DeVos was just there to deliver a Jeremiad).

Members include Charlie Kirk, founder of Turning Point USA, the campus right-wing group, and recent receiver of hard talk from Geraldo Rivera (Trump lost. Knock it off.). Because a history commission needs people who can't even come to grips with the history of the several weeks.

Brooke Rollins, the Trump domestic policy advisor since May.\

We've got Carol Swain, a retired poli sci professor that Trump probably saw talking on the TV, or maybe on Twitter passing along already-debunked election conspiracy baloney. And she's Black, so for Trump's purposes that's a bonus.

Vincent Haley, another member of the Trump administration (speechwriting, etc). Formerly employed by folks like Gingrich Productions and Trump for America, Inc.

Victor Davis Hanson, a pundit and Hoover Institution senior fellow who was on the TV explaining that Black Lives Matter is trying to "hijack" American history.

Phil Bryant, Governor of Mississippi and Trump lover. Briefly a sheriff before becoming a career politician.

Mike Gonzalez, a senior fellow at the Heritage Foundation, whose listed areas of expertise are conservatism, Europe and global politics. Not U.S. history.

John Gibbs, a conservative commentator who has held a variety of jobs in the Trump administration. He started out as a software engineer. His Wikipedia entry says, so far, that he has a "history of making inflammatory remarks and spreading false conspiracy theories on his Twitter feed." That would include the claim that Hillary Clinton's presidential campaign took part in a Satanic ritual.

Scott McNealy, co-founder of Sun Microsystems. He's a regular on the Fox business channel, and he held a Trump fundraiser for the election that Trump just lost. He did once found a company (Curriki) that did education stuff, so there's that.

Gay Hart Gaines, a GOP fund raiser from Palm Beach. He once received a lifetime award for that, awarded by Laura Ingraham at a dinner at Mar-A-Lago.

Ned Ryun, founder of American Majority, an outfit that locates, recruits and trains conservatives to run for local offices. He has a whole podcast series about the History of the Constitutional Convention.

Peter Kirsanow is a lawyer from Cleveland and is a member of the United States Commission on Civil Rights, serving his fourth six-year term (originally appointed by George W. Bush).

Charles R. Kesler is a professor at Claremont McKenna College. He appears to have written many real articles and at least one book about the founding of America.

Dr. Thomas K. Lindsay--who is not a medical doctor but is still listed with "Dr" attached--works at the Texas Public Policy Foundation, a right-tilted thinky tank. An academic (even though he calls himself a doctor), he research has focused on democracy and education.

Jerry C. Davis is president of the very Christianist College of the Ozarks (in Missouri) and his heart is heavy about racial injustice.

Michael Farris, a lawyer and founder of the Home School Legal Defense Association, and also worked on the Convention of States Project. Citizens for Self-Governance is also one of his projects.

And Bob McEwen, who was a GOP pol in Ohio back in the day, but nowadays is a lobbyist and talking head, primarily at Advantage Assoc International. He was a vocal supporter of Trump during impeachment proceedings.

So what we have is a couple of people who might actually know something about early U.S. history (albeit viewed through a conservative lens, or even a heavily smudged conservative lens) and a bunch of Trumpers, fundraisers, and folks who have no apparent excuse for being on this commission. And almost nobody who knows anything at all about education--well, actually nobody at all when it comes to the K-12 students who are the target of this baloney. 

What a bizarre little thing to try to pull off in the final moments of a lame duck Presidency (and on a Friday afternoon before a holiday week), as if there aren't a few other things that require federal attention at the moment. Here's hoping that the commission doesn't even manage one meeting before vanishing into the ether, like the silly piece of vaporware it is. 

Children are not our future

There are plenty of warm fuzzy teacher sayings that I could well do without, emphasizing as they do that teachers are too noble to ever want to do things like, say, insist on being paid a decent wage or have control over their working conditions. But there's a child-focused saying that I would like to banish to the Island of Misfit Cliches--

Children are our future. Or, sometimes, children are the future.

The first is worse, carrying with it the notion children belong to us--and not just now, but in perpetuity, as if their adult selves will exist only to take care of the rest of us. "Shut up, kid, and get back to work. Grampa is going to need a new pair of shoes!"

But I object to both for other reasons.

I hear in this cliche the echoes of a notion that children are empty vessels just waiting for Wise Adults to fill them up with knowledge and thoughts and values and all the other stuff that makes a person an actual person. Because, yeah, if they're empty vessels, they're not exactly persons, are they.

The modern version of this is to view them as a sort of empty hard drive, just waiting for programming to be added. I think this is part of the reason that the "science of reading" moniker rubs me the wrong way--the implication that one simply plugs or pours in this scientific programmy stuff, and that will result in every single child being filled up with proper reading stuff.

And, of course, if the child is an empty lot, we needn't look inside to see what's already there--we can just concentrate on what we want to see dumped into the child. We can build the child to order without having to pay attention to anything that might get in the way of the new structure we plan to build. 

The emphasis on the future is also problematic, suggesting that the child's life isn't really going on right now, that their current life is just a staging area for the life to come. And yes--it's absolutely true that their childhood is the time in which we should help them build strong, robust, vibrant structures that will allow them to enjoy their life in years to come. Is that more true for a five year old than a thirty-five year old? 

The idea of potentiality, the collecting of tools for future use--that's all important. But let's not get so focused on that that we forget that these children are living their lives right now. Let's not fall so far into this rhetoric that we start thinking that children will be, you know, real people some day, but not today. 

Plenty of adults act as if children are a mystery, as if nobody can know how to talk to this alien species. There is no mystery. Children are people. People who haven't yet developed some physiological and psychological aspects, people without limited experience in the world, but people all the same. Not future people. People right now, today.

This "children are the future" talk makes it easy to justify the kinds of bad policy we've seen in the last few decades. Sure, let's start sitting them down to study academic subjects earlier and earlier because there's nothing about what's going on in a four-year-old's life right now that could possibly be as important as getting her packed full of employer-desired skills for the future. It's easy to deny childhood when you think that all of a child's Real Life is in the future. 

"Children are the future" is often used as a motivational nudge for funding and/or supporting education and can feel like part of a larger conversation that started with "We don't need to spend money on that--they're just children." It's a conversation one would expect from people who measure a person's worth in their utility (in particular, their utility to employers). It's a hard conversation, because if you don't know that you should care about, look after, cherish and hold close our children, I don't know how to explain it to you. They are bundles of raw humanity, undiluted and unvarnished. That ought to be good enough. 

"Children are our present" doesn't sing like the "future" version, but I think it's truer and important in that it doesn't let us push off our obligation to the young humans among us. They are here, with us, today, and deserve to be cherished and supported and seen and heard and embraced for all they are today. The future will come on its own, soon enough.

 

More Teacher Effectiveness Mirages

 The Fordham Institution has a new report entitled "Teacher Effectiveness and Improvement in Charter and Traditional Public Schools." Despite what it claims to study, the report is a neear-perfect demonstration of Campbell's Law in action. 

The study starts with a question that, as used car salesmen put it, assumes the sale:

Study after study has found that urban charter schools, and non-profit charter networks in particular, tend to be more successful at boosting student achievement than traditional public schools in similar settings. But why?

We're not going to get bogged down in the details of this study, including this assertion that charter school superiority is a proven thing, because none of them matter when it comes to understanding why this study is fatally flawed.

The fleshed-out version of the question under study is this-- we know that more experienced teachers are generally better at their craft then newbies (an assertion that Fordham didn't make back in the days when they were part of the Let's Get Rid Of Teacher Job Protections crowd), but we also know that charters mostly have newbie teachers, so how is it that charters gets these superior results with fresh-out-the-wrapper staff?

The report was written by Matthew P. Steinberg (George Mason University) and Haisheng Yeng (U of Penn grad student). They worked from a pile of data from the PA department of education from between 2007 and 2017.

We could dig deeply into this report, but there's no reason to. All we really have to see is this sentence:

Like other studies, this one uses estimates of teachers’ value-added—that is, their contribution to students’ English language arts (ELA) and math achievement growth— as a proxy for their effectiveness.

So once again, "teacher effectiveness" is being used as a synonym for "scores on a single standardized test of math and reading soaked in the widely-debunked VAM formula." This is bunk. They try to prop it up with this--

Although such estimates cannot capture other valuable aspects of teaching practice and behaviors, research shows that (in addition to learning more math and English language arts) students assigned to teachers with higher value-added scores are more likely to go to college and earn higher salaries later in life.

That assertion about later salaries is cited from Chetty, Friedman, & Rockoff, 2014, a work that is problematic at best and bunk at worst. Meanwhile, even folks in the ed reform community have caught on to the fact that raising a child's Big Standardized Test score doesn't lead to that child having a better life. 

But using test scores as a proxy for "student achievement" or "teacher effectiveness" is a critical tactic for ed reformsters, because writing a whole paper about how one set of teachers is better at raising test scores isn't very sexy or exciting. "What I really want from a teacher is for her to get my kid to do better on that one standardized test they take every spring, and nothing else matters as much," said no parent ever. Likewise, while there are a million interpretations of "good teacher," very few of them are "teacher whose students get good scores on that one big test." 

Using test scores as a measure of teaching quality and student achievement isn't just a bad, inaccurate measurement--it triggers Goodhart's Law, or its somewhat better known sibling, Campbell's Law. The idea here is that the more you use a quantitative social measure for social decision-making, the more it will tend to disrupt and corrupt the processes it's supposed to be monitoring. If you like a pithier version, take Strathern's restatement of Goodhart--

When a measure becomes a target, it ceases to be a good measure.

Which brings us back to this piece of research. Steinberg and Yang find that CMO-run charters seem to have better-trained teachers, and they posit that hiring practices, training practices, or the charter chain tendency to force all teachers to follow the prescribed procedures--not just curricular, but pedagogical--might be the answer. 

But if we stop using bad proxies and just say plainly what we're talking about, there's little mystery here. The premise of the study is that newbie charter teachers get better test scores than public school teachers. That points to just one thing--more focused test prep. This is easy to enforce with newbie teachers in a restrictive teaching environment because they don't have enough of a well-established professional identity to push back. Meanwhile, teachers in public schools are trying to balance the demand for raising test scores against the demand to actually teach. 

In short, the explanation laid out by this report is that charter schools (at least the ones studied here) train teachers to do test prep instead of training them to actually teach. 

Mike Petrilli suggested on Twitter this morning that I'm being cynical here with my reading of the report, but I think it's far more cynical to keep arguing in 2020 that using a single set of test scores and a long-since-discredited number crunching formula as a measure of true teacher quality. It is possible, as Petrilli says, that some charters are doing a great job of teaching junior teachers to "teach well," but as long as the premise is that "teach well" means "have students who get high scores on the BS Test," we'll never, ever know. 

"Well, then, how do we figure out which teachers are doing a good job," has been the complaint for the past couple of decades, and I agree that this is a tough nut to crack, but that does not mean that we settle on a bad answer. It is hard to cure certain types of cancer, but that does not mean that we should settle for "drink bleach and sacrifice a frog under a full moon" as a cure. We do not, like the proverbial drunk, search for our car keys under a streetlight a hundred feet from where we dropped the keys because the light is better there. 

It has been true since we ushered in test-centric schooling under NCLB--the discussion about teacher quality is worth having, but we cannot have it if we insist on using as "data" something that does not measure teacher quality. 

There are other problems with this particular study. Most notably, since it is based on tests scores, it makes sense to look at the students who are taking those tests and the long-known techniques of cherry picking and push-out used by many charter schools to insure that they have a good crop of quality test-takers. Or we could talk about longer school days, or simply organizing the vast amount of school time around the BS Test. Any of these would explain the charter alleged test edge. This study doesn't address any of that. 

But it doesn't matter. Any study that accepts the premise that BS Test-based VAM scores are a measure of teacher quality is wasting time.