Raj Chetty for Dummies

The name Raj Chetty has been coming up a great deal lately, like a bad burrito that resists easy digestion. A great deal has been written about Chetty and his scholarly work, much of it by other scholars in various states of apoplexy.My goal today is not to contribute to that scholarly literature, but to try to translate the mass of writing by various erudite economists, scholars and statisticians into something shorter and simpler than ordinary civilians can understand.

In other words, I'm going to try to come up with a plain answer for the question, "Who is Raj Chetty, what does he say, how much of it is baloney, and why does anybody care?"

Who is Raj Chetty?

Chetty immigrated to the US from New Delhi at age nine. By age 23 he was an associate professor of economics at UC Berkeley, receiving tenure at at age 27. At 30, he returned to his alma mater and became the Bloomberg Professor of Economics at Harvard.

He has since become a bit of a celebrity economist, consulted and quoted by the President and members of Congress. He has won the John Bates Clark Medal; Fortune put him on their list of influential people under forty in business.

Chetty started attracting attention late in 2010 with the pre-announcement of a publication of research that would give a serious shot in the arm to the Valued Added Measurement movement in teacher evaluation. His work has also made special appearances in the State of the Union address and the Vergara trial.

What does Chetty say?

The sexy headline version of Chetty is that a child who has a great kindergarten teacher will make more money as an adult.

The unsexy version isn't much more complicated than that. What Chetty et al (he has a pair of co-authors on the study) say is that a high-VAM teacher can raise tests scores in younger students (say, K-4) and while that effect will disappear around 8th grade, eventually those VAM-exposed children will start making bigger bucks as adults.

Implications? Well, as one of Chetty's co-authors told the New York Times--

“The message is to fire people sooner rather than later,” Professor Friedman said.

Chetty's work has been used to buttress the folks who believe in firing our way to excellence-- just keep collecting VAM scores and ditching the bottom 5% of your staff. Chetty also plays well in court cases like Vergara, where it can be used to create the appearance of concrete damage to students (if Chris has Mrs. McUnvammy for first grade, Chris will be condemned to poverty in adulthood, ergo the state has an obligation to fire Mrs. McUnvammy toot suite). You can read one of the full versions of the paper here.

Who disagrees with Chetty?

Not everybody. In particular, economist Eric Hanushek has tried to join this little cottage industry, and lots of reformy poicy makers love to quote his study.

But the list of Chetty naysayers is certainly not short. Chetty appears to evoke a rather personal reaction from some folks, who characterize him as everything from a self-important twit to a clueless scientist who doesn't understand that he's building bombs that blow up real humans. I've never met the man, and nothing in his writing suggests a particular personality to me. So let's just focus on his work.

Moshe Adler at Columbia University wrote a research response to Chetty's paper for NEPC. This provoked a response from Chetty et al, which provoked yet another response from Adler. You can read the whole conversation here, but I'll warn you right now that you're not going to just scan it over lunch.

Meanwhile, you'll recall that the American Statistical Association came out pretty strongly opposed to VAM, which also put them in the position of being critical-- directly and indirectly-- of Chetty. Chetty et al took it upon themselves to deliver the ASA a lesson in statistical analyses ("I will keep my mouth shut because these people are authorities in areas outside my expertise," is apparently really hard for economists to say) which led to a conversation recounted here.

What do the scholarly and expert critics say?

To begin with, the study has a somewhat checkered publication history, debuting as news blurbs in 2010 and making its way up to publication in a non-peer-reviewed journal, then to republishing as two articles, then in a peer-reviewed journal. That history, along with many criticisms of the study, can be found here at Vamboozled, the blog of Audrey Amrein-Beardsley (the blog is a wealth of resources about all things Vam).

Many criticize Chetty's methodology. Adler's critique suggests that Chetty may have fudged some numbers, dis-included some data, and ignored previous research that didn't fit his framework. Amrein-Beardsley (and others) accuse Chetty of ignoring context of the data. Many critics suggest that Chetty is trying to make a mountain out of a molehill. 

You can chase scholarly links all day long, though the NEPC link to Moshe's work and simply typing "Chetty" into the VAMboozled search box will provide more than enough reading for an afternoon. Or two.

So how much of Chetty's work is bunk?

I'm going to go with "most of it."

Chetty's idea was to link VAM measures to later success-- to be able to say, "Look! High-VAM teachers grow successful students." There are several problems with this.

First, studies of VAM-based teacher effectiveness always seem to descend into the same tautology. Use test scores to measure VAM. Use VAM to id the best teachers. Check to see if VAM-certified teachers raise test scores. Strip out the fancy language and funky math and you're left with a fairly simple tautology-- "Teachers who get students to have high test scores tend to get students to have high test scores." This is no more insightful or useful than research to show that bald men tend to be bald.

Second, Chetty doesn't seem to distinguish between correlation and causation. His results seem to scream for that consideration-- six year olds who do better on tests don't grow into twelve year olds who do better tests, but they do grow into twenty-eight olds who make more money. I'm no economist, but to me, the yawning gulf between the alleged cause and the supposed effect leaves enough room for a truckload of other possible causes. This holds together just about as well as "because I buried a toad under a full moon a year ago, I met my true love today."

And as it turns out, an explanation is readily available. We know who does better on standardized tests-- the children of high income families. We know who's more likely to get better-paying jobs as adults-- the children of high income families. It seems highly probable that the conclusion to be drawn from Chetty's research is, "Children of higher-income families do better on tests and get higher-paying jobs." 

Chetty himself tried to plug that last hole, with research about economic mobility that concluded that it's not any worse than it was a decade ago-- but it's still pretty lousy. Chetty et al also insist that the students were distributed across the classrooms in completely random fashion. This strikes many as an assumption without foundation.

Put another way-- a mediocre teacher with a classroom full of rich kids who test well would earn a high VAM and those well-heeled students would still go on to have well-paying jobs, and nothing in Chetty's model would ever reveal that Mr. McMediocre was less than awesome.

There are other detail-inhabiting devils. The "big difference" in future earnings seems to vary according to which draft of the report we're looking at, and Chetty only claims them as far as the students turning twenty-eight-- the "lifetime earnings" claims are based on the assumption that the subjects will just keep getting the same raises for the rest of their lives that they got up until age twenty-eight. That is a heck of a bold assumption.

Chetty's work rests on the unproven assumption that VAM is not junk. VAM, in turn, rests on the assumption that 1) the Big Standardized Tests provide meaningful data and 2) that a magical formula can filter out all other factors related to student results on the BS Tests. Chetty's work also assumes that adult success is measured in monetary terms. And Chetty's work ignores the difference between correlation and causation, and instead makes a huge leap of faith to link cause and effect. I wold bet you dollars to donuts that we could perform research that would "prove" that eating a good breakfast when you're six, or having a nice pair of shoes when you're ten, can also be linked to higher-paying jobs in adulthood. As it is, we have "proof" that Nicholas Cage causes death by drowning, and that margarine causes divorce in Maine.

Chetty's work is not going to go away because it's sexy, it's simple, and it supports a whole host of policy ideas that people are already trying to push. But it is proof positive that just because somebody teaches at Harvard and wins awards, that doesn't mean they can't produce "research" that is absolute baloney.