Raj Chetty for Dummies

The name Raj Chetty has been coming up a great deal lately, like a bad burrito that resists easy digestion. A great deal has been written about Chetty and his scholarly work, much of it by other scholars in various states of apoplexy.My goal today is not to contribute to that scholarly literature, but to try to translate the mass of writing by various erudite economists, scholars and statisticians into something shorter and simpler than ordinary civilians can understand.

In other words, I'm going to try to come up with a plain answer for the question, "Who is Raj Chetty, what does he say, how much of it is baloney, and why does anybody care?"

Who is Raj Chetty?

Chetty immigrated to the US from New Delhi at age nine. By age 23 he was an associate professor of economics at UC Berkeley, receiving tenure at at age 27. At 30, he returned to his alma mater and became the Bloomberg Professor of Economics at Harvard.

He has since become a bit of a celebrity economist, consulted and quoted by the President and members of Congress. He has won the John Bates Clark Medal; Fortune put him on their list of influential people under forty in business.

Chetty started attracting attention late in 2010 with the pre-announcement of a publication of research that would give a serious shot in the arm to the Valued Added Measurement movement in teacher evaluation. His work has also made special appearances in the State of the Union address and the Vergara trial.

What does Chetty say?

The sexy headline version of Chetty is that a child who has a great kindergarten teacher will make more money as an adult.

The unsexy version isn't much more complicated than that. What Chetty et al (he has a pair of co-authors on the study) say is that a high-VAM teacher can raise tests scores in younger students (say, K-4) and while that effect will disappear around 8th grade, eventually those VAM-exposed children will start making bigger bucks as adults.

Implications? Well, as one of Chetty's co-authors told the New York Times--

“The message is to fire people sooner rather than later,” Professor Friedman said.

Chetty's work has been used to buttress the folks who believe in firing our way to excellence-- just keep collecting VAM scores and ditching the bottom 5% of your staff. Chetty also plays well in court cases like Vergara, where it can be used to create the appearance of concrete damage to students (if Chris has Mrs. McUnvammy for first grade, Chris will be condemned to poverty in adulthood, ergo the state has an obligation to fire Mrs. McUnvammy toot suite). You can read one of the full versions of the paper here.

Who disagrees with Chetty?

Not everybody. In particular, economist Eric Hanushek has tried to join this little cottage industry, and lots of reformy poicy makers love to quote his study.

But the list of Chetty naysayers is certainly not short. Chetty appears to evoke a rather personal reaction from some folks, who characterize him as everything from a self-important twit to a clueless scientist who doesn't understand that he's building bombs that blow up real humans. I've never met the man, and nothing in his writing suggests a particular personality to me. So let's just focus on his work.

Moshe Adler at Columbia University wrote a research response to Chetty's paper for NEPC. This provoked a response from Chetty et al, which provoked yet another response from Adler. You can read the whole conversation here, but I'll warn you right now that you're not going to just scan it over lunch.

Meanwhile, you'll recall that the American Statistical Association came out pretty strongly opposed to VAM, which also put them in the position of being critical-- directly and indirectly-- of Chetty. Chetty et al took it upon themselves to deliver the ASA a lesson in statistical analyses ("I will keep my mouth shut because these people are authorities in areas outside my expertise," is apparently really hard for economists to say) which led to a conversation recounted here.

What do the scholarly and expert critics say?

To begin with, the study has a somewhat checkered publication history, debuting as news blurbs in 2010 and making its way up to publication in a non-peer-reviewed journal, then to republishing as two articles, then in a peer-reviewed journal. That history, along with many criticisms of the study, can be found here at Vamboozled, the blog of Audrey Amrein-Beardsley (the blog is a wealth of resources about all things Vam).

Many criticize Chetty's methodology. Adler's critique suggests that Chetty may have fudged some numbers, dis-included some data, and ignored previous research that didn't fit his framework. Amrein-Beardsley (and others) accuse Chetty of ignoring context of the data. Many critics suggest that Chetty is trying to make a mountain out of a molehill. 

You can chase scholarly links all day long, though the NEPC link to Moshe's work and simply typing "Chetty" into the VAMboozled search box will provide more than enough reading for an afternoon. Or two.

So how much of Chetty's work is bunk?

I'm going to go with "most of it."

Chetty's idea was to link VAM measures to later success-- to be able to say, "Look! High-VAM teachers grow successful students." There are several problems with this.

First, studies of VAM-based teacher effectiveness always seem to descend into the same tautology. Use test scores to measure VAM. Use VAM to id the best teachers. Check to see if VAM-certified teachers raise test scores. Strip out the fancy language and funky math and you're left with a fairly simple tautology-- "Teachers who get students to have high test scores tend to get students to have high test scores." This is no more insightful or useful than research to show that bald men tend to be bald.

Second, Chetty doesn't seem to distinguish between correlation and causation. His results seem to scream for that consideration-- six year olds who do better on tests don't grow into twelve year olds who do better tests, but they do grow into twenty-eight olds who make more money. I'm no economist, but to me, the yawning gulf between the alleged cause and the supposed effect leaves enough room for a truckload of other possible causes. This holds together just about as well as "because I buried a toad under a full moon a year ago, I met my true love today."

And as it turns out, an explanation is readily available. We know who does better on standardized tests-- the children of high income families. We know who's more likely to get better-paying jobs as adults-- the children of high income families. It seems highly probable that the conclusion to be drawn from Chetty's research is, "Children of higher-income families do better on tests and get higher-paying jobs." 

Chetty himself tried to plug that last hole, with research about economic mobility that concluded that it's not any worse than it was a decade ago-- but it's still pretty lousy. Chetty et al also insist that the students were distributed across the classrooms in completely random fashion. This strikes many as an assumption without foundation.

Put another way-- a mediocre teacher with a classroom full of rich kids who test well would earn a high VAM and those well-heeled students would still go on to have well-paying jobs, and nothing in Chetty's model would ever reveal that Mr. McMediocre was less than awesome.

There are other detail-inhabiting devils. The "big difference" in future earnings seems to vary according to which draft of the report we're looking at, and Chetty only claims them as far as the students turning twenty-eight-- the "lifetime earnings" claims are based on the assumption that the subjects will just keep getting the same raises for the rest of their lives that they got up until age twenty-eight. That is a heck of a bold assumption.

Chetty's work rests on the unproven assumption that VAM is not junk. VAM, in turn, rests on the assumption that 1) the Big Standardized Tests provide meaningful data and 2) that a magical formula can filter out all other factors related to student results on the BS Tests. Chetty's work also assumes that adult success is measured in monetary terms. And Chetty's work ignores the difference between correlation and causation, and instead makes a huge leap of faith to link cause and effect. I wold bet you dollars to donuts that we could perform research that would "prove" that eating a good breakfast when you're six, or having a nice pair of shoes when you're ten, can also be linked to higher-paying jobs in adulthood. As it is, we have "proof" that Nicholas Cage causes death by drowning, and that margarine causes divorce in Maine.

Chetty's work is not going to go away because it's sexy, it's simple, and it supports a whole host of policy ideas that people are already trying to push. But it is proof positive that just because somebody teaches at Harvard and wins awards, that doesn't mean they can't produce "research" that is absolute baloney.

David Brooks Gets Everything Wrong

In an attempt to add some nuance to the positioning of Hillary Clinton, David Brooks took to the New York Times to say a series of not-very-wise things, including some deeply confused observations about education.

He sets up the idea of a main camp of Democrats who have believed for some unspecified period of time that by making American workers smarter and more productive, the country can rebuild its middle class.

He creates that notion so that he can explain its opposition--"populist progressives" who argue that education levels is not the root of all inequality. These are the folks who, Brooks observes, say that the game is rigged by the oligarchy, the workers' share is stagnant, and that corporate power has stifled worker gains.

People in this camp point out that inflation-adjusted wages for college grads has been flat for the past 14 years. Education apparently hasn't lifted wages. The implication? Don't focus on education for the bottom 99 percent. Focus on spreading wealth from the top. Don't put human capital first. Put redistribution first.

This is a leap that Evel Knevel would be impressed by. I am not sure who exactly finds that the implication of stagnant wages is that education should be a low priority. Education is awesome, still awesome, always awesome and desirable. But it does not magically transform the economy.

Brooks then leaps to the idea that redistribution will appeal to Clinton because it allows her to hit Wall Street and CEO's, which is kind of like suggesting that Scott Walker is looking for a policy that allows him to strike out against hard right conservatives or Paul Ryan is looking for a good argument to use against Ayn Rand.

But mostly Brooks wants to argue for education as the miracle engine of economic justice. And to make his argument, he trots out the work of Raj Chetty, a piece of research that proves conclusively that even researchers at Harvard can become confused about the difference between correlation and causation. (Chetty, for those of you unfamiliar with the "research," asserts that a good teacher will result in greater lifetime earnings for students. What he actually proves is that people who tend to do well on standardized tests tend to grow up to be wealthier, an unexciting demonstration of correlation best explained by things we already know-- people who score well on standardized tests tend to be from a higher-income background, and people who grow up to be high-income tend to come from a high-income background.)

Brooks also cites magical researcher David Autor of MIT, who believes that if everyone graduated from college with a degree, everyone would make more money because, reasons. Because if everyone had a college degree, flipping burgers would pay more? Because if everyone had a college degree, corporations would suddenly want to hire more people? The continued belief in the astonishing notion that a more educated workforce causes higher-paying jobs to appear from somewhere is big news to a huge number of twenty-somethings who are busy trying to scrape together a living in areas other than the ones they prepared.

Brooks isn't done spouting nonsense:

Focusing on human capital is not whistling past the graveyard...No redistributionist measure will have the same effect as good early-childhood education and better community colleges, or increasing the share of men capable of joining the labor force.

Because the vast number of high-paying jobs currently going unfilled is..... what?

Brooks says that redistributionists don't get it, that they believe that modern capitalism is fundamentally broken, but that their view is biased by short-term effects of the recession. I have two responses for that pair of thoughtbubbbles.

First, it's not clear whether capitalism is broken or not because we are currently tangled up in some sort of twisted fun-house mirror version of faux capitalism where the free market has been obliterated by a controlled money-sucking machine run by the government on behalf of the oligarchs. I'm actually a fan of capitalism, but what we currently have in this country is not much like capitalism at all.

Second, your argument about the "temporary evidence" of the recession is invalid because the recession was (and is) not the result of some mysterious serious of natural events. The economy went in the tank because the CEOs and Wall Street put it there. The economy broke because the "capitalists" broke it, and consequently the recession itself is Exhibit A in the case against modern faux capitalism and the greedheads who run it.

Throwing all this back at a magical belief in education is simply another way to blame poor people for being poor. So sorry you need food stamps and health care, but if you'd had the guts and character to go to college and get a degree, you wouldn't be in such a mess. Your poverty is just the direct result of your lack of character and quality. Well, that and your terrible teachers. But it certainly has nothing to do with how the country is being run. It's all on you, lousy poor person. And also your teachers.

Parents Demanding Testing

Given the rhetoric in the world of education, there are some things that I would expect to see, and yet don't. For example:

The Chetty Follow-up

Chetty et al are the source of the infamous research asserting that a good elementary teacher will results in an extra coupe of hundred thousand lifetime dollars for the students in their classroom.

Where are the follow-up and confirming studies on this? After all-- all we need are a pair of identical classrooms with non-identical teachers teaching from the same population. Heck, in any given year my own department has two or three of us are teaching randomly distributed students on the same track. All you'd have to do is follow them on through life.

In fact, I would bet that where the Chetty effect is in play, it's the stuff of local legend. For years people have been buzzing about how Mr. McStinkface and Ms. O'Awesomesauce teach the same classes with the same basic sets of kids, but her students all grow up to be successful, comfortably wealthy middle class folks and his students all grow up to a life of minimum wage jobs and food stamps.

I can't think of anything that would more clearly confirm the conclusions and implications of Chetty's research. So where is that report?

Parents Demanding Testing

To listen to testing advocates speak, one would think that our nation is filled with parents desperate for some clue about how their children and their schools are doing.

So surely, somewhere, there is a Parents Demand Tests group. Somewhere there must be a group of parents who have banded together to demand that schools give standardized tests and release the results, so that at last they know the truth.

"I just don't know," says some unhappy Mom somewhere in America. "I have no idea if Chris is learning to read or not. If only I had some standardized test results to look at."

"Dammit," growls some angry Dad somewhere in America. "I've had it with that school. Tomorrow I'm going down there to the principal's office to demand that Pat get a standardized test so we know if the kid can add and subtract or not."

But I can't find any such group on Facebook. Googling "Parents demanding testing" just gets me a bunch of articles about parents who are demanding tests of asbestos, air quality, other safety issues.

This is a striking gap. After all, we have plenty of robust-ish astro-turf groups to convince us that parents are, for example, deeply incensed over tenure-related policy. We are shown that parents really, really want tests to be steeped in VAM sauce and lit afire, so that terrible teachers can be roasted atop them.

And yet, as the crowds increasingly call for the standardized tests to be tossed out with last week's newspapers, it's chirping crickets from parent-land. Not CCSSO, not Arne Duncan, not any of the test-loving advocates has punctuated their pro-test protestations with a moment of, "And I'd like you to meet Mrs. Agnes McAveragehuman who will now tell you in her own words why she thinks lots of standardized testing is just totes swell."

But the reformsters must know plenty of people like Agnes. After all, they keep insisting that we need the tests or else people will not know how well students are learning, what schools are teaching, what progress is being made. Why, just Friday, there was Charles Barone, policy director for Democrats for Education Reform (which I am going to call DERP because somebody ought to) in the Washington Post opining, "I don’t know how else you gauge how students are progressing in reading and in math without some sort of test." Now maybe he imagines that there's a danger of schools in which no tests are being given whatsoever, but my own use of context clues leads me to believe that he is speaking of standardized testing.

When Arne Duncan spoke up to pretend to join the CCSSO initiative to pretend to roll back testing, he made his case for standardized testing by saying, "Parents have a right to know how much their children are learning," implying that only a standardized test could provide that answer.

It is possible that Arne's theory is that parents think they know what's going on with their school and their own children, but are actually deluded and misled (as witnessed in his classic genius quote from late 2013). But by now, over a year later, don't you think we'd have some converts, some parents saying, "Thank you, Mr. Duncan. Now that I have seen some test results, the scales have fallen from my eyes and I realize that merely living with and raising this tiny human has blinded me to a truth that only a standardized test could reveal. Don't let them take those tests away, sir!! I need them to tell me who my child is!" And yet, they don't seem to have appeared.

Maybe these parents are simply disorganized. Maybe they're uniformly shy. Maybe they use some of those underground web thingies so they can operate with cyberninja-like stealth. Or maybe they are raising snipes on a special farm where the ranch-hands ride unicorns and the pumps run on cold fusion. Maybe this world where parents are clamoring for standardized tests to reveal the truth about their children is a world that doesn't actually exist.

Punishing Teachers More Effectively

David Brooks' column praising small miracles made note of a new piece of research from Harvard that argues that while carrots may work better than sticks, the best way to use the carrot is to jam it into the horse's eyeball. (h/t to edushyster for the tip, not the carrot)

Our magic term for the day is "loss aversion," which is a fancy term for "people hate to give stuff up." The paper we'll be looking at is "Enhancing the Efficacy of Teacher Incentives through Loss Aversion:A Field Experiment" written by Roland G. Fryer, Jr. (Harvard University), Steven D. Levitt (The University of Chicago), John List (The University of Chicago), and Sally Sadoff (University of California San Diego). Let's learn some stuff, shall we?

Intro

Sigh. We know we're in just great shape when we lead with references to the baloneyfied research that "proves" that a measurable improvement in teacher quality creates the same measurable improvement in student achievement as a decrease in class size (the old "we don't need small classes-- just great teachers" research) and follows it up with Chetty's silly "a good teacher means your kid will grow up to make more money" research. And that's just the first paragraph.

In the second, we get the sideways assumption that VAM is a good measure of teacher quality and that unions make it too hard to get rid of bad teachers. In the third, we lament that merit pay hasn't done any good. "Good" of course means "has students with high test scores." Because when people talk about "good teachers," all they're thinking of is students scores on standardized tests. That's all we want from teachers, right?

On this foundation of sand and jello, our intrepid researchers set out to build a mansion of teacher improvitude.

The experiment (oops-- "field test") was performed in Chicago Heights. Teachers were randomly assigned to one of two groups-- either they were in the Gain group, working toward a possible end-of-year bonus, or they were in the Loss group, receiving a bonus up front which they would lose if their students didn't achieve bonus-worthy results. Bonuses for both groups were the same. Additionally, the researchers used the "pay for percentile" method developed by Barlevy and Neal, which is basically a stack and rank system where there are winners and losers. One would think that might have some significant effects on the field test, but apparently we're just going to barrel on assuming that it's a great idea and not a zero-sum dog-eat-dog approach that might shade the effects of a merit pay system.

Their findings were that there was a significant gain in math scores for Loss teachers' students (the significance was between 0.076 and 0.129, so make of that what you will) and, as expected, no significant affect for Gain teachers' students.


To the library

Part two of the paper is the review of the literature. If you're interested in this, you're on your own.

Program details

Chicago Heights is about thirty miles south of Chicago. They have a 98% free and reduced lunch population in elementary and middle school. The program was implemented with the cooperation of both the superintendent and the union. Of 160 teachers, 150 opted in. Maximum possible bonus pay was $8,000.

Working out the assignments of teachers was hard. So hard that apparently the researchers kind of gave up on tracking the reading side of this experiment and focused on the math. This was further complicated in that the design called for some teachers to be up for bonus on their own, while others were bonusing it up in team fashion.

And while the researchers keep saying that the teachers were assigned randomly, it turns out they were re-randomized with an algorithm that kept swapping teachers based on a set of rules until they were best aligned with the selection rules. So, unless I'm missing something, this was kind of like saying, "We randomly assigned people to groups of people with identical hair color and gender. So, we put all the blond women randomly in one group."

Teachers in the Loss group were given $4,000 at the beginning of the year and signed a contract stating they would give back the difference if their earned bonus came in below that amount. If they earned more, they got more. The tests used were the ThinkLink tests, which are described as otherwise low stakes tests, which again strikes me as a fairly critical factor that the researchers breeze right past.

Data and research design

Basically, these guys went in the back room and whipped up a big kettle of VAM sauce. You know. The same kind of thing that has been so widely discredited that the National Association of Secondary School Principals has come out against using it as a means of evaluating individual teachers. Also, they use some more math to deal with the event of a student having mixed teachers (on Loss group, one Gain group) during the day.

Results

You've already heard the big take-away. Other interesting bits of data include a much higher effect for K-2 students (though, since the VAMsauce depends on data going back four years, I'm wondering how exactly we crunched the little kids' numbers). There is a bunch of statistics-talk here as well, but much of it boils down to fancily-worded "Nothing to see here." There are charts for those who enjoy charts.

Interpretation and pre-emptive kibbitzing

The interpretation is simple. Merit pay will yield better test results if you let teachers hold it in their hands for nine months and threaten to take it back if their students don't do well on the Big Test.

The researchers anticipate three areas that might be used to dispute their results, so they address them ahead of time.

First, attrition. They anticipate the complaint that teachers will find other ways to improve their test scores including getting Little Pat McFailsalot out of their classroom, at least on test day. They tran some numbers and decided this didn't happen to any notable degree.

Second, liquidity restraints. We're talking about teacher money here. Teachers might spend their own money in the classroom to improve their bonus-earning chances, which would be a level playing field if all teachers were wealthy, but in a world where teachers have very little extra money to spend in the classroom (or Wal-mart or anywhere else), an extra $4K in September might tilt the field. In other words, did the group that got a $4K run out and spend it to make sure they kept it? Survey says no. Interesting sidelight-- when asked in March, 69% of the Loss teachers had not cashed their bonus checks yet.

Third, cheating. They decided that wasn't a factor because, reasons. Seriously-- isn't the whole hypothesis here that the bonus will motivate teachers to raise test scores any way they can? I have no reason to believe these teachers were cheating, but if this were my experiment, that would certainly be something I'd look for. What kind of pressure and temptation do you suppose will will be felt by a teacher who has already spent his "bonus" on house payments and groceries?

But it gets better. They argue that the proof that no cheating occurred is that results on the state test-- which had nothing to do with their incentive program-- came out about the same. So, the test results from the incentivized program were pretty much the same as the results that they got with no incentives at all. Maybe that means that test prep for the one test is also good test prep for the state test. Or maybe it means that the incentive program had no effect on anything.

Wrapping it up

I see enough holes in this very specific research to drive a fleet of trucks through. But let's pretend for a moment that they've actually proven something here. What would we do with it?

First, we'd need to convince a school district business office to let teachers hold a big pile of district money for nine months, thereby giving up a bunch of interest income and liquidity. At the same time, we'd have to get the administration and board to budget a merit pay line item for "Somewhere between a small amount and a huge mountain." These are great ideas, because if there's anything business managers love, it's letting someone else hold their money, and they only love that slightly less than starting a year with an unknowable balloon payment of indeterminate size next June.

When school districts talk about merit pay, they talk about a merit lump sum set aside at the beginning of the year so that teachers can fight over a slice of the already-set merit pie. As I've said repeatedly, no school board in this country is ever going to say to the public, "Our teachers did such a great job this year that we need to raise taxes to cover all the well-earned merit pay bonuses we owe them."

Of course, somebody would have to figure out the merit system. How many Harvard grad students work in your district? And how exactly will you figure out the math score bonus for your phys ed teacher? 

Districts could manage the financial challenges of this risk aversion model by pre-determining the aggregate merit pay in the district. This, combined with "pay for percentile," would absolutely guarantee open warfare among staff members, who would be earning their merit bonuses by literally ripping them dollars out of colleagues' hands. Boy, I bet teaching in that school would be fun.

The largest thing they haven't thought through

Instituted, this system will not play out like a merit bonus at all. If I start every year with an "extra" $4K (or whatever amount), I've gotten a raise, and every year I don't make my numbers is a year that I get a punitive retroactive pay cut.

In no time at all, this system morphs from a merit pay bonus system of rewards to a bad score DEmerit system of punishments. Rather than a bonus that really lifts up teachers, these folks have come up with a way to make punishment for low results even more painful and effective. A miracle indeed.