In an attempt to add some nuance to the positioning of Hillary Clinton, David Brooks took to the New York Times to say a series of not-very-wise things, including some deeply confused observations about education.
He sets up the idea of a main camp of Democrats who have believed for some unspecified period of time that by making American workers smarter and more productive, the country can rebuild its middle class.
He creates that notion so that he can explain its opposition--"populist progressives" who argue that education levels is not the root of all inequality. These are the folks who, Brooks observes, say that the game is rigged by the oligarchy, the workers' share is stagnant, and that corporate power has stifled worker gains.
People in this camp point out that inflation-adjusted wages for college grads has been flat for the past 14 years. Education apparently hasn't lifted wages. The implication? Don't focus on education for the bottom 99 percent. Focus on spreading wealth from the top. Don't put human capital first. Put redistribution first.
This is a leap that Evel Knevel would be impressed by. I am not sure who exactly finds that the implication of stagnant wages is that education should be a low priority. Education is awesome, still awesome, always awesome and desirable. But it does not magically transform the economy.
Brooks then leaps to the idea that redistribution will appeal to Clinton because it allows her to hit Wall Street and CEO's, which is kind of like suggesting that Scott Walker is looking for a policy that allows him to strike out against hard right conservatives or Paul Ryan is looking for a good argument to use against Ayn Rand.
But mostly Brooks wants to argue for education as the miracle engine of economic justice. And to make his argument, he trots out the work of Raj Chetty, a piece of research that proves conclusively that even researchers at Harvard can become confused about the difference between correlation and causation. (Chetty, for those of you unfamiliar with the "research," asserts that a good teacher will result in greater lifetime earnings for students. What he actually proves is that people who tend to do well on standardized tests tend to grow up to be wealthier, an unexciting demonstration of correlation best explained by things we already know-- people who score well on standardized tests tend to be from a higher-income background, and people who grow up to be high-income tend to come from a high-income background.)
Brooks also cites magical researcher David Autor of MIT, who believes that if everyone graduated from college with a degree, everyone would make more money because, reasons. Because if everyone had a college degree, flipping burgers would pay more? Because if everyone had a college degree, corporations would suddenly want to hire more people? The continued belief in the astonishing notion that a more educated workforce causes higher-paying jobs to appear from somewhere is big news to a huge number of twenty-somethings who are busy trying to scrape together a living in areas other than the ones they prepared.
Brooks isn't done spouting nonsense:
Focusing on human capital is not whistling past the graveyard...No redistributionist measure will have the same effect as good early-childhood education and better community colleges, or increasing the share of men capable of joining the labor force.
Because the vast number of high-paying jobs currently going unfilled is..... what?
Brooks says that redistributionists don't get it, that they believe that modern capitalism is fundamentally broken, but that their view is biased by short-term effects of the recession. I have two responses for that pair of thoughtbubbbles.
First, it's not clear whether capitalism is broken or not because we are currently tangled up in some sort of twisted fun-house mirror version of faux capitalism where the free market has been obliterated by a controlled money-sucking machine run by the government on behalf of the oligarchs. I'm actually a fan of capitalism, but what we currently have in this country is not much like capitalism at all.
Second, your argument about the "temporary evidence" of the recession is invalid because the recession was (and is) not the result of some mysterious serious of natural events. The economy went in the tank because the CEOs and Wall Street put it there. The economy broke because the "capitalists" broke it, and consequently the recession itself is Exhibit A in the case against modern faux capitalism and the greedheads who run it.
Throwing all this back at a magical belief in education is simply another way to blame poor people for being poor. So sorry you need food stamps and health care, but if you'd had the guts and character to go to college and get a degree, you wouldn't be in such a mess. Your poverty is just the direct result of your lack of character and quality. Well, that and your terrible teachers. But it certainly has nothing to do with how the country is being run. It's all on you, lousy poor person. And also your teachers.
Showing posts with label David Brooks. Show all posts
Showing posts with label David Brooks. Show all posts
Friday, March 6, 2015
Sunday, December 14, 2014
Punishing Teachers More Effectively
David Brooks' column praising small miracles made note of a new piece of research from Harvard that argues that while carrots may work better than sticks, the best way to use the carrot is to jam it into the horse's eyeball. (h/t to edushyster for the tip, not the carrot)
Our magic term for the day is "loss aversion," which is a fancy term for "people hate to give stuff up." The paper we'll be looking at is "Enhancing the Efficacy of Teacher Incentives through Loss Aversion:A Field Experiment" written by Roland G. Fryer, Jr. (Harvard University), Steven D. Levitt (The University of Chicago), John List (The University of Chicago), and Sally Sadoff (University of California San Diego). Let's learn some stuff, shall we?
Intro
Sigh. We know we're in just great shape when we lead with references to the baloneyfied research that "proves" that a measurable improvement in teacher quality creates the same measurable improvement in student achievement as a decrease in class size (the old "we don't need small classes-- just great teachers" research) and follows it up with Chetty's silly "a good teacher means your kid will grow up to make more money" research. And that's just the first paragraph.
In the second, we get the sideways assumption that VAM is a good measure of teacher quality and that unions make it too hard to get rid of bad teachers. In the third, we lament that merit pay hasn't done any good. "Good" of course means "has students with high test scores." Because when people talk about "good teachers," all they're thinking of is students scores on standardized tests. That's all we want from teachers, right?
On this foundation of sand and jello, our intrepid researchers set out to build a mansion of teacher improvitude.
The experiment (oops-- "field test") was performed in Chicago Heights. Teachers were randomly assigned to one of two groups-- either they were in the Gain group, working toward a possible end-of-year bonus, or they were in the Loss group, receiving a bonus up front which they would lose if their students didn't achieve bonus-worthy results. Bonuses for both groups were the same. Additionally, the researchers used the "pay for percentile" method developed by Barlevy and Neal, which is basically a stack and rank system where there are winners and losers. One would think that might have some significant effects on the field test, but apparently we're just going to barrel on assuming that it's a great idea and not a zero-sum dog-eat-dog approach that might shade the effects of a merit pay system.
Their findings were that there was a significant gain in math scores for Loss teachers' students (the significance was between 0.076 and 0.129, so make of that what you will) and, as expected, no significant affect for Gain teachers' students.
To the library
Part two of the paper is the review of the literature. If you're interested in this, you're on your own.
Program details
Chicago Heights is about thirty miles south of Chicago. They have a 98% free and reduced lunch population in elementary and middle school. The program was implemented with the cooperation of both the superintendent and the union. Of 160 teachers, 150 opted in. Maximum possible bonus pay was $8,000.
Working out the assignments of teachers was hard. So hard that apparently the researchers kind of gave up on tracking the reading side of this experiment and focused on the math. This was further complicated in that the design called for some teachers to be up for bonus on their own, while others were bonusing it up in team fashion.
And while the researchers keep saying that the teachers were assigned randomly, it turns out they were re-randomized with an algorithm that kept swapping teachers based on a set of rules until they were best aligned with the selection rules. So, unless I'm missing something, this was kind of like saying, "We randomly assigned people to groups of people with identical hair color and gender. So, we put all the blond women randomly in one group."
Teachers in the Loss group were given $4,000 at the beginning of the year and signed a contract stating they would give back the difference if their earned bonus came in below that amount. If they earned more, they got more. The tests used were the ThinkLink tests, which are described as otherwise low stakes tests, which again strikes me as a fairly critical factor that the researchers breeze right past.
Data and research design
Basically, these guys went in the back room and whipped up a big kettle of VAM sauce. You know. The same kind of thing that has been so widely discredited that the National Association of Secondary School Principals has come out against using it as a means of evaluating individual teachers. Also, they use some more math to deal with the event of a student having mixed teachers (on Loss group, one Gain group) during the day.
Results
You've already heard the big take-away. Other interesting bits of data include a much higher effect for K-2 students (though, since the VAMsauce depends on data going back four years, I'm wondering how exactly we crunched the little kids' numbers). There is a bunch of statistics-talk here as well, but much of it boils down to fancily-worded "Nothing to see here." There are charts for those who enjoy charts.
Interpretation and pre-emptive kibbitzing
The interpretation is simple. Merit pay will yield better test results if you let teachers hold it in their hands for nine months and threaten to take it back if their students don't do well on the Big Test.
The researchers anticipate three areas that might be used to dispute their results, so they address them ahead of time.
First, attrition. They anticipate the complaint that teachers will find other ways to improve their test scores including getting Little Pat McFailsalot out of their classroom, at least on test day. They tran some numbers and decided this didn't happen to any notable degree.
Second, liquidity restraints. We're talking about teacher money here. Teachers might spend their own money in the classroom to improve their bonus-earning chances, which would be a level playing field if all teachers were wealthy, but in a world where teachers have very little extra money to spend in the classroom (or Wal-mart or anywhere else), an extra $4K in September might tilt the field. In other words, did the group that got a $4K run out and spend it to make sure they kept it? Survey says no. Interesting sidelight-- when asked in March, 69% of the Loss teachers had not cashed their bonus checks yet.
Third, cheating. They decided that wasn't a factor because, reasons. Seriously-- isn't the whole hypothesis here that the bonus will motivate teachers to raise test scores any way they can? I have no reason to believe these teachers were cheating, but if this were my experiment, that would certainly be something I'd look for. What kind of pressure and temptation do you suppose will will be felt by a teacher who has already spent his "bonus" on house payments and groceries?
But it gets better. They argue that the proof that no cheating occurred is that results on the state test-- which had nothing to do with their incentive program-- came out about the same. So, the test results from the incentivized program were pretty much the same as the results that they got with no incentives at all. Maybe that means that test prep for the one test is also good test prep for the state test. Or maybe it means that the incentive program had no effect on anything.
Wrapping it up
I see enough holes in this very specific research to drive a fleet of trucks through. But let's pretend for a moment that they've actually proven something here. What would we do with it?
First, we'd need to convince a school district business office to let teachers hold a big pile of district money for nine months, thereby giving up a bunch of interest income and liquidity. At the same time, we'd have to get the administration and board to budget a merit pay line item for "Somewhere between a small amount and a huge mountain." These are great ideas, because if there's anything business managers love, it's letting someone else hold their money, and they only love that slightly less than starting a year with an unknowable balloon payment of indeterminate size next June.
When school districts talk about merit pay, they talk about a merit lump sum set aside at the beginning of the year so that teachers can fight over a slice of the already-set merit pie. As I've said repeatedly, no school board in this country is ever going to say to the public, "Our teachers did such a great job this year that we need to raise taxes to cover all the well-earned merit pay bonuses we owe them."
Of course, somebody would have to figure out the merit system. How many Harvard grad students work in your district? And how exactly will you figure out the math score bonus for your phys ed teacher?
Districts could manage the financial challenges of this risk aversion model by pre-determining the aggregate merit pay in the district. This, combined with "pay for percentile," would absolutely guarantee open warfare among staff members, who would be earning their merit bonuses by literally ripping them dollars out of colleagues' hands. Boy, I bet teaching in that school would be fun.
The largest thing they haven't thought through
Instituted, this system will not play out like a merit bonus at all. If I start every year with an "extra" $4K (or whatever amount), I've gotten a raise, and every year I don't make my numbers is a year that I get a punitive retroactive pay cut.
In no time at all, this system morphs from a merit pay bonus system of rewards to a bad score DEmerit system of punishments. Rather than a bonus that really lifts up teachers, these folks have come up with a way to make punishment for low results even more painful and effective. A miracle indeed.
Our magic term for the day is "loss aversion," which is a fancy term for "people hate to give stuff up." The paper we'll be looking at is "Enhancing the Efficacy of Teacher Incentives through Loss Aversion:A Field Experiment" written by Roland G. Fryer, Jr. (Harvard University), Steven D. Levitt (The University of Chicago), John List (The University of Chicago), and Sally Sadoff (University of California San Diego). Let's learn some stuff, shall we?
Intro
Sigh. We know we're in just great shape when we lead with references to the baloneyfied research that "proves" that a measurable improvement in teacher quality creates the same measurable improvement in student achievement as a decrease in class size (the old "we don't need small classes-- just great teachers" research) and follows it up with Chetty's silly "a good teacher means your kid will grow up to make more money" research. And that's just the first paragraph.
In the second, we get the sideways assumption that VAM is a good measure of teacher quality and that unions make it too hard to get rid of bad teachers. In the third, we lament that merit pay hasn't done any good. "Good" of course means "has students with high test scores." Because when people talk about "good teachers," all they're thinking of is students scores on standardized tests. That's all we want from teachers, right?
On this foundation of sand and jello, our intrepid researchers set out to build a mansion of teacher improvitude.
The experiment (oops-- "field test") was performed in Chicago Heights. Teachers were randomly assigned to one of two groups-- either they were in the Gain group, working toward a possible end-of-year bonus, or they were in the Loss group, receiving a bonus up front which they would lose if their students didn't achieve bonus-worthy results. Bonuses for both groups were the same. Additionally, the researchers used the "pay for percentile" method developed by Barlevy and Neal, which is basically a stack and rank system where there are winners and losers. One would think that might have some significant effects on the field test, but apparently we're just going to barrel on assuming that it's a great idea and not a zero-sum dog-eat-dog approach that might shade the effects of a merit pay system.
Their findings were that there was a significant gain in math scores for Loss teachers' students (the significance was between 0.076 and 0.129, so make of that what you will) and, as expected, no significant affect for Gain teachers' students.
To the library
Part two of the paper is the review of the literature. If you're interested in this, you're on your own.
Program details
Chicago Heights is about thirty miles south of Chicago. They have a 98% free and reduced lunch population in elementary and middle school. The program was implemented with the cooperation of both the superintendent and the union. Of 160 teachers, 150 opted in. Maximum possible bonus pay was $8,000.
Working out the assignments of teachers was hard. So hard that apparently the researchers kind of gave up on tracking the reading side of this experiment and focused on the math. This was further complicated in that the design called for some teachers to be up for bonus on their own, while others were bonusing it up in team fashion.
And while the researchers keep saying that the teachers were assigned randomly, it turns out they were re-randomized with an algorithm that kept swapping teachers based on a set of rules until they were best aligned with the selection rules. So, unless I'm missing something, this was kind of like saying, "We randomly assigned people to groups of people with identical hair color and gender. So, we put all the blond women randomly in one group."
Teachers in the Loss group were given $4,000 at the beginning of the year and signed a contract stating they would give back the difference if their earned bonus came in below that amount. If they earned more, they got more. The tests used were the ThinkLink tests, which are described as otherwise low stakes tests, which again strikes me as a fairly critical factor that the researchers breeze right past.
Data and research design
Basically, these guys went in the back room and whipped up a big kettle of VAM sauce. You know. The same kind of thing that has been so widely discredited that the National Association of Secondary School Principals has come out against using it as a means of evaluating individual teachers. Also, they use some more math to deal with the event of a student having mixed teachers (on Loss group, one Gain group) during the day.
Results
You've already heard the big take-away. Other interesting bits of data include a much higher effect for K-2 students (though, since the VAMsauce depends on data going back four years, I'm wondering how exactly we crunched the little kids' numbers). There is a bunch of statistics-talk here as well, but much of it boils down to fancily-worded "Nothing to see here." There are charts for those who enjoy charts.
Interpretation and pre-emptive kibbitzing
The interpretation is simple. Merit pay will yield better test results if you let teachers hold it in their hands for nine months and threaten to take it back if their students don't do well on the Big Test.
The researchers anticipate three areas that might be used to dispute their results, so they address them ahead of time.
First, attrition. They anticipate the complaint that teachers will find other ways to improve their test scores including getting Little Pat McFailsalot out of their classroom, at least on test day. They tran some numbers and decided this didn't happen to any notable degree.
Second, liquidity restraints. We're talking about teacher money here. Teachers might spend their own money in the classroom to improve their bonus-earning chances, which would be a level playing field if all teachers were wealthy, but in a world where teachers have very little extra money to spend in the classroom (or Wal-mart or anywhere else), an extra $4K in September might tilt the field. In other words, did the group that got a $4K run out and spend it to make sure they kept it? Survey says no. Interesting sidelight-- when asked in March, 69% of the Loss teachers had not cashed their bonus checks yet.
Third, cheating. They decided that wasn't a factor because, reasons. Seriously-- isn't the whole hypothesis here that the bonus will motivate teachers to raise test scores any way they can? I have no reason to believe these teachers were cheating, but if this were my experiment, that would certainly be something I'd look for. What kind of pressure and temptation do you suppose will will be felt by a teacher who has already spent his "bonus" on house payments and groceries?
But it gets better. They argue that the proof that no cheating occurred is that results on the state test-- which had nothing to do with their incentive program-- came out about the same. So, the test results from the incentivized program were pretty much the same as the results that they got with no incentives at all. Maybe that means that test prep for the one test is also good test prep for the state test. Or maybe it means that the incentive program had no effect on anything.
Wrapping it up
I see enough holes in this very specific research to drive a fleet of trucks through. But let's pretend for a moment that they've actually proven something here. What would we do with it?
First, we'd need to convince a school district business office to let teachers hold a big pile of district money for nine months, thereby giving up a bunch of interest income and liquidity. At the same time, we'd have to get the administration and board to budget a merit pay line item for "Somewhere between a small amount and a huge mountain." These are great ideas, because if there's anything business managers love, it's letting someone else hold their money, and they only love that slightly less than starting a year with an unknowable balloon payment of indeterminate size next June.
When school districts talk about merit pay, they talk about a merit lump sum set aside at the beginning of the year so that teachers can fight over a slice of the already-set merit pie. As I've said repeatedly, no school board in this country is ever going to say to the public, "Our teachers did such a great job this year that we need to raise taxes to cover all the well-earned merit pay bonuses we owe them."
Of course, somebody would have to figure out the merit system. How many Harvard grad students work in your district? And how exactly will you figure out the math score bonus for your phys ed teacher?
Districts could manage the financial challenges of this risk aversion model by pre-determining the aggregate merit pay in the district. This, combined with "pay for percentile," would absolutely guarantee open warfare among staff members, who would be earning their merit bonuses by literally ripping them dollars out of colleagues' hands. Boy, I bet teaching in that school would be fun.
The largest thing they haven't thought through
Instituted, this system will not play out like a merit bonus at all. If I start every year with an "extra" $4K (or whatever amount), I've gotten a raise, and every year I don't make my numbers is a year that I get a punitive retroactive pay cut.
In no time at all, this system morphs from a merit pay bonus system of rewards to a bad score DEmerit system of punishments. Rather than a bonus that really lifts up teachers, these folks have come up with a way to make punishment for low results even more painful and effective. A miracle indeed.
Subscribe to:
Posts (Atom)