Houston Slams VAM (Plus: All About SAS)

As a Pennsylvanian teacher, I am paying particular attention to the news from Houston, where VAM just suffered another well-deserved loss. I'll get to that in a second, but let me set the stage and tell you a little story of how we arrived here.

Houston and Reformsters

The Houston Independent School District has always been out in front of education reformsterism. It was Houston where Superintendent Rod Paige performed the "Texas Miracle" of  raising test scores even for non-wealthy, non-white students. The Texas Miracle became part of the justification for the test-driven baloney of No Child Left Behind, and Rod Paige was whisked to Washington to employ his miracle-inducing powers as George Bush's Secretary of Education.

Only, there was no Texas Miracle. Houston was not an example of how test-centered accountability could create excellence; instead, it was an example of Campbell's Law, of how using bad measure as proxy for a complex social behavior just leads to increasing corruption (aka gaming, spinning and cheating) of that measure. Houston schools had pushed low-scoring students out the door, or held them back a year and then leapfrogged them over the testing grade.

Fast forward a few years, and we find HISD signing on with SAS to use their nifty Value-Added instrument called EVAAS.

The SAS Story

SAS has been in the analytics for a while ("Giving you the power to know since 1976"). Founder James H. Goodnight was born in 1943 in North Carolina. He earned a Masters in statistics; that combined with some programming background landed him a job with a company that built communication stations for the Apollo program. It was also, apparently, a really crappy company with something like a 50% turnover rate. That apparently stuck with him.

He next went to work as a professor at North Carolina State University, where he and some other faculty created Statistical Analysis Software for analyzing agricultural data, a project funded mainly by the USDA. Once the first SAS was done and had acquired 100 customers, Goodnight et al left academia and started the company. William Sanders, often credited as the developer of EVAAS (which sometimes trades its first letter for the state of adoption, so TVAAS in Tennessee, PVAAS in Pennsylvania) was also a North Carolina University researcher, who was also employed by SAS. While SAS works with analytics for many fields, EVAAS is their prize value-added education product.

Goodnight has done okay. The man owns two thirds of the company and is now worth $8.5 billion-with-a-B. But give him credit, apparently remembering his first crappy job, Goodnight has made SAS one of the world's best places to work--  in fact, it is SAS that influenced the more famously fun-to-work culture of Google.

Now, I know I'm already digressing, but the irony here is too rich to avoid. SAS has established its great workplace reputation by focusing on four core leadership values:

* Value People Above All Else
* To Give Is To Get
* Trust Above All Things
* Ensure Employees Understand the Significance of Their Work

As I said-- fraught with irony. Because one of SAS's many famous products is now regularly used to help create workplaces that are hostile to all of these core ideas. Value people? No, with value-added, what we literally value is data and test scores. Not people at all. The give-get thing? That is rejected by reformsters who explicitly argue that we must look only at outcomes (what we get) and ignore inputs (what we give). The whole basis for using VAM systems is that teachers and schools cannot be trusted to tell the truth about how students are learning and how teachers are teaching. And the use of VAM attempts to strip teaching of all significance except the job of getting students to score better on standardized tests.

Somehow Goodnight has built a little world where people live and work among dancing rainbows and fluffy fairy dust clouds, and they spend their days manufacturing big black rainclouds to send out into the rest of the world.

Back To Houston

No school system in the country embraced VAM with more ardent fervor than the Houston district (largest district in Texas and among the top ten largest in the US). In 2007, the district created a teacher evaluation program called Accelerating Student Progress: Increasing Results and Expectations (ASPIRE)and it contracted with SAS to provide the VAMmy sauce in which this program was to be soaked. By 2007, SAS was not the only company milking this particular market, but they were arguably the most wide-spread and most successful. Their EVAAS system must have seemed like just the thing for HISD.

You can read the whole sad story in painful detail in this piece of serious research published in the Education Policy Analysis Archives in 2012 by Audrey Amrein-Beardsley and Clarin Collins (both of Arizona State University). As Gene Glass notes, this was not the first time that HISD had tried implementing some sort of test-based incentive pay for teachers-- Glass has conducted even earlier research looking at such systems, again paying special attention to Houston. Way back in the 1980s Glass had delineated the problems inherent in teacher evaluation systems based on student scores. Those include that such a system "elevates tests themselves to the level of curriculum goals, obscuring the distinction between learning and performing on tests" as well as being a system that was developed mainly to show the public that leaders are taking education real serious and holding high standards.

As Amrein-Beardsley and Collins showed (and they showed this, mind you, not just with EVAAS in general, but with Houston in particular), the value-added scores were only slightly less random than rolling ten-sided dice in a washing machine mounted on the back of a agitated elephant. The EVAAS scores proved to be inconsistent and unpredictable. Knowing a teacher's score from last year did not give you the lightest hint of what this year's score might be. And that paper was published in 2012.

Houston Gets Antsy

Using EVAAS for incentive/merit/bonus was old news in Texas when in 2010, HISD decided to use it for firing purposes. The board voted 7-0 to take that plunge, with 1,000 angry teachers in the room. The Texas Tribune noted that EVAAS was a proprietary mystery and nobody could or would say how it worked. But the HISD teachers learned a hard lesson in political activity-- five of the board members who voted to VAM them had been opposed by the union, although two of the five had not even had opponents on the ballot.

In 2011, there was further debate about the use of VAM scores in formal evaluations, with HISD proposing a whopping 50% of teacher eval to be vammified. The debate was still chock full of VAM-loving baloney:

"Reasonable people can disagree on how much high stakes should be placed on the information," said Rob Meyer, director of the Value-Added Research Center at the University of Wisconsin-Madison.

Nope. Reasonable people wouldn't use VAM information to decide much of anything.

But the push back would come, even if it took a few years. A group of HISD teachers, with the support of their union, is taking HISD to court over their scores. Along with Sheri Lederman's successful (though very specific) lawsuit against VAM in NY, this could mark a front along which value-added baloney may be successfully pushed back.

But There's Another Reason VAM Is In Trouble

EVAAS is in trouble in Houston for the simplest of reasons-- it doesn't work.

Last week, when it was time to renew the EVAAS contract (and thereby renew the multi-million dollar federal grant that comes with it), Amrein-Beardsley reports part of the conversation went like this:

“If not now, then when?” the board member asked. “I remember talking about this last year, and the year before. We all agree that it needs to be changed, but we just keep doing the same thing.” A member of the community said to the board: “VAM hasn’t moved the needle. It hasn’t done what you need it to do. But it has been very expensive to this district.” He then listed the other things on which HISD could spend (and could have spent) its annual $680K EVAAS estimate costs.

HISD has been doing the EVAAS thing for years, and it hasn't made a dent in the all-important test scores.

And so, the HISD vote ended in a tie, which means the EVAAS was not renewed, which means SAS is out half a million in revenue and the district will have to come up with some new way to evaluate teachers.

Why Do I Care?

As mentioned earlier, EVAAS is TVAAS in Tennessee and PVAAS in Pennsylvania, where it is a senseless piece of junk. It makes SAS a lot of money, but it does not provide anything of use for classroom teachers (nor can it be explained in any satisfactory manner). It should go away, and it should go away today.

Don't ask me "Well, what should we do instead." First of all, this post is already too long. Second of all, when you object to someone trying to treat your sprained ankle by hitting you in the head with a hammer, they don't get to say, "Well, I'm going to keep doing this till you come up with a better idea." If you want to propose something as a treatment, the burden of proof is on you and your particular brand of snake oil-- not on your patient.

EVAAS does not help educate students, does not help teachers improve in their craft, does not help taxpayers and administrators determine who's doing a good job. All it does is help James Goodnight remain one of the richest guys in America.