It was a bit of a shock. I picked up my morning paper, and there was an article on the front page touting our school district's PVAAS scores, the commonwealth of Pennsylvania's version of VAM scores, and I was uncomfortably reminded that value-added measures are still a thing.
Value Added Measures are bunk.
We used to talk about this a lot. A. Lot. But VAM (also known as Something-VAAS in some states) has departed the general education discussion even though it has not departed the actual world of education. Administrators still brag about, or bemoan, their VAM scores. VAM scores still affect teacher evaluation. And VAM scores are still bunk.
So let's review. Or if you're new-ish to the ed biz, let me introduce you to what lies behind the VAM curtain.
The Basic Idea
Value Added is a concept from the manufacturing and business world. If I take a dollar's worth of sheet metal and turn it into a forty dollar toaster, then I have added thirty-nine dollars of value to the sheet metal. It's an idea that helps businesses figure out if they're really making money on something, or if adding some feature to a product or process is worth the bother.
Like when you didn't fix the kitchen door before you tried to sell your house because fixing the door would have cost a grand but would allowed you to raise the price of the house a buck and a half. Or how a farmer might decide that putting a little more meat on bovine bones would cost more than you'd make back from selling the slightly fatter cow.
So the whole idea here is that schools are supposed to add value to students, as if students are unmade toasters or unfatted calves, and the school's job is to make them worth more money.
Yikes! Who decided this would be a good thing to do with education?
The Granddaddy of VAAS was William Sanders. Sanders grew up on a dairy farm and went on to earn a PhD in biostatistics and quantitative genetics. He was mostly interested in questions like “If you have a choice of buying Bull A, compared to Bull B, which one is more likely to produce daughters that will give more milk than the other one.” Along with some teaching, Sanders was a longtime statistical consultant for the Institute of Agricultural Research.
He said that in 1982, while an adjunct professor at a satellite campus of the University of Tennessee, he read an article (written by then-Governor Lamar Alexander) saying that there's no proper way to hold teachers accountable for test scores.
Sure there is, he thought. He was certain he could just tweak the models he used for crunching agricultural statistics and it would work great. He sent the model off to Alexander, but it languished unused until the early 90s, when the next governor pulled it out and called Sanders in, and Educational Value-Added Assessment System (EVAAS) was on its way.
The other Granddaddy of VAAS is SAS, an analytics company founded in 1976.
Founder James H. Goodnight was born in 1943 in North Carolina. He earned a Masters in statistics; that combined with some programming background landed him a job with a company that built communication stations for the Apollo program.
He next went to work as a professor at North Carolina State University, where he and some other faculty created Statistical Analysis System for analyzing agricultural data, a project funded mainly by the USDA. Once the first SAS was done and had acquired 100 customers, Goodnight et al left academia and started the company.
William Sanders also worked a North Carolina University researcher, and it's not clear when, exactly, he teamed up with SAS; his EVAAS system was proprietary, and as the 90s unfolded, that made him a valuable man to go into business with. The VAAS system, rebranded for each state that signed on, became a big deal for SAS, who launched their Education Technologies Division in 1997.
Sanders passed away in 2017. Goodnight has done okay. The man owns two thirds of the company, which is still in the VAAS biz, and he's now worth $7.4 billion-with-a-B. But give him credit, apparently remembering his first crappy job, Goodnight has made SAS one of the world's best places to work-- in fact, it is SAS that influenced the more famously fun-to-work culture of Google. It's a deep slice of irony--he has sustained a corporate culture that emphasizes valuing people as live human beings, not as a bunch of statistics.
This is a highly complex model that three well-paid consultants could not clearly explain to seven college-educated adults, but there were lots of bars and graphs, so you know it’s really good. I searched for a comparison and first tried “sophisticated guess;” the consultant quickly corrected me—“sophisticated prediction.” I tried again—was it like a weather report, developed by comparing thousands of instances of similar conditions to predict the probability of what will happen next? Yes, I was told. That was exactly right. This makes me feel much better about PVAAS, because weather reports are the height of perfect prediction.
EVAAS helps create professional goals
EVAAS helps improve instruction
EVAAS will provide incentives for good practices
EVAAS ensures growth opportunities for very low achieving students
EVAAS ensures growth opportunities for students
EVAAS helps increase student learning
EVAAS helps you become a more effective teacher
Overall, the EVAAS is beneficial to my school
EVAAS reports are simple to use
Overall, the EVAAS is beneficial to me as a teacher
Overall, the EVAAS is beneficial to the district
EVAAS ensures growth opportunities for very high achieving students
EVAAS will identify excellence in teaching or leadership
EVAAS will validly identify and help to remove ineffective teachers
EVAAS will enhance the school environment
EVAAS will enhance working conditions
said, “When I figured out how to teach to the test, the scores went up.” A fifth grade teacher added,
“Anything based on a test can be ‘tricked.’ EVAAS leaves room for me to teach to the test and
2) When tested against another VAM system, EVAAS produced wildly different results.
3) EVAAS scores are highly volatile from one year to the next.
4) EVAAS overstates the precision of teachers' estimated impacts on growth. The system pretends to know things it doesn't really know.
5) Teachers of English Language Learners (ELLs) and “highly mobile” students are substantially less likely to demonstrate added value. Again, the students you teach have a big effect on the results that you get.
6) The number of students each teacher teaches (i.e., class size) also biases teachers’ value-added scores.
7) Ceiling effects are certainly an issue. If your students topped out on the last round of tests, you won't be able to get them to grow enough this year.
8) There are major validity issues with “artificial conflation.” (This is the phenomenon in which administrators feel forced to make their observation scores "align" with VAAS scores.) Administrators in Houston were pressured to make sure that their own teacher evaluations confirmed rather than contradicted the magic math.
9) Teaching-to-the-test is of perpetual concern. Because it's a thing that can raise your score, and it's not much like actual teaching.
10) HISD is not adequately monitoring the EVAAS system. HISD was not even allowed to see or test the secret VAM sauce. Nobody is allowed to know how the magic maths work. Hell, in Pennsylvania, teachers are not even allowed to see the test that their students took. You have to sign a pledge not to peek. So from start to finish, you have no knowledge of where the score came from.
11) EVAAS lacks transparency. See above.
12) Related, teachers lack opportunities to verify their own scores. Think your score is wrong? Tough.
They cite the "peer-reviewed study" funded by Gates and published by AERA which stated emphatically that "Value-added performance measures do not reflect the content or quality of teachers' instruction." This study went on to note that VAM doesn't seem to correspond to anything that anybody considers a feature of good teaching.