From the Vault: PVAAS in 2009

In one of my many side jobs, I'm a columnist at a local paper. After some of us were sent off for training in PVAAS back in October of 2009, I wrote this piece. This is where Pennsylvania was with this business six years ago.

(News-Herald, October 22) This week I was schooled by the state about more awesomeness that is Pennsylvania’s System of School Assessment (the PSSA tests). This latest big vat of coolaid was served up, ironically, in the Hemlock Room at IU6. When the state lowers itself to send consultants to instruct the poor hicks who toil in local school districts, there is always lots to learn.

For those of you still following the PSSA’s, we are down to the crunch. Remember, No Child Left Behind mandates that in four years, every single American school child will test above average. Since this is only slightly more likely than pigs flying out of Ed Rendell’s nose, the ever-benevolent state has leapt to the rescue with—more statistical tools!

The number crunching is called the Pennsylvania Value-Added Assessment System. “Value Added” is a useful term from the manufacturing world. Simple explanation: If I take a ten cent piece of sheet metal and turn it into a two dollar widget, I’ve added a buck ninety’s worth of value.

What that principle has to do with testing or educating students is not clear, unless the state means to suggest that students are the same as sheet metal and widgets. I was prepared to argue that point, but it turns out that the state’s meaning is something else; words mean whatever they want them to. And I can call my bicycle a stealth bomber.

PVAAS uses a thousand points of data to project the test results for students. This is a highly complex model that three well-paid consultants could not clearly explain to seven college-educated adults, but there were lots of bars and graphs, so you know it’s really good. I searched for a comparison and first tried “sophisticated guess;” the consultant quickly corrected me—“sophisticated prediction.” I tried again—was it like a weather report, developed by comparing thousands of instances of similar conditions to predict the probability of what will happen next? Yes, I was told. That was exactly right. This makes me feel much better about PVAAS, because weather reports are the height of perfect prediction.

It was hard not to well up with that sort of sarcasm during the indoctrination. We were there to copy numbers from websites onto papers, as if the zillions of tax dollars had suddenly crumped out before the developers could add the capability of printing reports. The consultant veered between trying to bludgeon us with jargon-filled gobbledegook and patronizing us with explanations of words like “excelling” and “improving.” And assurances that if we just taught what the state wants us to, everything will be great.

The fallacy at the heart of the PSSA remains. A bunch of multiple choice questions are a lousy measure of the reading skills of live humans. (The PSSA, we were told, is not a standardized test. Okay. I’ll think about that while I pedal my stealth bomber to the store.) You can run numbers through statistical models all day, but if the numbers are near-meaningless to start with, a massage doesn’t improve them.

The intent of the state has not changed much since they first launched the PSSA’s—Harrisburg wants to write the curriculum for every district in the state. What has changed is their tone. Ten years ago they were still trying to gently con us; now their contempt for local districts is beginning to shine through. They are really tired of talking to all these yokels; they would just as soon simply roll right over us and whip us into shape.

So prepare next for the proposed Keystone Exams. Students currently in 7th grade may face ten exit exams in order to graduate. And because the state wants to wield a big hammer, the exams will count for a full third of students’ final grades.

The process remains a two-handed slap in teachers’ faces. On the one hand, we’re treated as if we are the problem and that schools need to be rescued from us by brave bureaucrats and consultants. On the other hand, we are pushed to do things that we know are professionally unsound. Imagine suits going into hospitals and telling doctors, “You are making all these people sick. Stop using pointy scalpels and start operating with shovels.” High stakes multiple choice tests are bad education.
And the final indignity is that after these sorts of sessions, one on one in the hall, many of these consultants will freely admit that they’re selling poisoned punch, but hey, they’re well paid and they’ve gotten used to the taste.