In Monday's New York Times, journalist Motoko Rich gives a master class in how to let the subjects of a story make themselves look ridiculous.
The piece takes us to San Antonio to give us a look at how the Big Standard Tests are actually graded, and in doing so, shows how the BS Tests are not one whit more advanced than the old school bubble tests they claim to replace.
Rich begins by noting that the scoring is not necessarily (or even probably) done by actual teachers. But Pearson's vp of content and scoring management is here to reassure us with this astonishing quote:
“From the standpoint of comparing us to a Starbucks or McDonald’s, where you go into those places you know exactly what you’re going to get,” said Bob Sanders, vice president of content and scoring management at Pearson North America, when asked whether such an analogy was apt.
“McDonald’s has a process in place to make sure they put two patties on that Big Mac,” he continued. “We do that exact same thing. We have processes to oversee our processes, and to make sure they are being followed.”
This is not news, really. For years we've been reading exposes by former graders and Pearson's advertisements in craigslist. It can be no surprise that the same country that has worked hard to teacher-proof classrooms would also find a test-scoring method suitable for folks with no educational expertise.
How low does the bar go? Consider this quote from one scorer, a former wedding planner who immigrated from France just five years ago:
She acknowledged that scoring was challenging. “Only after all these weeks being here,” Ms. Gomm said, “I am finally getting it.”
Sigh. I cut and pasted that. It is not one of my innumerable typos.
Look, here's the real problem revealed by this article (and others like it).
The test manufacturers have repeatedly argued that these new generation tests are better because they don't use bubble tests. They incorporate open-ended essay(ish) questions, so they can test deeper levels of understanding-- that's the argument. A multiple choice question (whether bubbling, clicking, or drag-and-dropping) only has one correct answer, and that narrow questioning strategy can only measure a narrow set of skills or understanding.
So essays ought to be better. Unless you score them like this, according to a narrow set of criteria to be used by people with no expertise in the area being tested. If someone who doesn't know the field is sitting there with a rubric that narrowly defines success, all you've got is a slightly more complicated bubble test. Instead of having four choices, the student has an infinite number of choices, but there's still just one way to be right.
Nobody has yet come up with a computerized system of grading writing that doesn't suck and which can't be publicly embarrassed. But if you're going to hire humans to act like a computer ("Just follow these instructions carefully and precisely"), your suckage levels will stay the same.
If it doesn't take a person with subject knowledge to score the essay, it doesn't take a person with subject knowledge to write it.
So the take-away from Rich's piece is not just that these tests are being graded by people who don't necessarily know what the hell they're doing, but that test manufacturers have created tests for which graders who don't know what the hell they're doing seems like a viable option. And that is just one more sign that the Big Standardized Tests are pointless slices of expensive baloney. You can't make a test like McDonalds and still pretend that you're cooking classic cuisine.