Test Scoring Monkeys

It's been less than a month since Motoko Rich traveled to San Antonio to hear a Pearson test scoring supervisor explain that scoring the tests is like making a Big Mac. Now Claudio Sanchez has made the same journey for NPR, and the results are no more flattering for Pearson than those from Rich's jaunt.

The center uses scorers from many walks of life, though a four-year degree is required. What is not required is any sort of opinion about the quality of the questions.

David Connerty-Marin, a spokesman for PARCC, says it's not up to a scorer or Pearson or PARCC to say, "Gee, we think this is too hard for a fourth-grader."

What is or is not developmentally appropriate, he says, is not an issue because the states have already made that decision based on the Common Core Standards.

One of these rainy summer days, I'll spend some time running up and down the internet and see if I can find, somewhere in the great chain of standards and testing, the person who says, "Me, I'm the one. I'm the guy who decides that this test item is appropriate for an eight year old." But until the day comes, we're stuck with test manufacturers who say, "Well, we just follow what the state tells us" and states that say, "Well, we lean on the professionals to design these things" and a whole bunch of people who point and shrug and say, "Well, you know, the standards" as if the standards were dropped down from heaven on the back of a golden cloud that deposited them on top of a burning bush.

The article's description of the scoring process reveals for the gazillionth time that the constructed open-ended responses are not any kind of open-ended response at all, but a bizarre exercise in blind matching.

Sanchez talked to one retired teacher who has worked eight years for Pearson.

She looks for evidence that students understood what they read, that their writing is coherent and that they used proper grammar. But it's actually not up to Vickers to decide what score a student deserves.
Instead, she relies on a three-ring binder filled with "anchor papers." These are samples of students' writing that show what a low-score or a high-score response looks like.

"I compare the composition to the anchors and see which score does the composition match more closely," Vickers says.

That's not an open-ended response. It's a newer, more gigantic form of multiple choice, where students choose from all the possible combinations of words in the English language in hopes of selecting the one combination that is acceptable to test manufacturers. Those folks in Texas have the same basic task as the guy checking the work of the million monkeys to see which one has typed a Shakespeare play. This is a test where students are given a box full of LEGOs and told to build something, but will only get credit if they build the right thing.

And, of course, reporters can't know any specifics about any of the actual test questions or responses.

Pearson does not allow reporters to describe or provide examples of what students wrote because otherwise, company officials say, everybody would know what's on the test.

I don't even know how to explain how insane that is. In my own classroom, my students know exactly what is going to be on a test. Any test that depends on super-duper secrecy is a terrible test. It is also possibly a test manufactured by cheap money-grubbing slackers who don't want to do the work of updating it annually.

Pearson delivers a backhanded acknowledgement that secrecy has not been their friend. One supervisor notes that since the public doesn't know what Pearson's doing, "misconceptions" abound. But Sanchez gets the last word on that subject:

Most Americans have been in the dark, says Thompson. So the risk for Pearson, PARCC and the states is that by trying to be more transparent this late in the game, people may very well end up with more questions than answers.