Most of us suffer from employment bias, the belief that we are doing work that is self-evidently important. On that list of Things They Don't Teach You In Teacher School is the realization that while we can see how obviously important our work is, not everyone shares that belief.
Our employment bias simply sets us up for discouragement. But the employment bias of the folks who work with surveys and tests creates larger problems for all of us.
Take the research reported by Holly Yettick over at EdWeek. Joseph P. Robinson-Cimpian at the University of Illinois at Urbana came up with the surprising news that when you give anonymous surveys to teens about personal sociological information, your results might not be accurate because the little buggers will lie to you!
Let's pause for just a moment so that every single high school teacher and parent in America can exclaim, "Shocked! I am shocked!!"
Robinson-Cimpian's research provides some awesome examples. Follow-up research revealed in one case that out of 253 teens who reported using artificial limbs, 251 were lying. And it appears that many teens report themselves as gay when they actually aren't. Says Robinson-Cimpian, "Just like these jokester youths think it's funny to say they are gay and blind, they also think it's funny to say that they are suicidal, engage in sexually risky behavior, and take drugs."
Yettick does not want us to be too amused by these "mischievous responders," because they "can pose a serious threat to the validity of survey-based research studies."
I think Yettick is missing the picture here. These responders do not pose a serious threat to survey validity. They reveal why survey validity is a tissue-thin construct in the first place.
Yettick quotes this exchange at the beginning of the piece:
Q: Last school year, did you ever have an unexcused absence or a ditched class?
A: No, but why would I tell?
She characterizes this as "silly sarcasm." I would characterize it as an honest answer. What she calls a "mischievous responder" I would call a teen who decides not to play the game, who doesn't even bother to employ an adult's more sophisticated techniques for pretending to play nice while thumbing his nose at the system. Are there survey writers who know better? Statistically that seems probable, but the bulk of surveys and tests suggest it's a tiny group (tinier than the group of mischievous responders).
Survey and test creators make one huge, huge assumption-- that the people who use their instruments owe them an honest answer. Their employment bias is so strong, their certainty that they are doing self-evidently Important Work so clear, that they don't imagine people not seeing it. These folks live in a magical land where, if they walk up to a total stranger and ask him what kinds of people he likes to have sex with, he will feel obliged to give an honest answer.
The same holds true for standardized testing. The foundational belief of the testing industry, the concrete on which every other piece of structure rests, is the assumption that students who take The Test must of course take it Seriously. If a student is bored or tired or distracted or just doesn't care or doesn't see any point or just feels like playing ACDC or thinks that high-stakes testing is stupid or wants to write open-ended answers in the form of dirty limericks-- if that happens, every single piece of precious data from student results to VAM to student growth to all of it is crap crap crap.
On some level, the test fans know this. That's why we make the tests high stakes and instruct teachers to say inspiring things-- because we know there is no earthly reason for students to take any of this bubblicious baloney seriously. Robinson-Cimpian estimates that about 12% of responders are not being straight. I think he's being overly optimistic. This is just further evidence that the whole model of analyzing what's inside a person's head by asking standardized test questions is just a failed, broken joke. Mischievous responders just see the joke, and respond accordingly.
http://theadvocate.com/home/9080875-125/students-give-common-core-tests
ReplyDeleteThe part of the link above that pertains to this post....
ReplyDelete"The department said that, in a survey, nearly 70 percent of students who answered said the PARCC tests were easier or about the same as their current schoolwork and nearly 85 percent said none or few questions touched on materials they had not discussed in class.
The agency said nearly 78 percent of students favored the online assessments over paper and pencil."
when you got to wake up half the class, twice, during a standardized test, someone in this high stakes dance doesn't like this song.
ReplyDelete