Wednesday, February 24, 2021

Mark Weber's Three Critical Testing Questions

This is not the first time I've piggy-backed on a post from Mark "Jersey Jazzman" Weber, but sometimes what he posts just sets a bright blazing lightbulb off in my head. Weber is a certified PhD-carrying academic of data crunching, and he is an absolute master of rendering complicated ideas comprehensible to civilians, and that's what he's done with his post  about why "We Don't Need Standardized Testing In A Pandemic."  

I encourage you very much to go read the whole post. But sometimes saying, "This! This! This!" isn't enough for me. So. What I'm going to center on is his three-question response to anyone who says that students must take a standardized test.

1) What are you going to do with the results?

This question is hugely powerful and by far the least answered. Here's Weber:

A core concept of assessment is that tests must be shown to be valid for the purposes in which they will be used. In other words: you should make a separate argument for every proposed use of a test. A test that may be valid to use for, say, determining whether there are enough overall resources in the education system isn't necessarily valid for the purpose of determining whether a student should pass to the next grade.

Psyshometricians often speak of making a validity argument in favor of the use of a test for a particular purpose. That argument should touch upon the relevancy of the outcomes to a specific use, the consequences of making decisions based on these outcomes, the opportunity to learn the content in the test, and other factors.

This is just so key. As Weber notes, one of the huge problems with the current Big Standardized Test system is that one test is supposed to serve a variety of purposes. Evaluate schools, evaluate teachers, assess curriculum, inform parents, plus serve as a go-to measure of whether Reform Program X is "effective" or not. Plus, now, figuring out how far "behind" students are. And most of the time nobody has made a validity argument for any of it.

The answer to the question also needs to be specific. The announcement that the administration won't waive the test leans on that old standard promise that test results will be used to "target resources and supports to the students with the greatest needs." That's great.


How do you plan to use the test results to do that targeting? Where is the bill or program proposal that shows how test scores will be linked to the targeting of resources?

Of course there is no such proposal. There never has been. What states have actually done is things like using low scores to target schools for charterization. 

When your doctor wants to perform a biopsy, there's a reason, and it's not "Just thought IK'd chop you open, dig around, see if I can figure out anything about your diet or your blood pressure or maybe check for cancer, and then we'll do, I don't know, something." Your doctor has a specific intent both for the operation and for hat's going to be done with the results.

When we get to specifics of these tests, we are forced to see how useless they are. Are they to help teachers find out how "behind" students are? How will that work-- months from now, when the teachers see the basic test scores (but not the questions that were answered)? What will this one test tell teachers (particularly those who don't teach math or reading) that they can't --and haven't-- already figure out on their own in their own classroom?

And on what planet will we find the parent (because the Biden administration throws them in the test-justification mix, too) saying this-- "I had no idea how my child was doing in school, but now that I can see some very general results on a test she took a few months ago, it's all clear to me." 

If you can't tell me exactly what you want to do with the results of a test, you have no business giving the test. 

2) What is the cost of administering the test?

Testing is, of course, a billion-dollar industry, supported with tax dollars. But as Weber points out, there are other costs. We've had the long-term costs of narrowing the curriculum, and the cost of telling children of poor families that they are academic failures.

But there is also opportunity costs. Time spent t=on test prep and test taking is time not spent on education. I guarantee you that among the first reactions among teachers to the Biden Testing Is On announcement, right after some fiery obscenity, was the thought, "Well, what I'm going to have to cut out now."

For most of my career, my year to year aspiration was to see what else I could add to my class. By becoming more efficient, more focused, and better at understanding where my students were, I could claim some space in the 180 days and fill it with something else. But towards the end, as my school became more "data driven," the question flipped, and year to year I found myself asking, "Now that I'm losing more time to testing, what can I stand to cut." 

This year, time is more valuable and rare than ever. That makes the cost of testing astronomical.

3) Is the cost of testing worth it?

Again, directly from Weber:

What, exactly, will statewide standardized test outcomes tell us that we didn't already know, or that we couldn't find out some other way? That students didn't gain as much learning as they would have without the pandemic? We already know this; and again, it's not like the tests will give educators data they couldn't get other ways that are much faster and more detailed.

Will we learn that students who are economically disadvantaged need more resources to equalize their educational opportunity? We already knew that before the pandemic. And does anyone really think that otherwise reluctant politicians will be persuaded to dole out more funds when they see this year's test scores? Really?

And on top of that, will there be any magical means of determining which part of the score is the result of student knowledge and which will be the result of students taking tests during a chaotic year under lousy circumstances under family stress? There is always reason to doubt that the tests measure what they purport to measure; this year that doubt can be multiplied a hundredfold.

It will be worth it to some folks. Test manufacturers will get their payday. And I'm betting that once test results are out, folks who like to declare that public schools are disastrous failures will have a field day pushing the critical corporate reforms that must be implemented to Save Students. The Big Standardized Test has always been a tool for those who want to break public education down and sell off the parts; these test results should be a disaster capitalist holiday, a national-scale Hurricane Katrina.

Weber's three questions are a useful response to testing in any year, yielding results that can be debated by reasonable people. But this year it's pretty clear that the cost will be high and the results will be mostly worthless. Remember these three questions for the years ahead, but this year, the Biden administration should reverse its bad decision.


  1. Is anyone officially lobbying for the decision reversal?

  2. Measurement is such a potentially poisonous thing. It can frame the most innocent experience as a failure or success. Imagine new parents, contemplating their baby, pulling out the sharpie and marking off the kid's *anticipated* growth on the door frame. Who would do such a crazy thing? Apparently, we would - that's basically what the BST does. It instils in kids a perception that they are a stuff to be pounded into shape. For years I'd been arguing with people who dragged out the old chestnut about how schools were modelled on factories to instill in children the principles of mechanization (not really true, and missing the point). But now, that really is the message: subjects don't matter, and you don't matter, but Staying On Track is the key to success - and Falling Off, well, that's certain doom. No wonder so many kids are anxious and miserable.

  3. 4) After nearly two decades of standardized testing under NCLB and ESSA can you identify or describe three substantial and measurable accomplishments supported by test score data?

    1. YES!! I would LOVE someone to address this. I have literally never heard of a single one.

  4. The Big Testing Machine is an objective failure according independent sampling by NAEP. The Brookings Institution sums it up: “In fact, for both fourth grade and eighth grade, the average scores from 2017 were identical to the average scores from 2009. This period includes a rare drop in math scores from 2013 to 2015.”

    It is irrational to continue such a failed approach.