Saturday, November 7, 2015

How Assessment Ruins Standards

Long time readers know that I do not subscribe to the whole "The standards are swell; it's just those evil tests that screw everything up" school of thought. I think there are plenty of reasons to oppose national standards no matter what standards they are, and plenty of reasons to believe that no set of national standards will ever accomplish any of the goals set for them.

But let's set all of that aside for a moment and talk about how the very attempt to assess standards-based outcomes ruins those standards.

For my example, I'm going to pick the oft-noted CCSS standards about evidence.

I pick it because it's a part of the Common Core that doesn't particularly bother me. Like most English teachers, I've been encouraging (in many cases, quite vigorously) my students to provide support for whatever idea they are trying to assert. ("No, Chris-- saying Huck Finn is a dynamic character because he does dynamic stuff in a dynamic way does not really make your case"). When I assign a paper, two of the main questions I consider when assigning a grade are 1) did you actually have a point and 2) did you support it with actual evidence.

So in this area, the Core and I can co-exist peacefully.

But this kind of evidence use is a tool, a technique, and so it can't be assessed in a vacuum, just as you cannot judge somebody's hammering skills by just watching them hold a hammer or judge their free-throw shooting skills without handing them a ball. And that's where we get into trouble.

Assessing this skill is easiest when tying it to an act of critical thinking. But critical thinking has to be an open-ended activity. (Here's a quick tip-- if your question only has a single possible correct answer, it is not assessing critical thinking skills.) In my classroom, the most obvious avenue is a response to literature, though it will also work to deal with social issues, human behavior puzzles, historical research, or evaluating somebody else's essay work. Just to name a few.

To make this assessment work, I have to really know my stuff. I have to know Huck Finn frontwards and backwards, and perhaps augment my own expertise with lots of reading of scholarly critiques of the work. I have to know the work well enough that I can give the student freedom to go where his ideas lead him without me having to say, "Sorry, but I don't know anything about that, so I can't grade it, so you can't do it."

Every area of study is like a big patch of real estate, and I need to be a well-informed native guide who knows the territory so well that no matter where the student wanders off to, I'll know the terrain. When I don't know the territory, I end up fencing students in. "Just stay on the path-- don't stray." At the very worst, I will lay rails and make everyone ride through the territory on a train that only goes where it is meant to. The train tour is by far the least interesting, the least useful, the least rewarding, the least educational, and the least authentic way to explore the territory. It rules out all discovery and invention, and it is certainly hard on any prospect for joy or excitement.











But again-- we can only scrap the train and the pathway if we have a knowledgeable native guide. That's how my juniors can do a unit built on research of local history-- because I've made myself a little bit of an expert on the subject. That's how my colleague can do a massive end-of-year project with her AP seniors about Paradise Lost-- because she knows and loves that work. If we had to trade projects, we would both be lost.

But within our areas, we are qualified to tell the difference between evidence that is really evidence, and evidence that is just a piece of camouflaged baloney.

So here's why a standardized test can't test this standard.

First, a standardized tests starts with the assumption that any person (or computer program) should be able to evaluate the student's work without possessing any actual expertise at all.

Second, the answers have to be evaluatable in a very short time span.

And that means we will travel through the territory strapped into a seat on a tightly run train.

Look through the PARCC samples. These are slightly spiffed up multiple-choice single answer questions. The new SAT essay is just a wordier version of the same thing-- look at a piece of writing, determine what point the test manufacturers think you should see, and support it with the evidence that the test manufacturers believe are the correct pieces of evidence. These folks keep coming up with more complicated ways to ask closed-ended questions. This is partly, I suspect, because it's simply faster and more efficient to have a test of closed-ended questions that can be scored by any non-expert (or non-human). But also partly because some of these folks just have a narrow, cramped, stilted view of life and the world. "I handed you a brush, a small flat surface, and a jar of blue paint. What else would any normal person do except paint the flat surface solid blue?"

But that's not critical thinking, and it's not supporting your ideas with evidence.

Now, what is often shortened to "support with evidence" in discussion of the CCSS is actually mangled pretty badly in the actual wording of the standards, but even if it weren't, the Big Standardized Tests would mangle this idea to death anyway.

What this should properly mean is, "Come up with an idea that makes sense to you, and support it with evidence that you believe backs up your idea." But the only person who can evaluate that is a classroom teacher who possesses 1) enough expertise to evaluate the student's process 2) fewer than 1,000 students so that said evaluation can occur within the next week or so.

So what the standard means in a standardized test situations is, "Figure out what idea the test manufacturer wants you to find, and then locate the details that the test manufacturer wants you to pick out."

When we talk about test prep in the ELA world, we're talking about getting students into that second mindset, about training them to figure out the One Correct Answer associated with each piece of reading and the Only Correct Evidence located as well. And then repeat it, year after year after year.

And what really sucks is that we are getting good at it, and our students are paying the price. This whole blog piece is the result of a conversation I had with a colleague, both of us concerned because, despite our best efforts, we find our students over the past few years have become progressively worse at really engaging with reading and writing. They have learned that it's not about thinking or reacting or engaging and gripping the material with your own brain. What do you think? Why do you think that? Can you convince me to agree with you? I feel like my students have only recently encountered these kinds of questions, and they aren't sure what to do with them, because they've learned that there's only one right way to do each reading and writing task, and that one right way is known by someone else, and it's up to them to figure that out.

Of course, this issue didn't start with Common Core and Big Standardized Testing, but those conjoined twins have made things so much worse. When you try to make a complicated idea something you can assess "at scale," you do enormous damage. When you write a standard specifically so that you CAN test it "at scale," you break it entirely.


 

11 comments:

  1. It is the same in mathematics. Students are less able to engage in problem-solving when handed open-ended situations because they have been trained to pick out an answer from the (usually) four options presented to them. To my repertoire of stock speeches I have had to add an "I don't care about the test" harangue. "How will I know the right answer choice?" triggers the speech. I also find myself debating the purpose of school with students, who assert that the purpose of school is to pass tests.

    ReplyDelete
    Replies
    1. I see the same thing in my college students. I teach philosophy at a community college in North Carolina. When I started years ago, some of my students could really think for themselves, but now they just want me to tell them the answers. Of course, it doesn't work that way in college, and it shouldn't be like that in K-12.. This year in particular they seem really stunned to find out that they can't just write down "the correct answer" on the essay tests.

      Delete
  2. SO, so right, Peter. I teach both high school and community college. As the NCLB era dragged forward, I definitely began to encounter a lot of pushback on open-ended questions. Students always seemed to want to want verification that they had identified and noted the CORRECT information. The trend has worsened. As a history teacher, I am trying to allow my students flexibility in their answers. Even when I say, "There is no single correct answer, but rather, you should sufficiently support your answer," it does little to advance their thinking.

    For a while I thought maybe it was the school system I was in. Perhaps this is what our elementary and middle school teachers teach. After all, they're subjected to BS tests for 6 consecutive years. But when I dealt with a variety of students from across the county in community college, I discovered that this was true for the overwhelming majority of students who have graduated over the past decade. My homework assignment are a combination of brief constructed response and full essay. Students like the class but they vociferously complain about the nature of the homework. When I said, "Would you prefer chapter review questions?", they brightened and nodded. Well, too bad.

    And there is a noticeable divide between my community college students according to age. Students over the age of 30 write excellent answers in most cases. Maybe it's a function of greater life experience but I don't think that is the sole factor.

    ReplyDelete
    Replies
    1. I have noticed the same difference in my community college students. My older, returning students write much better in general. And when I talk to them about their high school education, it turns out that they did more reading and writing and took fewer standardized tests.

      The young students I have now all want multiple choice tests, but I still give them essay tests. Who gives a multiple choice test in a college philosophy class?

      Delete
  3. Another problem with standards is the assumption that mastering them indicates that learning has taken place. Some people are just quick with the bizarre logic of thoses tests. Those same people are often unable to have a meaningful conversation about the topics involved. Another student who is not wired for those tests may have worked his butt off, even if he didn't earn an A. That student may have had an insightful teacher who inspired deep tbought. Neither the standardized test nor the grade reflect how much learning took place. I have always felt that life success is more about self discipline, consistency and a good attitude. You don't have to master academic skills but you certainly need to try. A teacher can recognize each student's unique set of abilities and achievements. Grades, awful as they are, can reflect a student's effort or work ethic, if the teacher has it set up that way.

    Standards, as they are currently defined, are more for widgets and airplane parts. I had a student tell me the other day that she feels like a set of numbers. It was a spontaneous comment. What will become of our society when millions of people spent twelve years feeling like a set of numbers? The people who design these standards and tests did not grow up in that environment. It did not exist until recently. It really is a ghastly social experiment.

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. This comment has been removed by the author.

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. There is a movie currently in release that really struck me in regards to standardized testing. It about social controls that makes people go along with what they know is wrong. It based on an experiment to study this in the early '60's where a "teacher" was instructed to administer ever increasing electrical shocks to a "learner" if they got a wrong answer. The shocks were simulated, but the "teacher" did not know that. The experiment was to see how many people would administer ever more painful shocks if they were instructed to do so. Within the first minute of the movie I thought, "This is about standardized tests." and that feeling never left through the whole movie.

    The movie is Experimenter.

    http://www.imdb.com/title/tt3726704/

    Trailer: https://www.youtube.com/watch?v=O1VOZhwRvWo

    ReplyDelete
  9. It's the same story all over. I am a Michigan science teacher. We were looking at the new Mi science standards last week. You know standards such as "plan and conduct an investigation regarding...." Will quickly turn into "answer these multiple choice questions about this experiment that we designed."

    ReplyDelete