Sunday, March 15, 2015

Why Critical Thinking Won't Be on The Test

Critical thinking is one of the Great White Whales of education. Every new education reform promises to foster it, and every new generation of Big Standardized Tests promises to measure it.

Everybody working in education has some idea of what it is, and yet it can be hard to put into a few words. There are entire websites devoted to it, and organizations and foundations dedicated to it. Here, for example, is the website of the Foundation for Critical Thinking. They've got a definition of critical thinking from the 1987 National Council for Excellence in Critical Thinking that goes on for five paragraphs. One of the shortest definitions I can pull out of their site is this one:

The intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action.

Bottom line-- critical thinking is complicated.

So can we believe test manufacturers when they say that their test measures critical thinking skills? Can a series of questions that can be delivered and scored on a national scale be designed that would actually measure the critical thinking skills of the test takers?

I think the obstacles to creating such a standardized test are huge. Here are the hurdles that test manufacturers would have to leap.

Critical thinking takes time.

Certainly there are people who can make rapid leaps to a conclusion, who can see patterns and structure of ideas quickly and clearly (though we could argue that's more intuitive thinking than critical thinking, but then, intuition might just be critical thinking that runs below the level of clear consciousness, so, again, complicated). But mostly the kind of analyses and evaluation that we associate with critical thinking takes time.

There's a reason that English teachers rarely give the assignment, "The instant you finish reading the last page of the assigned novel, immediately start writing the assigned paper and complete it within a half hour." Critical thinking is most often applied to complex constructions, and for most people it takes a while to examine, reflect, re-examine and pull apart the pieces of the matter.

If you are asking a question that must be answered right now, this second, you are at the very best asking a question that measures how quickly the student can critically think-- but you're probably not measuring critical thinking at all.

Critical thinking takes place in a personal context.

We do not do our critical thinking in a vacuum. We are all standing in a particular spot in space and time, and that vantage point gives us a particular perspective. What we bring to the problem in terms of prior understanding, background, and our own mental constructs, profoundly influences how we critically think about any problem.

We tend to make sense out of unfamiliar things by looking for familiar structures and patterns within them, and so our thinking is influenced by what we already know. I've been an amateur musician my whole life, so I can readily spot structures and patterns that mimic the sorts of things I know form the world of music. However, I am to athletics what Justin Bieber is to quantum physics, and my mental default is not to look at things in athletic terms. Think about your favorite teachers and explainers-- they are people who took something you couldn't understand and put it in terms you could understand. They connected what you didn't know to what you did know.

None of this is a revolutionary new insight, but we have to remember that it means every individual human beings brings a different set of tools to each critical thinking problem. That means it is impossible to design a critical thinking question that is a level playing field for all test takers. Impossible.

Critical thinking is social.

How many big critical thinking problems of the world were solved single-handedly by a single, isolated human being?

Our sciences have a finely-tuned carefully-structured method for both carrying on and acknowledging dialogue with the critical thinkers of the past. If a scientist popped up claiming to have written a groundbreaking paper for which he needed no citations nor footnotes because he had done it all himself, he would be lucky to be taken seriously for five minutes. The Einsteins of history worked in constant dialogue with other scientists; quantum theories were hammered out in part by dialogue by a disbelieving Einstein ("God does not play dice") and the wave of scientists building on the implications of his work.

On the less grand scale, we find our own students who want to talk about the test, want to compare answers, want to (and sometimes love to) argue about the finer points of every thinking assignment.

Look at our own field. We've all been working on a big final test question-- "What is the best way to take American education forward?"-- and almost everyone on every side of the question is involved in a huge sprawling debate that sees most of us pushing forward by trying to articulate our own perspective and thoughts while in dialogue with hundreds of other thinkers in varying degrees of agreement and disagreement. One of the reasons I trust and believe David Coleman far less than other reformsters is that he almost never acknowledges the value of any other thinker in his development of Common Core. To watch Coleman talk, you would think he developed the entire thing single-handedly in his own head. That is not the mark of a serious person.

Do people occasionally single-handedly solve critical thinking problems on their own, in isolation, like a keep-your-eyes-on-your-own-paper test? It's certainly not unheard of-- but it's not the norm. If your goal is to make the student answer the question in an isolation chamber, you are not testing critical thinking.

Critical thinking is divergent.

Let's go back to that critical thinking problem about how to best move forward with public education. You may have noticed that people have arrived a wide variety of conclusions about what the answer might be. There are two possible explanations for the wide variety of answers.

The first explanation is the childish one, and folks from both sides indulge in it-- people who have reached a conclusion other than mine are some combination of stupid, uninformed, and evil.

The more likely explanation is that, given a wide variety of different perspectives, different histories, and different values, intelligent people will use critical thinking skills and arrive at different conclusions.

Critical thinking is NOT starting with the conclusion that you want to reach and then constructing a bridge of arguments specifically designed to get you there, and yet this is perilously close to the kind of thinking a standardized test requires.

But here's a good rule of thumb for anyone trying to test critical thinking skills-- if you are designing your assessment and thinking, "Okay, any student who is really using critical thinking skills must come up with answer B," you are not testing critical thinking skills. No-- I take that back. Oddly enough this is a sort of critical thinking question, but the actual question is, "Given what you know about the people giving you the test and the clues they have left for you, what answer do you think the testmakers want you to select?" But that is probably not the question that you thought you were asking. As soon as you ask a question with one right answer (even if the one right answer is to select both correct answers), you are not testing critical thinking.

Critical thinking must be assessed by critical thinking.

How do you assess the answer to your critical thinking question? Again, I direct you to the education debates, where we "grade" each others' work all the time,  checking and analyzing, probing for logical fallacies, mis-presentation of data, mis-reading of other peoples' writing, honesty of logic, etc etc etc.

To assess how well someone has answered a critical thinking question, you need to be knowledgeable about the answerer, the subject matter, and whatever background knowledge they have brought to the table (if I answer a question using a music analogy and you know nothing about music, will you know if my analogy holds up). On top of all that, you need some critical thinking skills of your own. And that means all of the issues listed above come back into play.

What are the odds that you can get all that in a cadre of minimum-wage test-scorers who can wade through a nation's worth of tests quickly, efficiently, and accurately?

Can it be done?

When I look at all those hurdles and try to imagine a nationally scaled test that gets deals with all of them, I'm stumped. Heck, it's a challenge to come up with good measure for my own classroom, and that's because critical thinking is more of a tool than an end in itself. Testing for critical thinking skills is kind of like testing for hammering skills-- it can be done, but it will be an artificial situation and not as compelling and useful and telling as having the student actually build something.

So I try to come up with assessments that require critical thinking as a tool for completion of the assignment. Then I try to come up with the time to grade them. Could I come up with something for the entire nation? Practically speaking, no. Even if I get past the first few hurdles, when I reach the point that I need a couple million teachers to score it, I'm stumped. Plus, standardized test fans are not going to like the lack of standardization in my test.

No, I think that standardized testing and critical thinking are permanently at odds and we'd be further ahead trying to develop a test to compare the flammability of the water from different rivers.

Critical thinking is not on the BS Tests. It will not be on the new generations of the BS Tests. It will never be on the BS Tests. Test manufacturers should stop promising what they cannot hope to deliver.


  1. Requiring students to think critically pre-supposes that they will probably NOT come up identical answers or solutions. That's how scientists, mathematicians etc come up with new ideas and discoveries, by NOT coming up with the SAME answer! The best test of critical thinking requires questions that would necessarily result in different answers and would require students to justify their answers. What teacher has time for much of that nor do we have the authority to create our own grading system that would satisfy parents who believe an "A" is important and represents learning.

    1. I don't see so much the problem of time or one teacher's grading system in teaching the skills, the problem is trying to do this in the form of a standardized test.

  2. So, I was thinking about the question you posed here: basically, is it possible to have a worthwhile standardized test that measures critical thinking? I think there might be some ways to do it. Maybe a test where the basic topic or question is released a year ahead of time (or maybe is even set as a constant from year to year): something like, in eighth grade, the topic is structures of power, and in ninth grade, effects of empathy, etc, etc. Big, philosophical ideas that take a lot of thought and research and discussion. At the end of the year, the kids have a one-question test on that topic, and they have several days and the ability to consult a number of resources (books or something- probably not teachers, so you don't get teachers writing the essay for the kid).

    It would be very difficult to grade, sure, but if it's one essay, I actually think it wouldn't be that bad- with a rubric, it couldn't be much more difficult than the current assessment my kids take with multiple open response questions. I'm not sure the results would be different, but at least the schools and classes that "teach to the test" would be doing something worthwhile.

    In short, If we *must* feed the data goblins (which I think we must), I think this would be better than the current setup.

    1. Well, yes, but what you just described is exactly the opposite of a standardized test.

  3. Imagine a classroom where the students are asked to actively and skillfully conceptualize, apply, analyze, synthesize, and/or evaluate what goes on in their school...

  4. I was inspired by Kathryn Schulz' comment in her fantastic book Being Wrong: "Doubt, it seems is a skill - and one that .. needs to be learned and honed. Credulity, by contrast, appears to be something very much like an instinct."

    Now I teach critical thinking as a way of learning and honing one's capacity to doubt. Doubt is great stuff; as you've said, Peter, we live in an atmosphere of marketing baloney - intelligent skepticism is as necessary now as under the pre-Reformation Catholic Church.

    As for "tests" to gauge this habit of mind, BOLLOCKS. There is no way to "test" for critical thinking ability, because
    (a) there's no such thing. Critical thinking is not a specific skill, like speaking French or building bridges. It's the name we give to people who seem to be engaging with the world in some useful way. You can't teach a "skill" that isn't a skill. You might as well talk about testing people for satisfaction.
    (b) Critical thinking, as researchers have pointed out, depends a lot on knowledge. Scientists can be complete idiots about foreign policy. A kid can be completely fooled by specious claims made by a cosmetics company, but sharp-eyed and insightful about a new baseball player's prospects. You can't think "critically" in a vacuum.
    (c) no standardized mc test can tell you what's in a student's head.

    What you *can* do is ask students to perform tasks - write essays, build things, whatever - then judge whether their work *looks like* the work product of people who are engaging usefully with the world, and who know a thing or two. This is literally ALL you can do. It's an exam, not a test, and it focuses on performance, not in intrinsic (and indefinable) skills. This is a seriously huge difference.

    Indeed, one of the problems I have with US education is that we have set these immeasurable "skills" (a.k.a. personality traits or habits of mind) as our educational goals. Colleges have defined learning outcomes for their Gen Ed courses like "students will apply critical thinking skills to a variety of problems" - but never things like "Students will understand the basic principles of physics, chemistry and biology and use these to judge the merits of claims made about the natural world." I can't make people think critically. But I can guide their gradual accumulation of knowledge, and as they learn more, nudge them into conversations about it. What they do with this is out of my hands.

  5. Critical thinking isn't something we can test for in a standardised way.

    Aspects of critical thinking in specififc domains might be something we can test for aspects of.

    Several commenters seem to be taking the example of scientific innovation as a template for ctirical thinking that you canlt test for, or that;s unpredictable and below the radar.

    Unpacking scientific innovation as a practice might help here. Like critical thinking generally, it;s critical thinking in a specific domain. And it's knolwedeg dependent. There are other things in the mix, which may alter and change. But knolwedge is something that tends to remain a stable core.

    In science, innovation tends to rest firmly on scientific method, rigor, and deep knowledge.

    If the question is, can we test for critical thinking, then the answer is hugely problematic. But if we ask, can we test for citical thinking in this specific domain, then we have a simoler, and more true to life proposition.

    It;s true to life, becasue critical thinking, good, deep critical thinking is usually domain specific to a degree, or a high degree. It;s simpler, becasue that domain specific aspect means we can target specific knolwedges.

    Testing for knolwedge won;lt tell you if the person is thinking critically in that domain, but it will tell you if they have one of the primary keys in place.

    Newtonsl quote about standing on the shoulders of giants riffs on this. You don;t get innovation and insight without a deep knolwedge of your field. Testing how steady our students are on those shoulders will give us some insight into how prepared they are to take flight.

    1. I agree the better and deeper your knowledge base is, the better you can apply critical thinking to a domain, but knowledge, though necessary, is not critical thinking. And sometimes you can even solve problems better if you apply knowledge/concepts from an entirely different domain.

    2. I think we agree on a lot here. I'd argue that knolwedge is key to most critical thinking, and that we generally need to focus, as a result, on teaching critical thinking in something.

      I'd also argue that, yes, you of course can import knolwedge from other disciplines or experiences to help solve issues in a particular domain, or provide answers that might not be otherwise available.

      But in that context, you are still, probably, relying on domain knolwedge anyway.

      You might import an idea from physics into, say, musical instrument design to solve an issue you are having with making a aprticular instrument. You still need to know how to make instruments to imporet the knoweldge usefully.

      Knolwedge is not critical thinking. But, usually, it is implicit, and indivisable from critical thinking. It's not the end of the road, or the entire journey, but it could be thought of as a precursor skill.

      It;s not the thing itself - it;s not critical thinking. But it is often a precondition.

      Talking generally about developing critical thinking is not that useful. Talking about developing critical thinking in something is very useful.
      And developing critical thinking in something involves factual knolwedge as a huge part of that journey.

      And factual knowledge is testable.

      What we do with that knowledge, and how we engage it, use it, activate it and challenge it is key.

    3. I think we do mostly agree. All I'm saying is you can't test critical thinking itself on a standardized test. And rather than testing knowledge as content - because that depends on what the teacher decides to teach and how to teach it and there can be disagreements here - it seems like teaching should be based on concepts or principles (laws of nature) though this may also depend on the subject matter and I don't see how you can test that well on a standardized test either.

    4. Keith has hit the nail on the head: "Knowledge is not critical thinking. But, usually, it is implicit, and indivisible from critical thinking."

      What is "knowledge?" Not just parrot repetition of facts - and no one, at least in the West, has ever believed that it is. A large part of philosophy is dedicated to the idea that the statement "Yeah, I know" in fact represents a complex tangle: of observation, deductions, and belief systems. It's not just mindless repetition, which no one advocates in schools. You can't "know" something without having drawn some conclusions, or at least, grasped how someone else drew conclusions.

      Postman (whom I'm re-reading) argued that all subjects should be taught as history, which strikes me as a GREAT idea. You can't learn *any* history - of science, politics, gender roles, nutrition, music, language, agriculture, *whatever* -- without encountering, over and over, the spectacle of people figuring stuff out, overturning previous dogma, and making new realities. History is a Thanksgiving Day Macy's parade of critical thinking.

      You can't test for critical thinking. I'm not sure you can exactly test for knowledge either. Someone (don't remember who) pointed out that when we really know things, we don't forget them. And if you forget your test-knowledge the minute you disgorge it for the BST, you never really knew it.

      You can, however, conduct examinations for both. That's what professions do - law exams, medical boards etc. never take the form of standardized tests, but of performances.

    5. This has been an informative exchange of ideas. Each of you have made me realize that I need to rethink what I thought I knew about critical thinking because I had not considered how difficult it would be to actually measure one's critical thinking skills and abilities. I appreciate your comments.

    6. Madeleine, I totally agree with teaching everything as history. I've always thought that the only way I will ever understand math is to go through it the way it was discovered or invented. When we had to use the quadratic formula, all I could think was " What do I care about plugging numbers into a formula? What I want to know is how someone figured this out!

  6. Critical thinking happens when one is able ask oneself questions (on whatever topic) which lead to a search for understanding beyond one's existing knowledge (schema). I've found that a great way to explore "critical thinking" skills is to challenge students to generate substantive questions on some topic. I'm not suggesting this as an empirical testing method, but it is a surprisingly difficult challenge for them. It takes them from the role of passive learner, trying to absorb someone else's constructed knowledge to the role of a self-activated navigator of learning. Currently my middle school students are comparing the inaugural speeches of presidents from different eras. If they had read them in isolation, the meaning would not have been significant, but by reading three speeches, paragraph by paragraph (the first paragraph of each speech, then the second, etc.), the contents and form become noticeably distinct and more meaningful. I must mention that the kids were skeptical about these speeches when I pulled them out, and now they cannot wait to continue doing this with three other presidents. They were completely enthralled. It is much easier to analyze each speech in this format. There is no way to test empirically what they learned, but if someone wants to know if they recognize what aspects to compare, the students would be able to do that. Personally, I don't consider that an example of mastering critical thinking. Can they generate questions for follow up? Like pretty much everything else that is not discrete or empirical, testing critical thinking is quite sticky and very likely not worth it.