Sunday, February 8, 2015

Sampling the PARCC

Today, I'm trying something new. I've gotten myself onto the PARCC sample item site and am going to look at the ELA sample items for high school. This set was updated in March of 2014, so, you know, it's entirely possible they are not fully representative, given that the folks at Pearson are reportedly working tirelessly to improve testing so that new generations of Even Very Betterer Tests can be released into the wild, like so many majestic lion-maned dolphins.

So I'm just going to live blog this in real-ish time, because we know that one important part of measuring reading skill is that it should not involve any time for reflection and thoughtful revisiting of the work being read. No, the Real Readers of this world are all Wham Bam Thank You Madam Librarian, so that's how we'll do this. There appear to be twenty-three sample items, and I have two hours to do this, so this could take a while. You've been warned.

PAGE ONE: DNA

Right off the bat I can see that taking the test on computer will be a massive pain in the ass. Do you remember frames, the website formatting that was universally loathed and rapidly abandoned? This reminds me of that. The reading selection is in its own little window and I have to scroll the reading within that window. The two questions run further down the page, so when I'm looking at the second question, the window with the selection in it is halfway off the screen, so to look back to the reading I have to scroll up in the main window and then scroll up and down in the selection window and then take a minute to punch myself in the brain in frustration.

The selection is about using DNA testing for crops, so fascinating stuff. Part A (what a normal person might call "question 1") asks us to select three out of seven terms used in the selection, picking those that "help clarify" the meaning of the term "DNA fingerprint," so here we are already ignoring the reader's role in reading. If I already understand the term, none of them help (what helped you learn how to write your name today?), and if I don't understand the term, apparently there is only one path to understanding. If I decide that I have to factor in the context in which the phrase is used, I'm back to scrolling in the little window and I rapidly want to punch the test designers in the face. I count at least four possible answers here, but only three are allowed. Three of them are the only answers to use "genetics" in the answer; I will answer this question based on guesswork and trying to second guess the writer.

Part B is a nonsense question, asking me to come up with an answer based on my first answer.

PAGE TWO: STILL FRICKIN' DNA

Still the same selection. Not getting any better at this scrolling-- whether my mouse roller scrolls the whole page or the selection window depends on where my cursor is sitting.

Part A is, well... hmm. If I asked you, "Explain how a bicycle is like a fish," I would expect an answer from you that mentioned both the bicycle and a fish. But PARCC asks how "solving crop crimes is like solving high-profile murder cases." But all four answers mention only the "crop crime" side of the comparison, and the selection itself says nothing about how high-profile murder cases are solved. So are students supposed to already know how high-profile murder cases are solved? Should they assume that things they've seen on CSI or Law and Order are accurate? To answer this we'll be reduced to figuring out which answer is an accurate summary of the crop crime techniques mentioned in the selection.

This is one of those types of questions that we have to test prep our students for-- how to "reduce" a seemingly complex question to the simpler question. This question pretends to be complex; it is actually asking, "Which one of these four items is actually mentioned in the selection?" It boils down to picky gotcha baloney-- one answer is going to be wrong because it says that crop detectives use computers "at crime scenes"

Part B.The old "which detail best supports" question. If you blew Part A, these answers will be bizarrely random.

PAGE THREE: DNA

Still on this same damn selection. I now hate crops and their DNA.

Part A wants to know what the word "search" means in the heading for the final graph. I believe it means that the article was poorly edited, but that selection is not available. The distractor in this set is absolutely true; it requires test-taking skills to eliminate it, not reading skills.

Part B "based on information from the text" is our cue (if we've been properly test prepped) to go look for the answer in the text, which would take a lot less time if not for this furshlugginer set up. The test writers have called for two correct answers, allowing them to pretend that a simple search-and-match question is actually complex.

PAGE FOUR: DNA GRAND FINALE, I HOPE

Ah, yes. A test question that assesses literally nothing useful whatsoever. At the top of the page is our selection in a full-screen width window instead of the narrow cramped one. At the bottom of the page is a list of statements, two of which are actual advantages of understanding crop DNA. Above them are click-and-drag details from the article. You are going to find the two advantages, then drag the supporting detail for each into the box next to it. Once you've done all this, you will have completed a task that does not mirror any real task done by real human beings anywhere in the world ever.

This is so stupid I am not even going to pretend to look for the "correct" answer. But I will remember this page clearly the next time somebody tries to unload the absolute baloney talking point that the PARCC does not require test prep. No students have ever seen questions like this unless a teacher showed them such a thing, and no teacher ever used such a thing in class unless she was trying to get her students ready for a cockamamie standardized test.

Oh, and when you drag the "answers," they often don't fit in the box and just spill past the edges, looking like you've made a mistake.

PAGE FIVE: FOR THE LOVE OF GOD, DNA

Here are the steps listed in the article. Drop and drag them into the same order as in the article. Again, the only thing that makes this remotely difficult is wrestling with the damn windows. This is a matching exercise, proving pretty much nothing.

PAGE SIX: APPARENTLY THIS IS A DNA TEST TEST

By now my lower-level students have stopped paying any attention to the selection and are just trying to get past it to whatever blessed page of the test will show them something else.

Part A asks us to figure out which question is answered by the selection. This is one of the better questions I've seen so far. Part B asks which quote "best" supports the answer for A. I hate these "best" questions, because they reinforce the notion that there is only one immutable approach for any given piece of text. It's the very Colemanian idea that every text represents only a single destination and there is only one road by which to get there. That's simply wrong, and reinforcing it through testing is also wrong. Not only wrong, but a cramped, tiny, sad version of the richness of human understanding and experience.

PAGE SEVEN: SOMETHING NEW

Here comes the literature. First we get 110 lines of Ovid re: Daedelus and Icarus (in a little scrolling window). Part A asks which one of four readings is the correct one for lines 9 and 10 (because reading, interpreting and experiencing the richness of literature is all about selecting the one correct reading). None of the answers are great, particularly if you look at the lines in context, but only one really makes sense. But then Part B asks which other lines support your Part A answer and the answer here is "None of them," though there is one answer for B that would support one of the wrong answers for A, so now I'm wondering if the writers and I are on a different page here.

PAGE EIGHT: STILL OVID

Two more questions focusing on a particular quote, asking for an interpretation and a quote to back it up. You know, when I say it like that, it seems like a perfectly legitimate reading assessment. But when you turn that assessment task into a multiple choice question, you break the whole business. "Find a nice person, get married and settle down," seems like decent-ish life advice, but if you turn it into "Select one of these four people, get married in one of these four ceremonies, and buy one of these four houses" suddenly it's something else.

And we haven't twisted this reading task for the benefit of anybody except the people who sell, administer, score and play with data from these tests.

PAGE NINE: OVID

The test is still telling me that I'm going to read two selections but only showing me one. If I were not already fully prepped for this type of test and test question, I might wonder if something were wrong with my screen. So, more test prep required.

Part A asks what certain lines "most" suggest about Daedelus, as if that is an absolute objective thing. Then you get to choose what exact quotes (two, because that makes it more complex) back you up. This is not constructing and interpretation of a piece of literature. Every one of these questions makes me angrier as a teacher of literature and reading.

PAGE TEN: ON TO SEXTON

Here's our second poem-- "To a Friend Whose Work Has Come To Triumph." The two questions are completely bogus-- Sexton has chosen the word "tunneling" which is a great choice in both its complexity and duality of meaning, a great image for the moment she's describing. But of course in test land the word choice only "reveals" one thing, and only one other piece of the poem keys that single meaning. I would call this poetry being explained by a mechanic, but that's disrespectful to mechanics.

PAGE ELEVEN: MORE BUTCHERY

Determine the central idea of Sexton's poem, as well as specific details that develop the idea over the course of the poem. From the list of Possible Central Ideas, drag the best Central Idea into the Central Idea box.

Good God! This at least avoids making explicit what is implied here-- "Determine the central idea, then look for it on our list. If it's not there, you're wrong." Three of the four choices are okay-ish, two are arguable, and none would impress me if they came in as part of a student paper.

I'm also supposed to drag-and-drop three quotes that help develop the One Right Idea. So, more test prep required.

PAGE TWELVE: CONTRAST

Now my text window has tabs to toggle back and forth between the two works. I'm supposed to come up with a "key" difference between the two works (from their list of four, of course) and two quotes to back up my answer. Your answer will depend on what you think "key" means to the test writers. Hope your teacher did good test prep with you.

PAGE THIRTEEN: ESSAY TIME

In this tiny text box that will let you view about six lines of your essay at a time, write an essay "that provides and analysis of how Sexton transforms Daedelus and Icarus." Use evidence from both texts. No kidding-- this text box is tiny. And no, you can't cut and paste quotes directly from the texts.

But the big question here-- who is going to assess this, and on what basis? Somehow I don't think it's going to be a big room full of people who know both their mythology and their Sexton.

PAGE FOURTEEN: ABIGAIL ADAMS

So now we're on to biography. It's a selection from the National Women's History Museum, so you know it is going to be a vibrant and exciting text. I suppose it could be worse--we could be reading from an encyclopedia.

The questions want to know what "advocate for women" means, and to pick an example of Adams being an advocate. In other words, the kinds of questions that my students would immediately id as questions that don't require them to actually read the selection.

PAGE FIFTEEN: ADAMS

This page wants to know which question goes unanswered by the selection, and then for Part B asks to select a statement that is true about the biography but which supports the answer for A. Not hopelessly twisty.

PAGE SIXTEEN: MORE BIO

Connect the two central ideas of this selection. So, figure out what the writers believe are the two main ideas, and then try figure out what they think the writers see as a connection. Like most of these questions, these will be handled backwards. I'm not going to do a close reading of the selection-- I'm going to close read the questions and answers and then use the selection just as a set of clues about which answer to pick. And this is how answering multiple choice questions about a short selection is a task not much like authentic reading or pretty much any other task in the world.

PAGE SEVENTEEN: ABIGAIL LETTER

Now we're going to read the Adams family mail. This is one of her letters agitating for the rights of women; our questions will focus on her use of "tyrant" based entirely on the text itself, because no conversation between Abigail and John Adams mentioning tyranny in 1776 could possibly be informed by any historical or personal context.

PAGE EIGHTEEN: STILL VIOLATING FOUNDING FATHER & MOTHER PRIVACY

Same letter. Now I'm supposed to decide what the second graph most contributes to the text as a whole. Maybe I'm just a Below basic kind of guy, but I am pretty sure that the correct answer is not among the four choices. That just makes it harder to decide which other two paragraphs expand on the idea of graph #2.

PAGE NINETEEN: BOSTON

Now we'll decide what her main point about Boston is in the letter. This is a pretty straightforward and literal reading for details kind of question. Maybe the PARCC folks are trying to boost some morale on the home stretch here.

Oh hell. I have a message telling me I have less than five minutes left.

PAGE TWENTY: JOHN'S TURN

Now we have to pick the paraphrase of a quote from Adams that the test writers think is the berries. Another set of questions that do not require me to actually read the selection, so thank goodness for small favors.

PAGE TWENTY-ONE: MORE JOHN

Again, interpretation and support. Because making sense out of colonial letter-writing English is just like current reading. I mean, we've tested me on a boring general science piece, classical poetry, modern poetry, and a pair of colonial letters. Does it seem like that sampling should tell us everything there is to know about the full width and breadth of student reading ability?

PAGE TWENTY-TWO: BOTH LETTERS

Again, in one page, we have two sets of scrollers, tabs for toggling between works, and drag and drop boxes for the answers. Does it really not occur to these people that there are students in this country who rarely-if-ever lay hands on a computer?

This is a multitask page. We're asking for a claim made by the writer and a detail to back up that claim, but we're doing both letters on the same page and we're selecting ideas and support only from the options provided by the test. This is not complex. It does not involve any special Depth of Knowledge. It's just a confusing mess.

PAGE TWENTY-THREE: FINAL ESSAY

Contrast the Adams' views of freedom and independence. Support your response with details from the three sources (yes, we've got three tabs now). Write it in this tiny text box.

Do you suppose that somebody's previous knowledge of John and Abigail and the American Revolution might be part of what we're inadvertently testing here? Do you suppose that the readers who grade these essays will themselves be history scholars and writing instructors? What, if anything, will this essay tell us about the student's reading skills?

DONE

Man. I have put this off for a long time because I knew it would give me a rage headache, and I was not wrong. How anybody can claim that the results from a test like this would give us a clear, nuanced picture of student reading skills is beyond my comprehension. Unnecessarily complicated, heavily favoring students who have prior background knowledge, and absolutely demanding that test prep be done with students, this is everything one could want in an inauthentic assessment that provides those of us in the classroom with little or no actual useful data about our students.

If this test came as part of a packaged bunch of materials for my classroom, it would go in the Big Circular File of publishers materials that I never, ever use because they are crap. What a bunch of junk. If you have stuck it out with me here, God bless you. I don't recommend that you give yourself the full PARCC sample treatment, but I heartily recommend it to every person who declares that these are wonderful tests that will help revolutionize education. Good luck to them as well.



24 comments:

  1. I had a headache after only 30 minutes with the Grade 4 ELS sample. I would have likely needed a new desktop computer after attempting what you just did, because I would have ended up doing it grievous bodily harm.

    ReplyDelete
  2. Thanks for taking one for the team Peter. My principal wanted us to prep our students, but fortunately I do not have computers in my classroom. When they tried the pilot of the SBAC with our lower grades, students were in tears, they were frustrated, and many of them just plain quit. That was last year, this year we are all taking them. I hope our computers burn and our server crashes...we have very slow service and a narrow bandwidth. This should be fun. I think it's time to find a new job and offer tutoring in reading. I have an M.A. in linguistics, we older types will be the last ones that know how to teach humans, as opposed to monitoring them on computers.

    ReplyDelete
  3. We took the 4th grade ELA sample test at a staff meeting in October. It was incredibly frustrating. I ended up skipping all the open response questions because at that point I was so aggravated thinking about our poor 4th graders being subjected to it that I no longer gave a crap. As a school counselor, I'm anticipating having plenty of business during testing this year.

    The only bright spot was that one of the selections was a story my younger sister had published in Highlights magazine a few years ago. That part was pretty cool.

    ReplyDelete
  4. We took the sample third grade ELA performance task in a staff meeting on Chromebooks (which were purchased withe RttT money specifically for testing purposes) and imagine doing the test on an even smaller screen. I also get to have a substitute during all of the ELA PARCC days in my school because I am needed for "tech support" in the classes that will be using the Chromebooks. So, all of my primary grade students will not see me for two weeks because I am needed for tech support that there is no training for, so my presence will be for naught.

    ReplyDelete
    Replies
    1. OMG, scrolling on the tiny Chromebook screens with a trackpad just about sent me over the edge. Partly why it took me so long to get as far as I did!

      Delete
  5. I think my students would have done me great bodily harm if I had forced them to take such an inappropriate and useless "assessment."

    ReplyDelete
  6. Maine is using Smarter Balanced (Assessment Consortium or SBAC) and the issue here is iPads, not computers, since many schools bought into that "solution" two years ago. Last year, my husband's class (grades 6/7/8 in one classroom) piloted the SBAC test and finally gave up on using the iPads because there were so many glitches, punted to the leftover MacBooks from the previous "solution" (which some of the students had never really used) and managed OK. This year, as the state heads into its first formal SBAC testing cycle, the Maine Department of Education just this week issued an advisory* that "keyboards are no longer required, only recommended" even though the SBAC has said otherwise. Should be interesting.

    [http://mainedoenews.net/2015/02/06/maine-doe-recommends-rather-than-requires-assessment-keyboards/]

    ReplyDelete
  7. My poor little fifth graders are being subjected to five rounds of standardized testing this year. Three NWEA MAP math assessments (allegedly only for placement into middle school math.... but oh, yes, also for our formative and summative assessments, as if we teachers don't even know the kids we've spent 600 hours with so far this year...), the SBACC and some leftover STAR science test. Five #@$%! rounds of standardized crappola. So much wasted time and resources, it sucks almost all the joy out of teaching. And it certainly sucks out a lot of joy-in-learning for the students, too.

    ReplyDelete
    Replies
    1. With you on this one. I also teach 5th grade. My students will face the NWEA MAP in three subjects three times during the year, the STAR reading and math at least three times during the year, the SBAC for over 14 hours in the Spring, and the ISAT science test. 14 years of loving my job and then there is this year...

      Delete
  8. I took the sample tests when they first came out and could have written almost the same blog but with more expletives. I have children subjected to these tests but they do better than I do thanks to all of their online test prep. I have lived with these tests for several years when my children had to do test prep at home using Study Island. One day when my son got particularly frustrated I tried to help him. There is a special kind of torture they do when you miss an item where they use and algorithm to add questions to make sure you cry. Since he has to answer questions until he gets 80% correct, if he gets a few wrong in a row that sometimes is a lot a questions to go through. I remember one night I sent him to bed and stayed up a couple more hours trying to get him to pass (torture). I then realized the insanity of it and told his teachers to quit assigning him Study Island. Unfortunately, he learned to like them as a challenge. He thinks it is funny to try and figure out what the test people want for answers- but he is very aware that it has nothing to do with learning and that their right answer is often not the one he would typically chose and make a case for. So my children have learned that their is authentic learning, and then their is a weird artificial test learning that they only use for standardized tests. I hope the sheer stupidity of the task makes them career and college ready!

    ReplyDelete
    Replies
    1. There have been a few posts by people who've taken sample tests; inspired me to do one after our elementary school's sparsely-attended PARCC Night: http://crunchyprogressiveparentingtwo.blogspot.com/2015/01/i-took-practice-parcc-and-i-liked-it-not.html

      And just so it's not just Peter and me, here's another, from Sarah Blaine: http://parentingthecore.com/2014/12/07/nj-governors-task-force-take-the-parcc-i-did/

      Delete
  9. Peter-I love your blog. You also play trombone-yes!

    I teach physics in an Indiana high school, so I am in the heart of reformy land. We have rejected CCSS, but I can't see where our standards are much different.

    I looked at the PARCC Math test, as math is the other language I speak when teaching. I am gobsmacked. My overall impression is that most of my kids would have no idea what to do on topics of data analysis which dominates this test. I get this sense because I have to teach them data analysis in my class; few come to me with any idea of fitting real data curves. Holy shite.

    The initial barrier is that one must use an equation editor to answer many of the parts. Equation editors are not only a royal pain, they are a tremendous time suck. It seems stupid to require a computers-are-dumb-as-a-box-of-rocks workaround when you actually want to know math skills. On timed tests, isn't a precious thing, well, time?

    Other issues I take with it are the ordering of the question parts. Prompts for answers appear before the request to create the graph upon which the questions are based. The questions are wordy. Your ELA DNA reading was an easy read, while the popcorn question on the math portion was so dense (no, the cute little picture and the incomplete data table were no help at all) that I gave up on it. The Chinese study is a beaut, asking kids to delineate what conclusions can be drawn from a statistical data set, which goes to the question of causation versus correlation. After being subjected to so much abuse as a teacher based on really sloppy conclusions drawn from data analyzed by econometricians (Chetty et al, anyone?), I had to laugh. Kids are being held to more rigorous standards than those who are planning to test the bejeezus out of them.

    On the other hand, when I have a lot of spare time, I think I would have fun with most of these questions. I doubt the same would be true of most of my kids. I would be interested in asking their opinion, perhaps after the AP exam. I love teaching teens because they still can be pretty frank.

    ReplyDelete
  10. If I had a PARCC sanctioned rubric for assessing takedown's of PARCC sample items, I am confident that your work would earn "Exemplary" status in every category.

    Since looking at the Ovid/Anne Sexton sample they released a few years back, I have often thought: "This company knew that they have billions...BILLIONS...of dollars to make from their tests and this is the best they could do to recruit buyers for their assessments?!" Give me just 1 billion dollars and a month and I will happily produce tests (even within the restrictions of PARCC's lousy computer-based format) that would make English teachers around the country think: "These aren't perfect, but these are tests that are asking my students to do the kinds of thinking and analysis and synthesizing of ideas/sources that have merit. And the sources/reading they are asked to work with are not of the sort that will lead many students to shut down within ten minutes of taking the test."

    ReplyDelete
  11. Refuse the tests. Don't make your kid have to participate in this illegal testing scheme! If enough parents refused for their kids to take these assessments we could crash their whole model for assessment going forward. We have to stop going along to get along with this madness. Time to rise up.

    ReplyDelete
    Replies
    1. Toni....you couldn't be more correct. STARVE THE BEAST is the only answer at this point. They have billions of $$$$$ but we have what they want....the kids to supply them with their heroin fix of data. Parents must stand up and REFUSE the NAEP and national tests like PARCC, Smarter Balance and whatever crap name they want to put on it. People also need to know that hiding behind any assessment company your state hires is either PARCC or AIR/Smarter Balance. TN hired Measurement Inc and they contract their work to AIR/Smarter Balance. These people should be sued for civil rights violations.

      Delete
  12. You know, there's something fishy about that DNA reading. You might track the DNA on a crop if Monsanto is looking to pick up royalties for contaminating farmers' seed lines, but not for "crop theft". This very cogent article recommends taking the actual thumbprint of the driver that picks up your crop, instead.
    http://agalert.com/story/?id=5583

    ReplyDelete
  13. Sounds like the same BS offered as "rigor" on the Pearson created Virginia SOL reading test. And SOL doesn't stand for what it usually does unless you are a student measured by this junk. http://walkingtoschool.blogspot.com/2014/04/in-belly-of-oligarchy.html?m=1

    ReplyDelete
  14. OMG, Peter! I'm shipping you a wine box from the Edushyster's private stock ASAP. After PARCCing you deserve to drink it all in one sitting.

    ReplyDelete
  15. I took the 4th grade ELA SBAC. It was awful - for all the reasons you cite. (I ranted about these too: http://gladlyteche.blogspot.com).

    It's not just that the tests don't measure reading comprehension - no standardized test does. You can't test for comprehension or knowledge directly, the way you can look at cholesterol. You can only ask students to do things that (presumably) can only be done if they do know or understand things. It's hard to fake this kind of performance; and frankly, if you can fake it perfectly, then perhaps you really do understand it - witness that boy who passed the Louisiana bar exam without ever having studied law.

    What bothers me is that these tests teach students to read in bizarre, fake, headache-inducing ways that yield nothing. It doesn't help students see how enjoying literature helps stimulate ideas, and it sure doesn't show them how to enjoy it. It kind of breaks your heart, doesn't it? In five years, I'm going to be spending my time in the community college classroom trying to un-teach all this crap. (Kids who've been to private school, or who have highly literate parents, will of course already have had their separate instructions.)

    Loathsome and convoluted as the SBAC/PARCC tests are, the root of the evil here is the multiple choice question. The International GCSE doesn't use them; its reading comprehension exam asks students to read an article by an actual writer, and then to answer questions which resemble the kinds of questions that we really do ask of people who've read something we haven't: What's this about? No kidding - a trip to Peru? What did the writer like best? What did she mean by that? Why does she say she won't go back?" And so on. All answers are just that, actual written answers that extend over several sentences. But then, there's no $$ in that kind of test.

    ReplyDelete
  16. I addressed this comment to RI Education Commissioner Deborah Gist on a post of hers on fb: Deborah A. Gist, Please read this blog post and tell me honestly and with integrity if the PARCC is a high quality assessment enabling us to see if schools are doing all they can to help students succeed:
    http://curmudgucation.blogspot.com/20…/…/sampling-parcc.html

    This is her response to me: Sheila, I would caution you to rest your opinions about PARCC based on someone who is a self-proclaimed "grumpy old teacher." I have also taken PARCC samples and have seen the amazing work of the teachers and other educators who have been working on this development for several years now. This is an exciting step in the right direction.

    ReplyDelete
  17. Google Anita Hoge. Read her research and WEEP! She lays out where we've been, where we are and where we are heading. It'll make you sick.

    ReplyDelete
  18. Thanks for carrying on the great American tradition of muckraking, Peter. The fraudulence of these "smarter" tests needs to be exposed. Sadly few teachers have the intellectual self-confidence to attack them.

    ReplyDelete
  19. Hi...........
    Thanks for sharing nice blog post Sampling Staff

    ReplyDelete
  20. As someone who has taught math from middle school to Ph.D. students and spent a career since writing software, my first reaction on seeing the equation editor is WHAT THE HELL? So, we waste class time teaching students to use a tool that they will never use except to take this test. Expletives are very warranted.

    ReplyDelete