FEE & the Honesty/Proficiency Gap

Oh, how quickly the talking points pass by.

Just a month ago, we were treated to the Honesty Gap, a gap that was revealed by comparing the percentage of students who beat the NAEP cut score to the number who hit the cut score for states. This is a not-very-valid comparison for any number of reasons, but to keep things brief (unless you're a link follower), I'll just mention one.

A 2007 NCES study followed students who had taken the NAEP and discovered that of those who had scored "basic"-- that would be "not proficient" and therefor "not ready for college or career" according to the current gappy discussion-- about 50% of those not-ready-for-college students successfully completed college. So right off the bat, saying that only students who made the "proficient or better" cut on the NAEP are proficient enough for college-- that statement would appear to suffer from an accuracy gap.

Tell us more!

Be that as it may, Honesty Gap was going to be the hot new buzz term. And then it wasn't. The initiative, backed by the Fordham and the Chamber and other of the usual crowd, even had its own hashtag. But now #honestygap hasn't been used supportively by anyone since the 23rd, and then five days before that, and...well, about sixty times total in all of June. For comparison, #Ilikepie has forty-two June mentions and #beiberdefensesquad has about thirty in the last twenty-four hours.

But fear not! The Foundation for Excellence in Education has stepped in. You will recall that FEE is the school privatization advocacy group that was run by Jeb Bush and occasionally launches new PR blitzes to varying degrees of effect or occasionally announces another piece of the sky falling (and only privatizing education can help). Bush actually stepped away from official leadership of the group, handing the com over to Condoleezza Rice so that he could try for the Bush Oval Office hat trick. I have actually wasted a chunk of time sitting here scouring the interwebs for something-- anything-- that Rice has done with the office after rising to it, but I cannot find a thing. All of the heavy lifting (and FEE comes up with some really large piles of shtuff to lift) is still being done by Patricia LeVesque, who has her own special brand of Umbridgian baloney unloading style.

So, anyway. FEE has renamed the gap-- it is no longer an Honesty Gap, but a Proficiency Gap!

Like the Honesty Gap before it, Proficiency Gap gets its own website. And it's here that we'll learn everything we could want to know. Well, almost everything. Let's travel through the five informative, slick, definitely not part of the free blogspot layout package, screens.

Let's define our terms.

Being proficient means a student has demonstrated mastery of the subject matter, including subject matter knowledge, application of such knowledge to real-world situations, and associated analytical skills.

Well, that sounds like a useful thing to know, as well as an impossible thing to measure. But let's just remember this-- that the tool we're using to measure this is a standardized test covering reading and math. When we say the student has "demonstrated mastery of the subject matter," the only subjects that we're talking about are reading and math. That's it.

Do you want to determine if young Chris is ready to major in music, study biology, or become a welder? Too bad-- none of those things are covered by the instruments that we are going to pretend measure proficiency. And that's just the issues we have before we even get to discussing what "mastery" means or what kind of 'real world situations" we're talking about or what's implied by "associated analytical skills."

But let's not forget-- what "proficiency" actually means is "high-enough test score on a single standardized test covering math and reading."

Why do we measure it?

To have an honest, objective benchmark of what a child is learning to ensure that every student is prepared for success in college, a career or the military.

FEE isn't going to let the "honest" thing go entirely, because reformsters are attached to the notion that the public school system is founded on lies and deception. I am impressed, however, that a benchmark exists that would allow us to know with certainty that a student is ready for those things. That would be awesome. Every college, prospective employer, and branch of the military could use it and be guaranteed that they would never, ever, accept/hire/enlist someone who couldn't cut it ever again. That is awesome news.

Boy, oh boy-- I just hope the next slide tells us HOW we are able to pull off such a difficult, complex benchmarking thingy.

What does proficiency cut score mean?

Oh, disappointment.

A proficiency cut score is an actual number (score) on an assessment that draws the line determining where a student is proficient. 

FEE would like you to know that some states draw the line too low, giving students a false sense of confidence when they actually suck and their teachers are big lying liars. There's a cool graphic showing a Greek column on which state score and NEP score lines have been drawn at different heights. Boo, state line drawers!

There is also an option to draw up the (beautifully rendered) stats for your state, so you can see how badly you're being lied to.

What is the issue with proficiency measurement?

If you guessed "that we have no idea how to measure proficiency," the BRRZZT sorry, but you failed. The issue is (somewhat redundantly) that states and NAEP define proficiency differently.

Therefore, state-reported proficiency is not equivalent to proficiency on NAEP. This is referred to as the “proficiency gap”.

Now, I might have called that a "testing gap" or a "test design gap" or just plain "test score" gap or even "proof that the state tests are crap gap" or even possibly "proof that cut scores are arbitrary and don't really reflect a damn thing" which is not technically a gap, so I'll deduct three style points from myself.

Why does proficiency matter?

Here come some factoids. Too many students do poorly on the Armed Forces Qualifications Test. There are too many manufacturing jobs begging for qualified applicants. Too many ACT test-takers came up as Not College Ready. And too many students have to spend a ton of money on remedial college courses.

Remember-- proficiency is "a high-enough test score on a single standardized test covering math and reading."

And yet-- if our students just had higher PARCC or SBA or Various Mongrel Test scores, that would make them ready for the army, ready to be welders, and never needing a remedial course whether they attended Harvard University or the University of Southern North Dakota at Hoople. Just two scores-- one math and one reading-- can tell us all that. My God-- but we live in a magical age!!

What question did we not ask?

We never did ask or answer how we determine proficiency. I don't mean how we set cut scores (though we didn't really answer that, either) but how do we determine whether a student is proficient or not? How do we measure it? Apparently the NAEP folks know exactly how to do it, so what's their secret? How do we determine that a student is ready to study at any college in the country or do any job in the country or serve in any branch of the military? I've plumbed the mysteries of proficiency before:

What could it even mean to call someone a proficient reader? Does it mean she can finish an entire novel? Does she have to understand it? Does she have to finish it in less than a month? A week? A year? Can it be any novel? Does it have to be a modern one, or can it be a classic? If I can get through The Adventures of Huckleberry Finn but not Moby Dick, am I still a proficient reader? If I read Huck Finn, but I just think it's a boy's adventure novel, and I proficient, or do I have to grasp the levels of satire to be proficient? Must I also be able to see symbolism tied to the search for identity in order to be proficient? What about poetry? Does someone have to be able to read poetry to be proficient? Any poetry? From any period? Is a proficient reader moved by what she reads, or does reading proficiency have to do only with the mechanics and thinky parts? And should proficient reader be able to read and follow instructions, say, for assembling a new media center? Would a proficient reader be able to follow the instructions even if the writer of the instructions was not a proficient English language writer? Can a proficient reader deal with any non-fiction reading? How about, say, Julian Jaynes Origin of Consciousness in the Breakdown of the Bicameral Mind? Can a proficient reader read a whole Glenn Beck book and spot which parts are crap? Because that was some pretty heavy stuff! How about legal documents? Does a proficient reader read legal documents well enough to understand them sort of, or completely, or well enough to mount a capable counter-argument to the legal document? Would I count as proficient if I only ever read chunks of reading that were all 1000 words or less (like, say, blog posts), or does proficiency mean dealing with longer, more involved stuff? If college readiness is part of proficiency, does that mean a proficient reader is ready to do the assigned reading for a class on Italian Literature at Harvard or a class on Engineering at MIT or How To Talk Good at West Bogswallup Junior College? Will a proficient reader get A's? C's? And speaking of levels of ability, would a proficient reader read all of a Dan Brown or Stephanie Myers novel and know that it was terribly written? Would a proficient reader have made it all the way through this unnecessarily lengthy paragraph, or would a proficient reader have figured out that I was using bulk to make a rhetorical point and just skipped to the end?
Or does "proficient" just mean "able to manage the dribs and drabs of reading-related tasks that we can easily work into a standardized test"?

I'm still wondering. It's not that I don't think there are levels of how well-educated a student is, or not. But when reformsters start throwing around words in ways that don't actually mean anything, I suspect they're busy trying to cloud an issue rather than illuminate it. I suspect they're trying to lay down a smokescreen to cover whatever piece of thievery they're up to now.

And "proficient" is a big, thick smokescreen, billowy and opaque and yet possessing no substance whatsoever.

Is there a proficiency gap?

Between what and what? If the assertion is that we have a gap between the results of one lousy standardized test and another different lousy standardized test, then, yeah, I guess so, but so what? If the gap is between what we tell students they can accomplish and what they actually are able to accomplish-- well, where's the evidence? Oh, I know what reformsters believe-- that all the poverty in the country is the result of students who couldn't score high enough on a standardized test. This strikes me as highly unlikely, though I get that there are many possible explanations for and solutions to widespread poverty. But if we've had the most terrible education system in the world, and we should fear that because it will lead to failure and collapse, I just feel as if the country isn't doing as badly as all these chicken littling privatizers want to say, and where I do see failure, I see problems of racism and systemic barriers to class mobility. Oddly enough, race and poverty do not appear as issues on the proficiency gap site.

So if FEE is declaring that states need to do more about closing the resource gap and the opportunity gap and the stupid racist barriers gap, that would be swell. But I've read enough FEE materials to suspect that they're chicken littling in one more act of "There's a terrible emergency, so you must do as we say!!" The Honesty Gap folks wanted us all to buy more PARCC and SBA tests, and Common Core harder, as well as handing over more public schools to private interests. Oh, and stop opting out. This seems like more of the same old stuff aimed primarily at helping privatizers close their revenue gaps.