The SAT-- Worse Than You Think

Folks have been questioning the accuracy, validity and usefulness of the SAT for decades, and the chorus of criticism only increased when College Board, the test manufacturing company responsible for the SAT, brought in Common Core architect David Coleman to take over. Coleman's fast and ugly rewrite of the venerable test was intended to bring it in line with the K-12 standards of Common Core. Coleman, whose ego has always seemed to be Grand Canyon sized, had finished redefining education for K-12 students-- now he was going to fix college, too. And, the College Board hoped, he was also going to recapture a share of the market dominated by the ACT. In fact, a plan to grab huge market share by getting states to use the SAT as their federally-mandated Big Standardized Test-- or even an exit exam.

New vocab. New math. New batteries of tests, and a PSAT set up to work as a marketing tool for AP courses (another College Board product). A flubbed delivery on PSAT scores, and then an awkward mess surrounding the spring rollout of the SAT-- so awkward that SAT prep professionals recommended sitting out the first round of the test.

It looked like a shaky product was even shakier. You never want to see them make the sausage, but apparently SAT sausage is being made with even worse parts of unspeakable animals than we suspected.

And now a whistle blower has stepped forward to add to the story, and what we're learning is that as bad as things seemed with the SAT, they were actually much worse.

The name we're all getting to know is Manuel Alfaro. He started out his college career at Williams College, where he co-authored a paper with Jason Zimba, the architect of the math side of the Common Core. Alfaro says they were classmates and friends, but says that the call twenty years later to come work for David Coleman came "out of the blue."

In between Alfaro worked for publishers McGraw Hill and Riverside Publishing. He then spent seven years with American Institutes for Research, another big name in standardized testing. He went to work for David Coleman and the College Board in May of 2013, less than a year after Coleman took over the venerable test maker.Alfaro lasted about a year and ten months.

What he found at the College Board apparently troubled him. In an open letter to his colleagues, he outlines the steps he tried to take:

Over the last year, I’ve explored many different options that would allow me to provide students and their families the critical information they need to make informed decisions about the SAT. At the same time, I was always seeking the option that would have minimal impact on your lives.

I gave David Coleman several opportunities to be a decent human being. Using HR and others, he built a protective barrier around himself that I was unable to penetrate. Being unable to reach him, I was left with my current option as the best choice.

Alfaro elsewhere has this to say about himself:

The College Board will tell you that I am a disgruntled employee. This statement would not be entirely wrong, but it would not be entirely correct either. I am a disillusioned idealist, shocked by the reality I encountered at the College Board.

And so for the last month or so, Alfaro has been blogging on LinkedIN, outlining some of what he saw on the inside that bothered him so much. Mercedes Schneider took a look at his output yesterday, and you can find the up to date list of his posts here.

But here are some of the strikingly awful revelations that he shares. From one of his first posts:

My first assignment with the College Board was to review a draft of the test specifications for the redesigned SAT. The document had been created by two of David Coleman’s cronies, two authors of the Common Core. This document is now known as the “research-based, empirical backbone” of the SAT Suite of Assessments. Back then, it was a subset of standards taken straight from the high school and middle school Common Core. My instructions were to rubberstamp the selection of standards and to rewrite the standard descriptions to make them unrecognizable, so that no one could tell they were Common Core.

So the College Board's story of all sorts of research-based alignment between their "backbone" and the CCSS is baloney-- they just used the standard high school sophomore trick of copying material and changing the language so that it wasn't obvious.

The software system for managing the test content? I suppose we all imagined some sort of complex tagging and tracking system for balancing and managing sophisticated testing strategies for each test item, but Alfaro says no-- just a bunch of files and folders and an excel spreadsheet for banking the test items.

Alfaro says that experimental items are spread throughout the test with no particular regard for difficulty. But that could be because, as he also shockingly maintains, many test items are never reviewed before they make it onto the test in the first place. These items are not reviewed or rewritten until they've already made it onto the test. 

Alfaro says that the work group that performs that review and revision had many concerns, some of which they attempted to raise fairly forcefully. The College Board's own internal review group was concerned about:

Item quality: Their biggest concern was the large number of items that were mathematically flawed; items that did not have correct answers; and items that did not have accurate or realistic contexts. Some members even went as far as stating that they had never seen so many seriously flawed items.

Development Schedule: They felt they didn't have enough time to address all of the item issues.

Development Process: They felt the overall process was inadequate to the task.

So, too many terrible items, not enough time to fix them, and a flawed process for creating a decent test. outside of that, everything right on track.

The short form of Alfaro's story-- the College Board has knowingly lied about using best practices in developing the "revamped" SAT, and in the process of selling the SAT as a state-wide and/or graduation exam, will be lying some more. And it would appear that even this stripped down, cut corners approach isn't letting the College Board get tests written fast enough, for as Schneider found poking around Reddit, the same form of the SAT was given in March and in June.

Alfaro is still out there and still writing. He says that the story has "more plot twists than the Da Vinci Code." It seems certain that those plot twists are not good news for the College Board. Go read some more about the full extent of Coleman's fraud. Stay tuned, and pass the word.

[Update: Alfaro has a White House petition started to demand investigation. Maybe a long shot, but why not sign on and maybe put some more pressure on Coleman's College Board.]