Monday, March 23, 2015

Who's Scoring the Test?

A sharp-eyed BAT spotted this ad, one more example of a genre that has become almost cliche-- the Craigslist advertisement for test scoring work.

Do You Have a College Degree?

Thank you for your interest in employment with Measurement Incorporated. We are a diverse company engaged in educational research, test development, and scoring tests that are administered throughout the world. Our company has grown to be a leader in the industry by providing consistent and reliable results to our clients. We are able to do this through the professional efforts of a flexible staff, and we welcome your interest in becoming a member.

Measurement Incorporated boasts ten scoring centers, which is a good thing because "to guarantee test security, all work has to be done at one of our Scoring Centers in Tennessee, Michigan, Florida, North Carolina, Kansas and Washington or from a secure work station in your home."

The ad, which went up ten days ago, is part of a recruiting drive for the test-correction high season of March and April. "These projects may include scoring test items in reading, math, science, social studies, or written essays. The tests come from many different states representing students at all grade levels."

The job starts at $11.20. After logging 450 hours, workers are eligible to bump up to $11.95. Day and night shifts are available, and workers are expected to put in five days a week.

Their employment website provides more details of the job in language that is apparently designed to sift out the better-educated prospects who can speak corporate balonese:

Within the field of performance assessment scoring, MI has distinguished itself by relying on an extensive and disciplined approach to training and monitoring a carefully selected workforce of qualified readers that is unparalleled in the industry. Due to the seasonal nature of scoring, MI hires and offers paid training to hundreds of temporary, highly-skilled, and well-educated employees to score tests on a project-by-project basis while maintaining strict guidelines for accuracy and quality control. 

Commenters on the job-rating site indeed made many comments about the seasonal nature of the work and the fact that it was an unreliable income. While they were mostly positive about working for MI (can I be amused that the corporate initials just take me back to a million medical tv shows and myorcardial infarctions?) it's clear that this is not a line of work for someone who has a real job. I suppose the labor pool of well-educated college grads who can't find a real job is fairly well-stocked at the moment, if employment ever does pick up in this country, the test-scoring industry could be in trouble. Well, more trouble.

This is, of course, one of the great undiscussed and unsolved issues of national-level Big Standardized Tests-- who the heck is going to grade it? The question is not just qualifications, but quantity-- how do you round up the human hours needed to score several million tests?

So how does MI stay in business if they're doing seasonal work. The NC-based company has some other products, including a service for providing test items and an AI program named PEG for assessing writing, because the computer-graded essay-scoring field can always use one more program that can't actually do the job. MI also has a writing instruction program; maybe I'll look at that another day.

I expect we'll continue to see many of these smaller companies scarfing up sub-contracts for the Big Guys and handling the business of hiring part-timers to help make decisions about the fate of America's children, teachers, and schools. Only one of two things can be true here-- either the system is so simplified and so user-proof that it doesn't really matter who's doing the scoring work (in which case it's a dopey system that gives back very little information and is easy to game) or it does matter who's doing the scoring (in which case, the use of part-time temps who are available only because they couldn't find a real job is not exactly comforting). Either way, this is one more big fat reminder that the Big Standardized Test is a dumb way to assess any part of America's education system.


  1. A sharp-eyed crack me up with your sly word usage, Mr. Greene. Also, the fate of everyone giving or doing the tests depends on people doing scoring only until they get their real job, what could go wrong?

  2. I'm betting this company is also owned by Pear$on (they realize that everyone's on to them), because the ad wording is suspiciously similar.
    Not one thing has changed since Todd Farley's 2009 book was published.
    Time to cut off the head of the monster & point the finger at the perp by letting EVERYONE know just how much of their education funding taxpayer $$$ are being WA$TED on this child, parent & teacher abuse.

  3. Measurement Incorporated is not owned by Pearson. They are an employee owned company.

  4. Looks like they have had their share of problems. Of course they could always claim “We’re Better than Pearson”

  5. An interesting negative article on the state of State testing, but one founded on a false premise. Most of those who score papers at MI (I would estimate between 70-80% at the scoring center I work at) are retired, or former, educators. They would get a kick out of being called recent college graduates, though they may take issue with calling their work either simple or irrelevant. There is plenty to criticize about the testing and scoring process. You could even, now that you are more educated about the topic you decided to write about (WHO scores the tests), go after them for new reasons previously not on your radar before. But most of us are smart, hardworking, dedicated educational professionals.

    1. Agree with July 17 2018 Anonymous! I worked at Durham, 1996-97 and Austin, 2001 (MI has since closed down their Texas facility).

  6. Measurement is going down in training quality and quality of everything in the employment area in the last 3 years. I started there a few years ago and you could have a phone number for technical assistance. For the last two years there has been none but there's not even the phone number to do anything but I don't know what they do with that phone number perhaps they just got to sell the services there. Also it seems, this year, there are no training videos. Apparently they have decided not to pay to make the training videos so that those of us who aren't something, I don't know, mind readers Maybe oh, those who can't just guess how they want us to do the scoring. No no no it's getting bad there is no phone number for supervisors there's no phone number for Tech Health it's all done by email and chat box there's no phone number to talk to anyone at any more now no training videos to explain the subtleties of the scoring process. Nope .they don't even advertise for people anymore so I guess their business has gone down LOL.
    And I know darn well one of those previous years at least on one of my training test that they scored it wrong but they passed me anyway here they're not doing it now. This year, I had some technical problems and I'm sure they interfered with me being able to pass the qualifying test but no I can't do it over here they won't let me. Several years of experience down the drain they don't care