Efficiency Report Is Only Mostly That Bad

The edublogonewsphere buzzed a bit this week about a report from GEMS Education Solutions regarding educational efficiency and the fact that it finds the US somewhat lacking. It is, however, a large report, and some reporters made the mistake of skipping through to just the shiny, sexy parts. I'm not sure that the report has anything useful to tell us, but I don't think it's quite as outrageous as some folks have painted it.

GEMS Education Solutions is the consulting wing of GEMS Education. This is the gigantic multi-national education corporation that American privateers dream of becoming when they grow up. Started by a pair of teachers in 1959, the company now has offices in London, New York, Singapore and Delhi, with headquarters in Dubai. Would anyone like to guess what office space costs in Dubai look like. You can get the quick company tour in this video:

Despite the incredibly creep title frame that youtube has selected, this is a slick and impressive video-- one that presents the international ed biz as a big, corporate business. Should the US education market ever be really, truly opened to outfits beyond hamfisted amateurs with crony connections, this is the company that will eat many private school operators' lunch. Star Trek got one thing wrong-- when the BORG come, they will be pretty.

Like any good multinational corporation, they do their homework, and they look at how governments are functioning. The study and consult side of the business (in addition to this and the school-running side, there is also a philanthropic side) is called GEMS Solutions, and those are the folks who cranked out that efficiency report.

If you want to see the highlights, go here. Hell, go there anyway. I read a lot of these reports, and nobody has created a prettier, slicker on-line presentation format than this group. Every thinky tank putting their "research" projects in lame pdf format should take a look at this. Then, when you want to go look at the full report (also in a slick package), go to this link.

I'm not going to get into great detail, because at the end of the day, the report is kind of a waste of time. The concept is not entirely ridiculous, but the foundation is rotten.

The concept? Let's look at sixty-some factors that could affect a nation's educational program and see which ones make a difference. Turns out, they say, that only class size and teacher pay matter. So then they do a basic efficiency study-- whose results come at the best price for their situation? Who is getting the most bang for the buck? At one point they compare it to studying the fuel efficiency of a vehicle, and that's not unfair.

Unfortunately, all of this is built on a bad foundation. Their measure for the bang, the How Well Is The Nation Doing in Education piece, is PISA scores. Even if we accept that PISA scores are a good measure of anything (and that's a debate for another day), do we want to say that good PISA scores are the point and purpose of a nation's education system?

The reports authors acknowledge that might not be the case:

...some [countries] are in the fortunate position to be able to focus on outcomes, because resources are plentiful. Customers buying luxury sports cars are not likely to be concerned with fuel efficiency; they can choose to prioritise other highly desirable features and are prepared to pay higher fuel costs for the privilege. It is very possible that some educational systems are similarly paying a premium for additional outcomes beyond PISA scores. Although providing an excellent method of comparing educational attainment across borders, they cannot measure every output of the system. In such cases, inefficiency may not be considered a problem, but these additional outcomes must be known and desired.

The report suggests that the US would be more efficient if it paid teachers less and had larger class sizes, but what does that even mean in this context?

Efficiency is not excellence, and the report doesn't pretend otherwise. The optimum teacher pay level is not where you get the best PISA scores. It's just that beyond that part, you have to spend larger amounts of money to get smaller results. The authors also use the automotive metaphor to note that even if you have a highly fuel-efficient car, taking a long trip still requires a whole lot of gas.Co-author Adam Still acknowledged as much talking to Joy Resmovits at HuffPost

"We're not saying that the U.S. should cut salaries or should increase class sizes in order to improve quality -- it doesn't make common sense," he said.

Is it even humanly possible to figure out the things GEMS pretends to have figured out? I'm unconvinced. What does average national teacher salary even mean in this context? Would not those figures as well as the class size figures be rather wonky because of various types of outliers? And although the report corrects straight money units into purchasing power measures, that still leaves us with a national average that puts San Francisco dollars and Pittsburgh dollars and Paducah dollars all in the same bucket.

The teacher salary portion of the report says that "too low" salary means "not enough to attract the best people." How does anybody possibly compute such a thing, particularly when not taking any of the cultural or social factors into account? The more one looks at the report, the more one sees a big bunch of mostly-made-up numbers.

Many teachers have bristled at the use of "efficiency" and other biz-world jargon. It doesn't bother me so much-- if we're honest, we have to admit that we make efficiency decisions daily, though our currency is usually time. What's the most bang I can get for my five minutes left at the end of class? But we're making efficiency judgments based on actual usable data, and not sets of information so large and sweeping as to be nearly meaningless sitting on a foundation too tiny and rotten to be useful. GEMS hasn't offended me or shifted my dudgeon-mobile into high gear; they've just wasted a lot of money on this thing, and since they've clearly got plenty of money to waste, I guess that producing a report like this is okay. It just doesn't seem very efficient.