Nobel-Winning Evaluation Advice

Sometimes it takes a Nobel Prize-winning economist to confirm what many of us in education have been saying for years-- reformsters are doing teacher and school evaluation all wrong.

The writer pointing this out if Derek Neal (University of Chicago), over at Education Next of all places, The prize-winner he's talking about is Bengt Holmstrom, a Finnish economist at MIT.

This guy has more Nobel Prizes than you

Holmstrom has done pretty much no work on education. What he has done is work with contract theory, particularly with contracted with incentives in areas "where worker actions are hard to observe and worker output is difficult to quantify." So, totally education.

Neal focuses on two major elements of Holmstrom's work, providing two insights into how an incentive pay system can go wrong.

1) Good incentive systems use all the data that provides more clarity and detail about employee performance. So, not just some sliver of data, and not data that doesn't really improve the picture of job performance.

2) There must be alignment between the performance task and the actual desired task.

Both which tell us that the test-based evaluation that has been favored since NCLB was a pup is seriously off the mark.

So, if out of the full range of teacher behaviors, you collect only data about how students do on a narrow reading-and-math test (which is also measures a mess of data unrelated to teacher performance), you cannot build a good incentive system on that data.

And if, for instance, you are not measuring how well students read, but rather how well they answer multiple-choice questions, your data is not suitable for creating performance incentives.

Basically, Holmstrom provides a fancy explanation of why Campbell's Law is a thing and how it works. When you measure the wrong thing and/or measure the right thing incompletely, you incentivize the wrong behavior, and you get lousy results.

Holmstrom also notes that in some settings, collecting enough of the right data can be really prohibitively expensive. So what to do instead?

In these settings, the best approach may be to adopt hiring procedures that identify workers who will perform well in order to satisfy their own personal norms and the norms espoused by the organization, and then pay these workers a fixed salary.

What?? Hire competent, self-directed people with solid training and then just pay them well?! That's crazy talk, you Nobel-winning loon.

Neal also notes that reformsters are more focused on doing evaluation than on doing it correctly, but that the data from the past decade or three shows the systems put in place are not working so well.

Many voices in current education policy debates are advocating an end to all forms of assessment-based incentives. These reactions are understandable given the evidence gathered over the last two decades or more, but we do not yet know how educators would respond to well-designed incentive schemes that incorporate the theoretical insights of Holmstrom and others, as well as the empirical insights produced by decades of research on the use of incentive systems in schools, government agencies, and businesses. We do know that, if policymakers continue to ignore these insights, those who oppose all forms of assessment-based incentives will continue to gather evidence that lends support to their cause.

It's not rocket surgery. If policymakers continue to design incentive systems based on measuring the wrong things badly, on measuring based on tasks that are not aligned withy the results we actually want, then these systems will continue to stink.