A Better Way To Measure School Quality

In every discussion, debate, argument and flame war about using test-centered accountability to evaluate schools, sooner or later, devotees of test-generated data will unleash this one:

Well, if you don't like this system. what do you want to do instead?

Now, I want to note that I reject the premise of that question. If I'm suffering from lung cancer and a garage mechanic comes into my hospital room with a chain saw and says, "We're cutting off your les\gs" and I reply, "Like hell you are," the mechanic does not get to say, "Well, then you'd better propose some other limbs to chop off." Just because the mechanic has some crazy ideas about how to treat me, that does not mean the burden of proof is on me. The burden of proof is on the crazy person with the chainsaw, or, in our case, the well-connected education amateur with a standardized test. If you can't prove you know what you're doing, you don't get to operate, and I don't need an alternative to demand that you and your chainsaw back off.

Not this

That said, it's not a bad idea to develop some means of evaluating schools. I've take a stab at it myself re: teachers. But now we have a better, more complete answer to the question, and it comes in the form of a must-read new book from Jack Schneider.

Schneider (who tweets as @Edu_Historian) is an assistant professor of education at the College of the Holy Cross. Schneider has written extensively about education, including some ambitious projects such as his long-running edweek blog in which he entered dialogue with a number of education voices, including She Who Shall Not Be Named, former DC ed chief. Currently he co-hosts the podcast Have You Heard with Jennifer Berkshire.




Beyond Test Scores: A Better Way to Measure School Quality is a look at both the general principles of how to create a better school evaluation system and also at the specific work of creating an actual system for some schools in Massachusetts.

You can tell that the book is serious because the cover is pretty bland and boring, but Schneider's voice as always is clear and conversational, a scholar who knows how stuff, but spends most of his days with non-ivory tower folks.

The basic idea here is as simple as it is on point.

Who's the best actor? We could ask ten people and get ten answers , because "best" is not quantifiably objective. Your choice depends on your personal values-- do you think chiseled good looks matter more than smooth, youthful, charm? What you value most will affect your ranking.

Or let's consider the nutritional panel on a food item at the grocery store. The panel gives us lots of information-- but it doesn't try to rank the foods. You do that yourself, based on what you value most.

It's the state, the bureaucratic level, that wants to come up with a too-simple overly-reductive ranking system that is actually not very informative for parents, students, taxpayers, community members, or even teachers. (Everywhere I look these days, I see echoes of the book Seeing Like a State-- thank you, Andy Smarick.)

In six brief, clear, and thorough chapters, Schneider lays out what we're getting wrong and how we can get it right, as well as the practical aspects of applying this approach in the Somerville, Mass, school district.

Chapter One: Wrong Answer

A great summary of how we got here and why testing doesn't really tell us what  proponents claim it does. In addition to the usual criticisms of the Big Standardized Test, Schneider mentions one that we don't bring up often enough:

While there may be some benefit to the skill of sitting quietly to focus on a test, that skill is separate from one's ability to read, write and compute.

He notes the late-nineteenth century attempt to create a scientific system, but also with the goal of tighter governance. "In other words, it was not enough to have created statewide networks of schools that would be free and open to children. They also wanted to control what was going on inside those schools." Then, as now, a tool was needed to essentially wrest autonomy away from the professional educators in the school.

Tests were, and are, such an instrument. And the tests were also hella profitable, so that "entrepreneurs would not just work to meet demand, but also to manufacture it."

All this was simply setting the stage before A Nation at Risk. Policy elites were looking for a way to put their foots down, and as Schneider puts it, "State-run testing offered a way of solving the problem of teach autonomy."

This chapter alone is a great piece of reading to offer anyone who wonders what the big deal is about the Big Standardized Test.

Chapter Two: Through a Glass Darkly

How do people (parents in particular) currently gauge school quality? Schneider unpacks research to show a variety of ways, from bad ways like the test scores and  the US News badly constructed list of schools, to really bad ways like checking to see how many black kids there are. Or just plain conversations. You can dig through the specifics, but here Schneider makes it clear that the available sources of data range from lousy to irresponsibly bad.

Chapter Three : What Really Matters

In chapter three, Schneider lays out his new framework for school quality. If we all have many ideas about what a good school does, are there commonalities, some areas that we can agree one? Schneider says yes, and in this chapter he lays out the framework for five areas, grouped into two headings. And he gives us a look at how these played out in the

For "Essential Inputs" we have  Teachers and the Teaching Environment, School Culture, and Resources. Under "Key Outcomes" we have Academic Learning,  and Character and Well-Being. There are subheadings for each, and a lot of details to get into in this chapter, but does this not already look better than any school evaluation system you've been subjected to previously. (My only nit here is the absence of any long-term results-- where are the grads in ten or twenty years).

This chapter also includes one of my favorite obvious-but-worth-saying quotes:

As one teacher put it: "Parents still expect us to help their kids grow up. Not just ace [the state standardized test]. But grow up."

There are clear definitions of each category and sub-category here, with banks of questions to ask and have answered. This chapter is the meat of the book, the chapter that you should be sending to your board members and strategic planning committee.

Chapter Four: But How Do We Get That Kind of Information

Schneider looks at some of the methods for gathering the information that his model calls for. That includes surveys -- yes, he argues, they really work, and he gets into the specifics of designing them to yield useful information.  He also talks about performance assessments, harkening back to the nineties. If you are of a Certain Age, you may recall that education was being swept by a wave of authentic assessment just about the time NCLB came along and put paid to that whole business, elevating the inauthentic assessment of BS Testing. And in all fairness, Schneider recalls one of the big sticking points-- assessment systems like portfolios are time-consuming and expensive?

Intriguingly, Schneider also devotes some space to how parents could create a DIY pilot of these methods.

Chapter Five: An Information Superhighway

Schneider here looks at how to make the information that has been collected available.

Everyone, whatever their level of expertise, has a right to know how the schools are doing.

In this chapter, Schneider focuses pretty specifically on the Somerset experience, but articulates some clear priorities for the system. He talks about how to visualize the performance ranges for each characteristic of the schools, noting that they did not allow users to rank schools by characteristics.

They also did some interesting work on how the system  might change perceptions of schools. The system didn't really move the needle on local school opinions which, in keeping with usual poll results, were pretty positive. But the system did change hearts and minds about those Other Schools-- in fact, those opinions could even be changed by second-hand reporting of results. And word of mouth remains hugely influential for schools with which folks don't have first-hand contact.

Chapter Six: A New Accountability

Finally, Schneider addresses accountability systems and how they can grow out of new school quality measures.

In particular, he lists nine  guidelines for a "fair and effective" accountability system.

1) Empower communities in the improvement of schools.,

2) Create a system of reciprocal accountability. Schools are responsible to perform; government is responsible to support.

3) Use multiple measures to assess and report on school quality.

4) Capture what communities want to know. Man-- this.

5) Establish benchmarks based on high -quality schools. I'm less excited about this.

6) Place greater weight on student growth than on absolute scores.

7) Emphasize support rather than punishment.

8) Capture a reality that members of the school community will be familiar with. In other words, if your evaluation system brings back results that contradict what everyone with first handf knowledge knows, something's wrong with your system.

9) Remain an unfinished poduct

Wrapping up.

There is a lot to chew on in this book But it's pretty exciting to hold the answer to "So how should we measure school quality" in your hand. I recommend you buy ten copies of this book and give nine away to administrators, school board members, and legislators. Yes, the field is currently owned by fifty mediocre-to-crappy ESSA compliant systems. But within a year or three, everyone will notice that those systems are failing, and when the call goes out for something better, this book can already have laid the groundwork.

In short, read this book.