Four Reformy Problems in a Single Tweet

@NeeravKingsland @CtrEdEcon @smarick @amandaripley How do we measure schools' progress without annual tests?
— Michael Petrilli (@MichaelPetrilli) August 24, 2014

This is the kind of statement that brings me up short. It's neither notably nefarious, nor is it larded with some deliberate obfuscation, and yet it is loaded with so many assumptions.

Measure schools' progress

Is "schools' progress" really a thing? It sounds more sciency than "is the school getting better"? But it has the same problem. Progress is a giant tribble of a word, so fuzzy that one cannot really make out a shape within the furry expanse. Progress from where to where? Become a better connector of the community? Become safer? Develop a stronger arts program, or more successful sports program? Do a better job of creating students who are able to function as self-directed learners? Graduate more students who get into college or get good jobs?

There are so many ways in which a school can progress (and most schools are tying to move in several of them at once. How exactly will standardized test results measure all of those many and varied forms of progress?

Measure schools' progress

Annual standardized tests will only measure one sort of progress-- how well the school is doing at housing students who do well on standardized tests. That is a standard for progress that can be met many ways, including paying close attention to which kind of students you're housing. This truly does measure the school's progress-- but not the progress of students, which would be a nice goal for a school. But we've moved from measuring student achievement to measuring student progress to measuring schools' progress.

Why? Why is measuring school progress useful? Why is it the end goal? It seems a little like checking the oven temperature as a way of determining if the turkey is cooked. Why not just check the turkey?

And that gets us right back to progress. When we talk about great schools, schools or students that are making progress-- what do we mean?

Without annual tests

Is that what we mean? Progress will be defined as "whatever we can measure with a standardized test?" Thomas Newkirk gets to the heart of this:

It all comes down to the parable of the drunk and his keys, an old joke that goes like this: A drunk is fumbling along under a streetlight when a policeman comes up and asks him what he doing. The drunk explains he is looking for his keys. “Do you think you lost them there?” the policeman asks.             
“No.But the light is better here.”

Neither of the necessary questions is being asked-- what do we really want to measure, and what would be the best way to measure it?

We

What do you mean "we"? Who is this we? And why do they need to have a measure of schools' progress?

Is it parents? I don't think so-- why would parents care about any plural schools beyond their own singular one? Who is it that needs a national scale everyschool rundown of how things are progressing? When exactly did the US appoint a Grand High Overseer to whom all schools must answer, and whom did we appoint? Is this the US DOE talking? A representative council elected by all the American taxpayers?

I know it's picky as shit to peel apart a simple tweet, but I want to highlight how many unexamined assumptions find their way into the education discussion. If we could start examining them more often, maybe we could start talking about the real concerns that should be on the table.

The Cult of Order

Many, many, many reformsters are members of the Cult of Order.

The Cult of Order believes in blind, unthinking devotion to Order. Everything must be in its proper place. Everything must go according to plan. Everything must be under control.

It is not new to find cult members in education. We all work with a least a couple. Desks must be just so. The surface of the teacher desk must be pristine and orderly enough that bacteria will avoid it and others will either stay back in awe (or experience a near-uncontrollable urge to violate it). Students lose a letter grade for putting their name in the wrong corner of a paper. In high schools, they believe that even seniors would benefit from going class to class in neat and orderly lines.

But reformster members take the Cult of Order to new levels.

They were bothered by the chaos of the crazy-quilt state standards, each different from the rest. They are alarmed at the possibility that individual teachers might be teaching differently from other teachers. Order, predictability, uniformity-- these are qualities to be pursued, not because they are a path to better outcomes of some sort, but because they are in and of themselves desirable outcomes. Standardization and a national curriculum that gets every student in every classroom on the same page at the same time-- this vision is good. Don't ask "Good for what?" To the Cult of Order that's a nonsense question, like asking about the utility of a kiss. For them, controlled, orderly  standardization is as beautiful as a sunrise.

The Cult of Order is all about fear-- fear that some sort of dark, menacing chaos lurks just beyond the borders. There's a horrible monster waiting just on the other side of that white picket fence, and the only way to keep it at bay is to make the fence just as neat and orderly as possible.

And yet, we know this is not how the world of human beings works. Human relationships are messy, wobbly, unpredictable, hard to plan. At first flutter of your heart, you cannot know how that story will end. Friendship may grow or wither, and no amount of orderly control can change it. And on the large scale, throughout human history, the dream of perfect order always travels hand in hand with aspirations of totalitarian despots.

It is only in the modern age that a true dream of perfect order seems attainable. The Romans maintained centuries of empire precisely because they developed a system that did not depend on perfect order and standardization, but left the many varied local governments in place. The Romans knew that complete order and control was unattainable. The modern Cult of Order believes it is.

Science also tells us that the Cult of Order is wrong. Chaos theory and information theory and quantum mechanics all tell the same story, The dream of simple linear Newtonian order, where insisting on A always gets you to B-- that dream is unattainable, a failed model that does not reflect how the world works. There is a kind of order in the chaos, but it is more rich and complex than we have ever imagined. More importantly, it is an order that does not respond to nor allow for planning and control.

Reformsters keep asking, "How can we precisely control the aspects of education in order to get the exact results that we want?" This is a oxymoronic nonsense question, like asking "How can I best kick that cute girl in the face in order to get her to love me?"

To try to exert that kind of exacting control over other human beings is not just futile-- it's damaging and destructive. To remodel American public education into the model preferred the Cult of Order is destructive and wrong.

ACT Report Finds ACT Really Important

Okay kids. Here's today's lesson in critical thinking (which, as you may have heard, is built right into the heart and sinew of the Common Core).

When a business releases a study showing the importance and effectiveness of that business's product, is there a possibility that the study might be aimed at something other than Telling the Whole Truth?

The folks at ACT must be really sweating these days. Their competitors at the College Board have scored a couple of coups, including A) hiring the well-connected author of the CCSS ELA standards, David Coleman and B) getting state and federal governments to adopt their line of AP products as the official education product of the US government. So the ACT people are in need of some realmPR work.

So they are here to report on the national level of college readiness.

This report provides the ACT folks with a great chance to let us know what they've been up to as they desperately try to grab some market share work to make US education sweller.

They have, for instance, released ACT Aspire, a test package for assessing student college readiness for grades 3 through 10. It aligns with the ACT College Readiness Standards, so take that, new multi-grade PSAT.

The ACT itself has been "enhanced" by the addition of stats and probability math questions, new questions that require students to "integrate knowledge and ideas across multiple texts, a STEM score, a new writing section enhanced with God-knows-what, and, of course, a computerfied version to take on line, because, computers. They will start throwing in ACT Profile, which appears to be career-suggestion software, so I guess the ACT is now an aptitude test. They also want you to know that they are committed to making sure that the test is continuously monitored for validity and evidence, which may seem obvious, but since David "An Education Is What I Say It Is" Coleman seems inclined to move SATs away from validity and evidence, it's worth the ACT's time to toot their own trumpet of obviousness.

Now, if you think any of that qualifies as a "report" and not "advertising copy," you are not exceeding expectations in your applying of critical thinking. But that's okay, because even Caralee Adams writing about the report at EdWeek, fails critical thinking regarding the meat of this report.

She takes away exactly what she was supposed to-- students who take lots of ACT courses do better in college, therefor, the ACT courses must have prepared them super-well for college.

Why is it that education "research" is so riddled with an inability or unwillingness to distinguish between correlation and causation?

If we were going to design an experiment to determine the effectiveness of a product (say, New Shiny ACT Creme) we would need to first start with subjects who were close to identical. So, let's start with 200 fifteen-year-olds who have similar amounts of acne, give 80 of them New Shiny ACT Creme, 80 of them nothing, and 40 of them mayonnaise. After several months, we'll check to see how the acne looks.

That is not what this "study" has done.

Instead, this study says, "We want the teens who think they have the best complexion to use ACT Creme. We'll let them try it, and in a few months, we'll check to see how they look."

What the ACT report seems to show is not terribly remarkable. Apparently the same high-achieving, hard-working, top-ability, ambitious students who do well in college also seek out the challenging courses in high school. Put another way, the students who value college success and take steps to achieve it in college, also take steps to achieve it in high school. In other news, the sun is expected to rise in the East tomorrow.

I'm not suggesting that the ACT is a scam and terrible waste of time. Like AP courses and the SAT, the ACT clearly has some benefits for some students, and that's a Good Thing.

But we really need to stop looking at these groups as if they are philanthropic organizations trying to make the education world a better place out of the goodness of their highly objective hearts. The ACT is a business, and any "report" they offer about their product is most reasonably viewed as advertising copy, not scientific research.

News organizations do not run stories with headlines like "Researchers Discover Frosted Flakes Are Grrrreat" or "Study Shows Men Who Wear AXE Get Laid More Often." At least not yet. In the meantime, let's keep those critical thinking caps firmly attached.

Objectivity Is a Unicorn

Objectivity is a unicorn. It's inspiring to believe in, pretty to create pictures of. Some folks love the idea of objectivity so much that they dress up horses or try their hand at photoshop. But at the end of the day, it doesn't actually exist.

Nevertheless, we continue to enshrine the idea of objectivity in places where it does not exist. Writers have repeatedly reminded us, for instance, that internet search engines are not objective. This article from Michael Kassner at Tech Republic provides a good summary of the basic arguments. When you google something, you do not pull up some objective summary of what the internet contains, but a list that has been weighted by programmers who judge that certain factors should be considered (including a cyber-reading of you and your own proclivities). Google results are just as subjective as if they were compiled by some guy and his buddy making their best guess about what you want and what you should see.

But we really like to believe in Objective Facts, and in particular the notion that certain instruments provide such facts for us.

Some simple instruments do provide some rudimentary objective data. A measuring stick will provide an objective indication of length of an object. But the stick might be measuring meters or feet, and it might include hatchmarks for very small units or not, and if you're using to measure something that's not exactly straight, you will have to make a judgment about how do that. Even the simplest measurement includes subjective judgments. And that's working with objects, not humans.

Complex measures are mostly subjective judgments, whether we are measuring all the articles on the internet and how well they match your search terms, or we are measuring how "college and career ready" your eight year old child might be.

We keep talking about standardized tests as if they are objective measures of... well... anything. They are not. A standardized test is the product of the individual personal judgment of the test writers, who have their ideas about which specific bits of knowledge and skill should be tested and who make their own judgments about what exact tasks would measure those bits. They may claim that research backs up some of their choices, but research is itself the result of individual subjective judgments and choices made by the researchers deciding what to measure, how to measure it, and how to interpret the data they generate. In some cases, such ads David Coleman's reconfiguration of SAT vocabulary, they simply baldly say, "I think we should do this, not that."

Track the elements of standardized tests in any direction you wish; you will soon arrive at human beings making personal subjective judgments about how the test should work.

Reformsters keep talking about the use of testing and data as if it will result in replacing the varied subjective judgments of a teacher with the pure objective results of the testing. No such thing is true. What they seek to do (whether they understand it or not) is replace the judgment of the teacher in the classroom with the subjective judgment of the persons who make the test.

When someone claims "this test is an objective measure of a students language use ability," they are wrong. The test is, at best, a pretty good subjective measure of some tasks that some test-writer guys believe probably indicate language use skill. It is no different from having some person come into the classroom and say, "I'm going to sit and talk to Pat for a couple of hours, and then I'll tell you how good a reader I think he is."

It is not humanly possible to remove subjective judgment from education (or, arguably, anything at all, but let's narrow our focus for now). Never even mind the question of whether or not we should-- it cannot be done.

How do we deal with the inherent subjectivity?

The problem with subjectivity is that it solves problems within a personal framework. We operate in our own little bubble or silo and we define problems and search for solutions based on what we see and know inside that little framework. The trick is to temper subjectivity by expanding that framework. We need to see and know more.

We expand the framework with professional knowledge. We train teachers to understand pedagogy and subject matter so that their instructional judgment is not based on considerations like "What kind of mood am I in" or "Do I like this kid."

We expand the framework with personal relationships. One of the terrible lies of the cult of objectivity is that we can make good decisions about students without knowing a thing about them. Management schools recommend that managers not live in the same communities as their employees, so that relationships are not part of their framework.

Baloney. One of the marks of a good decision is that you can talk about it out loud with the people who are affected by it right there in the room with you. When our framework is expanded to include the people who are part of the choices and the results, our subjective view of the situation is more complete, more useful to people beyond ourselves.

Reformsters have done their damnedest to keep teachers, students, and much of accumulated wisdom of American public ed outside of their framework. This leads them to say naive things (Hey, did you guys know that water is wet) and stupid things (Hey, if you tried hard, I bet you could build a house out of water). It doesn't ever lead them to say objective things.

They claim that their goal is to inject objectivity into the education world, when in fact they're simply trying to impose their own subjective judgments in place of others'. When someone wearing a white hat rides into town on a unicorn, you'd better check to see how the horn is attached.

I've Launched at EdWeek

As noted a while ago, I have taken on blogging for Education Week, and the new blog launched a few days ago.

The EdWeek blog will be called "View from the Cheap Seats," and I'll be posting there at least weekly. I even get a cool graphicky thing for the header!I am excited about the potential for reaching a new audience. The first post is about the absence of the teacher voice from education debates. Feel free to stop over, read it, and shower them with demands that I be paid a zillion dollars. Or just enjoy it.

Superintendents Speak Up

On the first day of school, my wife's superintendent got choked up.

He was delivering the usual kick-off speech, and she said he started to talk about testing and numbers and the students. He reminded his staff that students were not just test scores, not just a number, and that the work they did as teachers was so much more than could be measured by numbers. It looked, she said, as if he was on the verge of tears.

My wife's superintendent is my former principal. He's a good man and a fine educator. And apparently he's done pretending that chasing test scores is a good way to run a school district.

This may well be the fall we remember for the number of school district leaders who have finally had enough and begin to speak up.

Examples abound. In Peru, Illinois, Superintendent Mark Cross sent out a letter that said in part

Unfortunately, there are many federal and state education initiatives that can very much be a distraction from what matters most. These initiatives are based on good intentions and are cloaked in the concept of accountability, but unfortunately most do little to actually improve teaching and learning. Most are designed to assess, measure, rank and otherwise place some largely meaningless number on a child or a school or a teacher or a district. That is not to say that student growth data is not important. It is very critical, and it is exactly why we have our own local assessment system in place. It is what our principals and teachers use to help guide instruction and meet the needs of your kids on a daily basis. In other words, it is meaningful data to help us teach your child.

He makes this commitment

This is why I wanted to let you know that we will not be talking to you that much about the PARCC assessment or Common Core or other initiatives that have some importance, but they are not what matters most to us. YOUR CHILDREN are what matter most, and we believe that kids should be well-rounded, with an emphasis on a solid foundation for learning across all subjects by the time they get to high school and later college. We believe that kids need to be creative and learn to solve problems. We believe that exposure to music and art, science and social studies, physical education and technology and a wide variety of curricular and extracurricular activities will serve them very well as they grow into young adults.

And he delivers this pointed (if grammatically suspect) indictment

The state and federal government have failed epically in their misguided attempts at “reforming” public education. Public education does not need reformed. 

Superintendent Cross is, of course, not alone. In Washington State, the education system has lost its waiver from Arne Duncan because the state legislature would not implement the federal Department of Education's preferred method of teacher evaluation. So Washington schools are operating under No Child Left Behind, which means that all schools not meeting requirements that 100% of students be above average (aka "all schools") are failing. You may recall that one of the punishments for failing schools under NCLB is that they must send out a "We Are Failing" letter to the public.

Superintendents in WA have sent their letter. However, 28 superintendents wrote a letter which includes the observation that the label under NCLB is "regressive and punitive." The basic layout of the letter is "The feds say we have to tell you this, which we are now doing, however you should know that the feds are full of it, their policies are stupid, and we are educating your students pretty well, thank you very much."

And as I noted here yesterday, the Board of Education for the entire State of Vermont has adopted a resolution calling out the feds on their stinky testing requirements.

The tone in administrative offices is continuing to shift. Ten years ago there was a lot of kool-aid drinking. Then we had "Well, it's the law." Then we had fatalism and resignation, "Well, let's just do our best work and hope that these tests take care of themselves and somehow things work out." What we have always needed are administrators to stand up and say clearly, "This is not right. It's not right for us, and it's not right for our students."

I know there are still districts and entire states where the school leaders have not only drunk the kool-aid, but are selling it themselves out of the back of a van. But it's heartening to see and hear more who are willing to speak out in a meaningful and public way. Duncan is clearly trying to stem the tide with his waiver-waiver, the offer of "Look, we'll just wait a year and then we'll punch you in the face." But postponing a stupid thing does not make it any less stupid, and in the meantime, more and more people are starting to point out that the emperor's clothes (which are no longer new) are woven out of air and empty promises.

Vermont BOE Hammers Fed-Style Testing

With states like North Carolina and Florida doing their best to bury public education and dance on its grave, it's nice to see some states can still stand up for their schools.

Earlier this week, the Vermont State Board of Education adopted a statement and resolution on assessment and accountability. It's worth a read, but let me hit the highlights for you.

The Board starts by recognizing that uniform standardized tests can be a useful tool for helping schools chart a path toward successful delivery of well-designed standards. And then comes the pivot-

Despite their value, there are many things tests cannot tell us. Standardized tests like the NECAP and soon, the SBAC, can tell us something about how students are doing in a limited set of narrowly defined subjects overall, as measured at a given time. However, they cannot tell us how to help students do even better. Nor can they adequately capture the strengths of all children, nor the growth that can be ascribed to individual teachers. And under high-stakes conditions, when schools feel extraordinary pressure to raise scores, even rising scores may not be a signal that students are actually learning more. At best, a standardized test is an incomplete picture of learning: without additional measures, a single test is inadequate to capture a years’ worth of learning and growth.

Unfortunately, the way in which standardized tests have been used under federal law as almost the single measure of school quality has resulted in frequent misuse of these instruments across the nation.

In order to avoid that sort of foolishness getting loose in the Green Mountains, the Board lists eight guiding principals for the appropriate use of standardized tests.

1) The proper role of large scale tests must be stated before giving the test, and that use must be demonstrated as scientifically and empirically valid. That includes proof that the test can predict performance on "other indicators we care about, including post-secondary success, graduation rates and future employment." And you can't use the test all by itself-- mix it up with other measures.

2) Public reporting. Schools need to do that, but they need to report a wide variety of indicators that give a full picture of what they're doing.

3) Judicious and proportionate testing. Reduce the amount of time on summative and standardized testing. The feds should back off on multiple subject testing grades 3-8 as well as high school (so, you know, all of it). "Excessive testing diverts resources and time away from learning while providing little additional value for accountability purposes."

4) Test development criteria. Any big standardized test used in Vermont needs to be built in accordance with principles of American Educational Research Association, National Council on Measurements in Education, and the American Psychological Association.

5) Value-added scores. Near as we can tell, these are crap. We will not be using them in Vermont "for any consequential purpose."

6) Mastery level or Cut-off scores. This whole paragraph is pretty awesome.

While the federal government continues to require the use of subjectively determined cut-off score, employing such metrics lacks scientific foundation. The skills needed for success in society are rich and diverse. Consequently, there is no single point on a testing scale that has proven accurate in measuring the success of a school or in measuring the talents of an individual. Claims to the contrary are technically indefensible and their application would be unethical.

7) Use of cut scores and proficiency categories for reporting purposes. The fed since NCLB was born have required this. Here's a list of ways in which it has been documented to create negative effects. We'll keep doing what the letter of the law requires, but it's crap.

8) Just as the state high quality education, the federal, state and local governments must provide adequate resources to get the job done. If you're going to demand a report on the quality of the school's work, demand a report on the sufficiency of the resources provided to the school "in light of the school's unique needs."

These are followed by several whereas's that note that the the nation's have been spending an ever-increasing amount of time and money on testing of a sort and in ways that are known to be No Damn Help to anyone and wrapping up with

WHEREAS, the culture and structure of the systems in which students learn must change in order to foster engaging school experiences that provide joy in learning, depth of thought and breadth of knowledge for students [emphasis mine, because, damn, wouldn't you like to see that in every school's mission statement!]

And then we get the Be It Resolved portion 

-- The Secretary of Education should re-examine the accountability system and come up with one that sucks less (I'm paraphrasing)

-- Congress should get off its collective keister and amend the ESEA

-- Other state and national groups should join us in this

I grew up just across the Connecticut River from Vermont, playing in my front yard and looking at the big beautiful mountains, but I have never loved Vermont more than I do reading this resolution. If you see Vermont today, give it a big hug for me. And send Arne Duncan a copy of their resolution. It's true these are just words-- but they are damn fine words.