ACT Report Finds ACT Really Important

Okay kids. Here's today's lesson in critical thinking (which, as you may have heard, is built right into the heart and sinew of the Common Core).

When a business releases a study showing the importance and effectiveness of that business's product, is there a possibility that the study might be aimed at something other than Telling the Whole Truth?

The folks at ACT must be really sweating these days. Their competitors at the College Board have scored a couple of coups, including A) hiring the well-connected author of the CCSS ELA standards, David Coleman and B) getting state and federal governments to adopt their line of AP products as the official education product of the US government. So the ACT people are in need of some realmPR work.

So they are here to report on the national level of college readiness.

This report provides the ACT folks with a great chance to let us know what they've been up to as they desperately try to grab some market share work to make US education sweller.

They have, for instance, released ACT Aspire, a test package for assessing student college readiness for grades 3 through 10. It aligns with the ACT College Readiness Standards, so take that, new multi-grade PSAT.

The ACT itself has been "enhanced" by the addition of stats and probability math questions, new questions that require students to "integrate knowledge and ideas across multiple texts, a STEM score, a new writing section enhanced with God-knows-what, and, of course, a computerfied version to take on line, because, computers. They will start throwing in ACT Profile, which appears to be career-suggestion software, so I guess the ACT is now an aptitude test. They also want you to know that they are committed to making sure that the test is continuously monitored for validity and evidence, which may seem obvious, but since David "An Education Is What I Say It Is" Coleman seems inclined to move SATs away from validity and evidence, it's worth the ACT's time to toot their own trumpet of obviousness.

Now, if you think any of that qualifies as a "report" and not "advertising copy," you are not exceeding expectations in your applying of critical thinking. But that's okay, because even Caralee Adams writing about the report at EdWeek, fails critical thinking regarding the meat of this report.

She takes away exactly what she was supposed to-- students who take lots of ACT courses do better in college, therefor, the ACT courses must have prepared them super-well for college.

Why is it that education "research" is so riddled with an inability or unwillingness to distinguish between correlation and causation?

If we were going to design an experiment to determine the effectiveness of a product (say, New Shiny ACT Creme) we would need to first start with subjects who were close to identical. So, let's start with 200 fifteen-year-olds who have similar amounts of acne, give 80 of them New Shiny ACT Creme, 80 of them nothing, and 40 of them mayonnaise. After several months, we'll check to see how the acne looks.

That is not what this "study" has done.

Instead, this study says, "We want the teens who think they have the best complexion to use ACT Creme. We'll let them try it, and in a few months, we'll check to see how they look."

What the ACT report seems to show is not terribly remarkable. Apparently the same high-achieving, hard-working, top-ability, ambitious students who do well in college also seek out the challenging courses in high school. Put another way, the students who value college success and take steps to achieve it in college, also take steps to achieve it in high school. In other news, the sun is expected to rise in the East tomorrow.

I'm not suggesting that the ACT is a scam and terrible waste of time. Like AP courses and the SAT, the ACT clearly has some benefits for some students, and that's a Good Thing.

But we really need to stop looking at these groups as if they are philanthropic organizations trying to make the education world a better place out of the goodness of their highly objective hearts. The ACT is a business, and any "report" they offer about their product is most reasonably viewed as advertising copy, not scientific research.

News organizations do not run stories with headlines like "Researchers Discover Frosted Flakes Are Grrrreat" or "Study Shows Men Who Wear AXE Get Laid More Often." At least not yet. In the meantime, let's keep those critical thinking caps firmly attached.

Objectivity Is a Unicorn

Objectivity is a unicorn. It's inspiring to believe in, pretty to create pictures of. Some folks love the idea of objectivity so much that they dress up horses or try their hand at photoshop. But at the end of the day, it doesn't actually exist.

Nevertheless, we continue to enshrine the idea of objectivity in places where it does not exist. Writers have repeatedly reminded us, for instance, that internet search engines are not objective. This article from Michael Kassner at Tech Republic provides a good summary of the basic arguments. When you google something, you do not pull up some objective summary of what the internet contains, but a list that has been weighted by programmers who judge that certain factors should be considered (including a cyber-reading of you and your own proclivities). Google results are just as subjective as if they were compiled by some guy and his buddy making their best guess about what you want and what you should see.

But we really like to believe in Objective Facts, and in particular the notion that certain instruments provide such facts for us.

Some simple instruments do provide some rudimentary objective data. A measuring stick will provide an objective indication of length of an object. But the stick might be measuring meters or feet, and it might include hatchmarks for very small units or not, and if you're using to measure something that's not exactly straight, you will have to make a judgment about how do that. Even the simplest measurement includes subjective judgments. And that's working with objects, not humans.

Complex measures are mostly subjective judgments, whether we are measuring all the articles on the internet and how well they match your search terms, or we are measuring how "college and career ready" your eight year old child might be.

We keep talking about standardized tests as if they are objective measures of... well... anything. They are not. A standardized test is the product of the individual personal judgment of the test writers, who have their ideas about which specific bits of knowledge and skill should be tested and who make their own judgments about what exact tasks would measure those bits. They may claim that research backs up some of their choices, but research is itself the result of individual subjective judgments and choices made by the researchers deciding what to measure, how to measure it, and how to interpret the data they generate. In some cases, such ads David Coleman's reconfiguration of SAT vocabulary, they simply baldly say, "I think we should do this, not that."

Track the elements of standardized tests in any direction you wish; you will soon arrive at human beings making personal subjective judgments about how the test should work.

Reformsters keep talking about the use of testing and data as if it will result in replacing the varied subjective judgments of a teacher with the pure objective results of the testing. No such thing is true. What they seek to do (whether they understand it or not) is replace the judgment of the teacher in the classroom with the subjective judgment of the persons who make the test.

When someone claims "this test is an objective measure of a students language use ability," they are wrong. The test is, at best, a pretty good subjective measure of some tasks that some test-writer guys believe probably indicate language use skill. It is no different from having some person come into the classroom and say, "I'm going to sit and talk to Pat for a couple of hours, and then I'll tell you how good a reader I think he is."

It is not humanly possible to remove subjective judgment from education (or, arguably, anything at all, but let's narrow our focus for now). Never even mind the question of whether or not we should-- it cannot be done.

How do we deal with the inherent subjectivity?

The problem with subjectivity is that it solves problems within a personal framework. We operate in our own little bubble or silo and we define problems and search for solutions based on what we see and know inside that little framework. The trick is to temper subjectivity by expanding that framework. We need to see and know more.

We expand the framework with professional knowledge. We train teachers to understand pedagogy and subject matter so that their instructional judgment is not based on considerations like "What kind of mood am I in" or "Do I like this kid."

We expand the framework with personal relationships. One of the terrible lies of the cult of objectivity is that we can make good decisions about students without knowing a thing about them. Management schools recommend that managers not live in the same communities as their employees, so that relationships are not part of their framework.

Baloney. One of the marks of a good decision is that you can talk about it out loud with the people who are affected by it right there in the room with you. When our framework is expanded to include the people who are part of the choices and the results, our subjective view of the situation is more complete, more useful to people beyond ourselves.

Reformsters have done their damnedest to keep teachers, students, and much of accumulated wisdom of American public ed outside of their framework. This leads them to say naive things (Hey, did you guys know that water is wet) and stupid things (Hey, if you tried hard, I bet you could build a house out of water). It doesn't ever lead them to say objective things.

They claim that their goal is to inject objectivity into the education world, when in fact they're simply trying to impose their own subjective judgments in place of others'. When someone wearing a white hat rides into town on a unicorn, you'd better check to see how the horn is attached.

I've Launched at EdWeek

As noted a while ago, I have taken on blogging for Education Week, and the new blog launched a few days ago.

The EdWeek blog will be called "View from the Cheap Seats," and I'll be posting there at least weekly. I even get a cool graphicky thing for the header!I am excited about the potential for reaching a new audience. The first post is about the absence of the teacher voice from education debates. Feel free to stop over, read it, and shower them with demands that I be paid a zillion dollars. Or just enjoy it.

Superintendents Speak Up

On the first day of school, my wife's superintendent got choked up.

He was delivering the usual kick-off speech, and she said he started to talk about testing and numbers and the students. He reminded his staff that students were not just test scores, not just a number, and that the work they did as teachers was so much more than could be measured by numbers. It looked, she said, as if he was on the verge of tears.

My wife's superintendent is my former principal. He's a good man and a fine educator. And apparently he's done pretending that chasing test scores is a good way to run a school district.

This may well be the fall we remember for the number of school district leaders who have finally had enough and begin to speak up.

Examples abound. In Peru, Illinois, Superintendent Mark Cross sent out a letter that said in part

Unfortunately, there are many federal and state education initiatives that can very much be a distraction from what matters most. These initiatives are based on good intentions and are cloaked in the concept of accountability, but unfortunately most do little to actually improve teaching and learning. Most are designed to assess, measure, rank and otherwise place some largely meaningless number on a child or a school or a teacher or a district. That is not to say that student growth data is not important. It is very critical, and it is exactly why we have our own local assessment system in place. It is what our principals and teachers use to help guide instruction and meet the needs of your kids on a daily basis. In other words, it is meaningful data to help us teach your child.

He makes this commitment

This is why I wanted to let you know that we will not be talking to you that much about the PARCC assessment or Common Core or other initiatives that have some importance, but they are not what matters most to us. YOUR CHILDREN are what matter most, and we believe that kids should be well-rounded, with an emphasis on a solid foundation for learning across all subjects by the time they get to high school and later college. We believe that kids need to be creative and learn to solve problems. We believe that exposure to music and art, science and social studies, physical education and technology and a wide variety of curricular and extracurricular activities will serve them very well as they grow into young adults.

And he delivers this pointed (if grammatically suspect) indictment

The state and federal government have failed epically in their misguided attempts at “reforming” public education. Public education does not need reformed. 

Superintendent Cross is, of course, not alone. In Washington State, the education system has lost its waiver from Arne Duncan because the state legislature would not implement the federal Department of Education's preferred method of teacher evaluation. So Washington schools are operating under No Child Left Behind, which means that all schools not meeting requirements that 100% of students be above average (aka "all schools") are failing. You may recall that one of the punishments for failing schools under NCLB is that they must send out a "We Are Failing" letter to the public.

Superintendents in WA have sent their letter. However, 28 superintendents wrote a letter which includes the observation that the label under NCLB is "regressive and punitive." The basic layout of the letter is "The feds say we have to tell you this, which we are now doing, however you should know that the feds are full of it, their policies are stupid, and we are educating your students pretty well, thank you very much."

And as I noted here yesterday, the Board of Education for the entire State of Vermont has adopted a resolution calling out the feds on their stinky testing requirements.

The tone in administrative offices is continuing to shift. Ten years ago there was a lot of kool-aid drinking. Then we had "Well, it's the law." Then we had fatalism and resignation, "Well, let's just do our best work and hope that these tests take care of themselves and somehow things work out." What we have always needed are administrators to stand up and say clearly, "This is not right. It's not right for us, and it's not right for our students."

I know there are still districts and entire states where the school leaders have not only drunk the kool-aid, but are selling it themselves out of the back of a van. But it's heartening to see and hear more who are willing to speak out in a meaningful and public way. Duncan is clearly trying to stem the tide with his waiver-waiver, the offer of "Look, we'll just wait a year and then we'll punch you in the face." But postponing a stupid thing does not make it any less stupid, and in the meantime, more and more people are starting to point out that the emperor's clothes (which are no longer new) are woven out of air and empty promises.

Vermont BOE Hammers Fed-Style Testing

With states like North Carolina and Florida doing their best to bury public education and dance on its grave, it's nice to see some states can still stand up for their schools.

Earlier this week, the Vermont State Board of Education adopted a statement and resolution on assessment and accountability. It's worth a read, but let me hit the highlights for you.

The Board starts by recognizing that uniform standardized tests can be a useful tool for helping schools chart a path toward successful delivery of well-designed standards. And then comes the pivot-

Despite their value, there are many things tests cannot tell us. Standardized tests like the NECAP and soon, the SBAC, can tell us something about how students are doing in a limited set of narrowly defined subjects overall, as measured at a given time. However, they cannot tell us how to help students do even better. Nor can they adequately capture the strengths of all children, nor the growth that can be ascribed to individual teachers. And under high-stakes conditions, when schools feel extraordinary pressure to raise scores, even rising scores may not be a signal that students are actually learning more. At best, a standardized test is an incomplete picture of learning: without additional measures, a single test is inadequate to capture a years’ worth of learning and growth.

Unfortunately, the way in which standardized tests have been used under federal law as almost the single measure of school quality has resulted in frequent misuse of these instruments across the nation.

In order to avoid that sort of foolishness getting loose in the Green Mountains, the Board lists eight guiding principals for the appropriate use of standardized tests.

1) The proper role of large scale tests must be stated before giving the test, and that use must be demonstrated as scientifically and empirically valid. That includes proof that the test can predict performance on "other indicators we care about, including post-secondary success, graduation rates and future employment." And you can't use the test all by itself-- mix it up with other measures.

2) Public reporting. Schools need to do that, but they need to report a wide variety of indicators that give a full picture of what they're doing.

3) Judicious and proportionate testing. Reduce the amount of time on summative and standardized testing. The feds should back off on multiple subject testing grades 3-8 as well as high school (so, you know, all of it). "Excessive testing diverts resources and time away from learning while providing little additional value for accountability purposes."

4) Test development criteria. Any big standardized test used in Vermont needs to be built in accordance with principles of American Educational Research Association, National Council on Measurements in Education, and the American Psychological Association.

5) Value-added scores. Near as we can tell, these are crap. We will not be using them in Vermont "for any consequential purpose."

6) Mastery level or Cut-off scores. This whole paragraph is pretty awesome.

While the federal government continues to require the use of subjectively determined cut-off score, employing such metrics lacks scientific foundation. The skills needed for success in society are rich and diverse. Consequently, there is no single point on a testing scale that has proven accurate in measuring the success of a school or in measuring the talents of an individual. Claims to the contrary are technically indefensible and their application would be unethical.

7) Use of cut scores and proficiency categories for reporting purposes. The fed since NCLB was born have required this. Here's a list of ways in which it has been documented to create negative effects. We'll keep doing what the letter of the law requires, but it's crap.

8) Just as the state high quality education, the federal, state and local governments must provide adequate resources to get the job done. If you're going to demand a report on the quality of the school's work, demand a report on the sufficiency of the resources provided to the school "in light of the school's unique needs."

These are followed by several whereas's that note that the the nation's have been spending an ever-increasing amount of time and money on testing of a sort and in ways that are known to be No Damn Help to anyone and wrapping up with

WHEREAS, the culture and structure of the systems in which students learn must change in order to foster engaging school experiences that provide joy in learning, depth of thought and breadth of knowledge for students [emphasis mine, because, damn, wouldn't you like to see that in every school's mission statement!]

And then we get the Be It Resolved portion 

-- The Secretary of Education should re-examine the accountability system and come up with one that sucks less (I'm paraphrasing)

-- Congress should get off its collective keister and amend the ESEA

-- Other state and national groups should join us in this

I grew up just across the Connecticut River from Vermont, playing in my front yard and looking at the big beautiful mountains, but I have never loved Vermont more than I do reading this resolution. If you see Vermont today, give it a big hug for me. And send Arne Duncan a copy of their resolution. It's true these are just words-- but they are damn fine words.

Another Solution: ESEA

There is, of course, another way out of this.

The tightly wound spring that keeps Race to the Top and waivers (RttR Lite) ticking away is the ESEA. Instead of dealing with the federal mandate-ish sort-of-regulations that have made Common Core and high stakes testing and data collection the kind-of-law of the land, we could address the underlying mess.

The ESEA was first passed in 1965, and periodically is up for "re-authorization" which means the current Congress gets to monkey with it. In 2002, a bipartisan group under George Bush rewrote it into No Child Left Behind. ESEA was due to be re-authorized in 2007, but that ugly step-child of a law was already so toxic that Congress couldn't bring itself to do anything more than sputter and posture. And so ever since, ESEA has been ticking away. (You can get a more complete run-down of the long convoluted mess here.)

Race to the Top and RttT Lite are simply end runs around ESEA, and the only reason anybody bothers to mess with the four federal requirements (CCSS-like standards, high stakes tests, teacher evals linked to HST, and data collection) is because right now, as we sit here, every public school in this country is in violation of NCLB (well, unless you have 100% of your students above average, in which case your school mascot is probably a unicorn).

That is why Washington State schools are being required to send out "We are failure" letters to their parents-- because they lost their waiver because they wouldn't tie teacher evals to test scores, and so now they are back to living under the reality-defying requirements of NCLB.

So we could pull the plug on the whole reformy mess by simply doing what we were supposed to do seven years ago, and re-authorize (and re-write) the ESEA. All it requires is for members of Congress to show their political courage and commitment to properly educating America's children through a public school system. And after they do that, we can all celebrate by riding around on our unicorns. Heck, all we would need if for Congress to do its job and not impose more stupid ed reform rules. How hard could it be?

There is one other possibility, and it could make the next Presidential election interesting. Because the anti-reform Presidential candidate could say, "The Obama waivers are illegal, and the first thing we'll do in office is throw them out." If that happens, Congress would be under tremendous pressure to get on their unicorns and Do Something. Of course, they were under that sort of pressure right up until the point that the waivers were conceived.

This is one huge argument against having the federal government regulate and control public education in this country-- because when they break the system, they break the entire system.

Granted, the re-authorization of the ESEA is a big unicorn hunt. But many of the goals that are proposed, on all sides of the education debates, are unicorn hunts. So let's no overlook the hunt for the biggest, most magical unicorn of all.

Duncan Tries To Hear Teachers

US Secretary of Education Arne Duncan is here with some back-to-school blogging to assure folks that he is totes listening to somebody. His back-to-school conversation comes with two messages.

First, he wants to send out a big thank you to all the folks who helped create some super-duper data points last year-- specifically, the high school graduation rate and the college enrollment rate. I might be inclined to wonder about A) the reality behind those juicy stats and B) what it actually means. But Arne knows what it means:

These achievements are also indications of deeper, more successful relationships with our students. All of us who’ve worked with young people know how much they yearn for adults to care about them and know them as individuals.

Reading Duncan's words always induces an odd sort of vertiginous disorientation as one tries to take in the huge measured-in-light-years distance between the things he says and the policies he pursues. What in the four requirements of Race to the Top would possibly indicate that Duncan's administration is pursuing policies that develop these kind of relationships or satisfy these alleged yearnings? Is it the way teachers fates have a federally mandated dependency on student test scores? Is it the sweet embrace of one-size-fits-all national standards? Maybe it's the grueling program of punishing tests.


Which brings us to the second message.

Duncan says he's been having many many conversations with teachers, "often led by Teacher and Principal Ambassador Fellows" (those teachy folks who have been carefully vetted and selected by the DOE, so you know they're a real collection of widely varied viewpoints). And in those conversations, he's picked a little something something about standardized testing. Which he still thinks is basically swell.

Assessment of student progress has a fundamental place in teaching and learning – few question that teachers, schools and parents need to know what progress students are making.

Also, a bicycle, because a vest has no sleeves. Sure, classroom assessment is important. But recognizing that importance has nothing at all to do with making a case for standardized testing, particularly of the current brand. "Medicine is important" is true, but it's no justification for jamming aspirin into somebody's compound fracture.

Anyway, Arne has picked up three specific concerns:

1. It doesn’t make sense to hold them [educators] accountable during this transition year for results on the new assessments – a test many of them have not seen before – and as many are coming up to speed with new standards.
2. The standardized tests they have today focus too much on basic skills, not enough on critical thinking and deeper learning.
3. Testing – and test preparation – takes up too much time.

Duncan is shocked-- shocked!!-- that anyone would think it's a good idea to make a high stakes test the measure of student achievement or teacher effectiveness.  "Growth is what matters. No teacher or school should be judged on any one test, or tests alone –" And here comes the vertiginous woozies (dibs on this as a band name) again, because that would be a heartening quote if it did not come from the very same office which decreed that by order of the federal government high stakes tests must be used as a measure of student achievement and teacher effectiveness. Duncan is talking about this test-based evaluation of students and teachers as if it just spontaneously occurred, like some sort of weird virus suddenly passed around at state ed department sleepover camp, and not a rule that Duncan's office demanded everyone follow. Has Duncan forgotten that he just made the entire state of Washington declare itself a Failing School Disaster Zone precisely because they refused to use high stakes tests as a measure of student achievement and teacher effectiveness?

No test will ever measure what a student is, or can be. It’s simply one measure of one kind of progress. Yet in too many places, testing itself has become a distraction from the work it is meant to support.

You know what one might conclude from that? One might conclude that the testing is a doing an ever-so-crappy job of supporting "the work it is meant to support."

States will have the opportunity to request a delay in when test results matter for teacher evaluation during this transition. As we always have, we’ll work with them in a spirit of flexibility to develop a plan that works...

I would like to check with someone from Washington to see what it feels to be flailed with that spirit of flexibility. But Duncan is opening the door to states postponing the most painful consequences of testing for one year, because, you know, teachers' voices.


Anthony Cody has correctly pointed out that one other voice has spoken up in favor of this-- the voice of Bill Gates. Unfortunately, we'll never know for certain how this all played out. Did Duncan decide to obey the Call of Gates and try to use it to mollify teachers? Is the Voice of Gates so powerful that it blasted the wax from Arne's ears and he could hear teachers finally? Is he bending to political realities, or trying to do damage control.

I have a question I'm more interested in-- what difference will a year make?

Duncan seems to think that some time will improve the tests themselves.

Many educators, and parents, have made clear that they’re supportive of assessment that measures what matters – but that a lot of tests today don’t do that – they focus too much on basic skills rather than problem solving and critical thinking. That’s why we’ve committed a third of a billion dollars to two consortia of states working to create new assessments that get beyond the bubble test, and do a better job of measuring critical thinking and writing.

Never going to happen. National standardized test means test that can be quickly checked and graded at large scale and low cost (or else the testmakers can't profit from it). The college board has had decades to refine their craft, and their refined craft looks like-- a bubble test.



As far as Duncan's other concerns go-- a year will not matter. Much of what he decries is the direct result of making the stakes of these tests extremely high. Student success, teacher careers, school existence all ride on The Test. As long as they do, it is absurd to imagine that The Test will not dominate the school landscape. And that domination is only made worse by the many VAMtastic faux formulas in circulation.

Too much testing can rob school buildings of joy, and cause unnecessary stress. This issue is a priority for us, and we’ll continue to work throughout the fall on efforts to cut back on over-testing.

Oh, the woozies. Duncan's office needs to do one thing, and one thing only-- remove the huge stakes from The Test. Don't use it to judge students, don't use it to judge teachers, don't use it to judge schools and districts. It's that attachment of huge stakes-- not any innate qualities of The Test itself-- that has created the test-drive joy-sucking school-deadening culture that Duncan both creates and criticizes. If the department doesn't address that, it will not matter whether we wait one year or ten-- the results will be the same.