DeVos Becomes an Actual Punchline

From the moment she became late night sketch fodder, Betsy DeVos has become something new in the world of education secretaries.

I doubt that the average citizen could name five education secretaries-- maybe not even three. But they know who Betsy DeVos is. She's been Kate McKinnoned multiple times on Saturday Night Live.  She's been a punchline repeatedly on late night tv She's used as an ancillary punchline-- in other words, not as the main part of a joke meant to skewer her, but as a useful barb to skewer someone else.

And the DeVos jokes just keep spreading. If you like to waste time sample the culture on YouTube, you may be familiar with the "Everything Wrong With..." series of videos that snarkily pick part various movies. Yesterday they released their clip picking apart The Shape of Water, and smack in the middle of it is a DeVos joke. If you want to see it for yourself, skip to about the 7:13 mark.

The joke is pretty simple. They run a quote from the movie in which the Evil Russians say, "We don't need to learn, we need Americans not to learn."

Then the voiceover artist simply says "Betsy DeVos."

DeVos is literally a punchline, one considered so reliable that there's not even an explanation attached. It's just one line in one video, but still....

I've written before about how I think it's a disservice to the cause to dismiss DeVFos as a dope-- she's no dope. But there's definitely something curious going on when 1) regular citizens know who the secretary of education is and 2) that secretary is widely understood to be anti-education-- even by people whose understanding of the issues is, shall we say, a bit shallow.

But this is where we are. The United States Secretary of Education is a punchline. Good luck to us all.

Ivanka, Vocational Education, and the Death of Irony

The crunch you just heard may be irony collapsing under the weight of a recent Ivanka Trump op-ed for Fox News about skill-based education.

This woman did not go to welding school

I've watched the growing enthusiasm for what we now call Career and Technical Education (CTE) with some amusement. My own district helped launch a county-wide Vocational-Technical School  almost fifty years ago, and the benefits have always been obvious. Students take core classes at their "home" school for a half day, then travel to what is now called the Venango County Technology Center to learn skills from operating heavy machinery to providing home health care. If I needed to have body or mechanical work done on my car, I could take it there and have students fix it up. Other students in the building course work every year; that house, once completed, is sold and moved to its new location. Welding, coding, protective services, and dental assisting are also in the wide range of offerings there.

I've watched the benefits of the school throughout my career. The CTE students are more focused, more goal-oriented, and best of all, have a clear understanding of the relationship between making an effort and getting a result. Studies like the one recently covered by Matt Barnum for Chalkbeat provide zero surprises for those of us for whom CTE is old hat-- the students are more likely to graduate, do better in school generally. They can strain the disciplinary boundaries some times, which is understandable-- it has to be jarring when one minute you are outdoors, operating a piece of heavy equipment and an hour later you're sitting in a desk, required to ask permission to get up and sharpen a pencil.

The benefits are so obvious that it hadn't really occurred to me that many school districts had no such facility.

But hey-- what could give CTE a bigger boost than an endorsement from Ivanka Trump.

Skill-based education is crucial to putting more Americans on a path to promising careers – and filling the jobs of the future, as well as those that are vacant today. Yet for too long, we have failed to recognize the importance of practical, skill-based learning.

On the one hand, sure. On the other hand, this headline....

Skill-based education is crucial to putting more Americans on a path to promising careers.

Because, yeah, Ivanka spent a lot of hours in school learning useful skills so that she could find a decent job. Just like her father.

This is the part where irony is just crushed to death; a woman whose profession is "Heiress" or possibly "Assistant Grifter" wants to extoll the virtues of good, honest, useful skills. She is not wrong about the value of these skills; I'm just not convinced she's the one to deliver the message.

Ms. Trump cites the large number of unfilled jobs as support for her point, and here she is also not wrong-- but like many folks who throw the stat around, she's gliding past an important part of the problem, a part that is captured in he subheading for this CNN/Money article last fall noting the record number of job openings:

American employers are trying to hire, but they can't find the right workers for the right price.

The number was 6.2 million job openings, with 1.2 million in healthcare and education. But let's look at that last prepositional phrase-- "for the right price." Right price for whom? Are we talking about the employers want to pay, or the amount that it will take to make the job attractive enough?

Once again, folks are choosing to remember how the free market works only when it suits them. The true value/price of a commodity is what it takes to get that commodity. Or as I've often said, just because you can't buy a Porsche for $1.95, that doesn't mean there's a car shortage. If you can't buy labor for $8.00 and hour, the invisible hand demands that you offer $9.00 or $12.00 or $20.00 or whatever it takes to get someone to sell you the labor that you want to buy.

In other words, "the right price" is not what you want to pay-- it's what the market dictates that you have to pay.

As when we discuss getting US works to compete with workers in China or India, I'm left with the impression that we're not really talking about job skills-- we're talking about the "skill" of being willing to work for below-market wages. This is not a "skill" that can, or should, be taught in a CTE program. My students work damned hard to master their crafts, and if employers want the benefit of those skills, they should be willing to fork over a damned good wage to those students. If you want to promote American workers with American skills, then offer to pay them instead of having your merchandise made for cheap in China; offering well-paying jobs would be far more powerful than offering weak op-ed essays.

Talking about training and skills and job preparation and career readiness is all meaningless if the actual problem is that corporate leaders really want to be able to get adequate skills for sub-adequate pay. Corporations could solve all of their workforce problems by offering jobs that paid really well. If calling for more CTE and career training is really just code for "we want our labor force to be happy being underpaid," then they (and the President's daughter) are just wasting everyone's time.

Who Will Take the D out of DFER?

DFER stands for Democrats for Education Reform, a group that has the distinction of possessing a name in which none of the words are accurate.

The Democrat part is a fun piece of trivia. One of the key founders of DFER is Whitney Tilson, a big time hedge fund manager (you can read more about him here). Leonie Haimson has a great quote from the film version of Tilson's magnum opus about ed reform, "A Right Denied," and it's a dream of mine that every time somebody searches for DFER on line, this quote comes up.

“The real problem, politically, was not the Republican party, it was the Democratic party. So it dawned on us, over the course of six months or a year, that it had to be an inside job. The main obstacle to education reform was moving the Democratic party, and it had to be Democrats who did it, it had to be an inside job. So that was the thesis behind the organization. And the name – and the name was critical – we get a lot of flack for the name. You know, “Why are you Democrats for education reform? That’s very exclusionary. I mean, certainly there are Republicans in favor of education reform.” And we said, “We agree.” In fact, our natural allies, in many cases, are Republicans on this crusade, but the problem is not Republicans. We don’t need to convert the Republican party to our point of view…”

That's the beauty of a label like "Democrat"-- anybody who wants to use it can use it. Now, there's little doubt that the corporate wing of the Democratic Party has long known that when it comes to education, they were largely indistinguishable from the GOP. At this stage, we can talk about the Bush-Obama education reform program as the one seamless thing it always was, and for some folks, it's a useful way to frame ed reform to look as if the DeVos agenda is radically different (spoiler alert-- it's not).

But recently something unusual happened. Last weekend, the Colorado Democratic state assembly overwhelmingly rejected DFER's agenda and told them to take the D out of their name. Here's the platform amendment they pushed through:

We oppose making Colorado’s public schools private or run by private corporations or becoming segregated again through lobbying and campaigning efforts of the organization called Democrats for Education Reform and demand that they immediately stop using the party’s name Democrat in their name.

The battle, as Erica Meltzer wrote for Chalkbeat "revealed a growing divide among party activists and establishment politicians on education policy." Colorado's DFER chapter is loaded with establishment Dems, while the roomful of actual Democrats was not. Speakers against DFER included people like Vanessa Quintana, whose own high school education had been creatively disrupted by ed reform in the state:

“When DFER claims they empower and uplift the voices of communities, DFER really means they silence the voices of displaced students like myself by uprooting community through school closure,” she told the delegates. “When Manual shut down my freshman year, it told me education reformers didn’t find me worthy of a school.”

Only two people spoke up in favor of DFER, but some DFER supporters are not too concerned. You have to read all the way to the end of the Chalkbeat article, but once you get there, you see the strategy that DFER and reformsters like them have always focused on:

Van Schoales, a DFER board member and CEO of A Plus Colorado, an education reform advocacy group, called it a “symbolic attack,” but he believes support for policies like school choice and charter schools remains strong among Democratic elected officials.

In other words, as long as they keep a hold on the politicians, it doesn't matter what actual Democratic Party members want. This is a symptom remaining from when, as Jennifer Berkshire puts it, "education reform ate the Democratic Party" and somehow Democrats became one more anti-union, teacher-bashing group of public school privatizers. Will the party figure out where it should stand on education policy? The time is ripe for them to get a clue, and with any luck, Colorado is a harbinger of things to come.

Can the Reading Pendulum Be Swinging?

Two articles and some twitter chatter do not a trend make, but a guy can hope.

One of the worst trends to come out of modern reform has been the repackaging of reading as a content- and context-free series of skills, a series of tasks to be completed in a vacuum. This has always struck me as twelve kinds of wrong. Like preparing an athlete by having them practice movements in a padded room without a team, ball or field. Like having a band musician practice alone with just a mouthpiece. Like trying to tune the audio characteristics of a set of speakers inside an airless chamber. I could go on all day with the different kinds of crazy this is.

And I must be clear-- not all reformsters have been on this bus, ever. Robert Pondiscio hollered for years about "57 most important words" in the Common Core. Here they are:

By reading texts in history/social studies, science, and other disciplines, students build a foundation of knowledge in these fields that will also give them the background to be better readers in all content areas. Students can only gain this foundation when the curriculum is intentionally and coherently structured to develop rich content knowledge within and across grades. Students also acquire the habits of reading independently and closely, which are essential to their future success.

These words appear in a "note on range and content of student reading." They don't carry a great deal of weight compared to, say, the 43-page appendix that spends so much time on "text complexity,"  a term tied to the mechanical content-free conception of reading, suggesting that reading levels are just a matter of how tricky the decoding puzzle is (but unrelated to the actual content itself).

Podiscio and I long disagreed about the importance of those 57 words. The argument of many rich-content supporters of CCSS has always seemed to me to boil down to, "This must be crucial, because without this, the Core is bunk." My argument is that the 57 words are a minor cosmetic adjustment, a feeble attempt to patch a gaping hole, and the Core is, in fact, bunk. The most telling detail-- "rich content knowledge" appears nowhere on the CCSS-related Big Standardized Tests. As I've pointed out a million times, I can prepare my students to get good scores on the BS Test by spending the whole year reading excerpts from the newspaper and practicing BS Test-style multiple choice questions (and never learning anything about writing, but that's a gripe for another day).

Not only do the BS Tests not respect rich content knowledge-- they bend over backwards to negate it, deliberately choosing test items whose content  is highly unlikely to intersect with student prior knowledge (elementary test excerpt about Turkish village economics, anyone). BS Tests are designed to be the higher-complexity version of the dreaded DIBELS test, a test for young readers that tests their reading skills by asking them to read words that aren't actually words (because that way they can't "cheat" by already knowing the word and recognizing it).

But while Pondiscio and I and others have long disagreed about the Core (and charters and a bunch of other stuff), we agree about the importance of content when it comes to reading. Knowledge matters. Content matters. To state what seems obvious to me, it's hard to read or learn to read if you don't know much. It's easier to read or learn to read if you have a wealth of background knowledge. New learning is most easily acquired when it can be connected to old knowledge. So if you want to teach children to read, building up their storehouse of prior knowledge is a critical-- maybe the most critical-- thing you can do to build a foundation.

This point of view has never gone away, but it seems to be gaining traction lately.

Note, for instance, this piece from Louisiana's education chief and committed reformster John White. White's point is that the NAEP is a faulty test because it doesn't measure "what students know."

The trouble is that by not requiring knowledge of any specific book or facts, reading tests have contributed to the false impression that reading is mainly about having skills such as being able to summarize, and not about background knowledge. Walk into many English classrooms today and you will see students capably identifying an article’s main idea. But you’re less likely to find students learning the historical context for a novel or discussing the novel’s broader meaning. By not requiring knowledge, tests create no incentive for particular knowledge to be taught.

This is not fair to adolescents, who need knowledge to become effective adult readers. It’s particularly not fair to students from historically disadvantaged backgrounds, whose summer breaks rarely involve trips abroad or afternoons at museums, and who are thus at a disadvantage on any test that, whether it’s acknowledged or not, measures background knowledge. And it’s not good policy for a nation struggling with the influence of falsified news reports over its citizenry.

Meanwhile, Natalie Wexler took to the pages of the Atlantic to watch a panel ask (and answer) the question, "Why American students haven't gotten better at reading in twenty years." The answer again addresses the very basis of reading instruction:

On a daily basis, teachers have their students practice skills and strategies like “finding the main idea” or “making inferences.” And teachers select books that match the given skill rather than because of the text’s content. Rarely do the topics connect: Students might read a book about bridges one day, zebras the next, and clouds the day after that.

Cognitive scientists have known for decades that simply mastering comprehension skills doesn’t ensure a young student will be able to apply them to whatever texts they’re confronted with on standardized tests and in their studies later in life.

Wexler is only partly right-- students are more likely to be reading short excerpts of those books, rarely reading any entire works at all. But yes-- we've known for a while that this approach is backward. If you want to check out a golden oldie, take a look at the Recht and Leslie baseball study, in which it turns out that low reading skills plus high knowledge beats high reading skills and low knowledge. Here's the chart:

The panel included Dan Willingham, another writer I don't always agree with, but who gets it when it comes to the importance of content.

Current education practices show that reading comprehension is misunderstood. It’s treated like a general skill that can be applied with equal success to all texts. Rather, comprehension is intimately intertwined with knowledge.

As Wexler puts it:

The best way to boost students’ reading comprehension is to expand their knowledge and vocabulary by teaching them history, science, literature, and the arts, using curricula that that guide kids through a logical sequence from one year to the next: for example, Native Americans and Columbus in kindergarten; the colonial era and the American Revolution in first grade; the War of 1812 and the Civil War in second grade, and so on. That approach enables children to make sense of what they’re learning, and the repetition of concepts and vocabulary in different contexts makes it more likely they’ll retain information. Not to mention that learning content like this can be a lot more engaging for both students and teachers than the endless practice of illusory skills.

Yes, that, please.

Why might the pendulum finally swing back from reading "skills"? The answer is hinted at in the two articles [update: and you can add this one to the list]-- White wrote to get out in front of bad test results, and Wexler wrote to explain them. We've been doing reading "skills" for almost two decades, and it simply hasn't worked. Not only that, but it's been shown to not work by tests designed by the principles valued by reading skills fans. In other words, the set up the game according to their preferred rules, and they are still losing.

And finally some folks are asking, "What else could we be trying?" And lots of folks on all sides of the education debates already know the answer.

Content-based reading instruction will never be clean or easily implemented. For one thing, as soon as we start trying to come up with lists of content that "should be" taught, huge fiery debates will break out, and they will never, ever be settled. But we would at least be moving in the right direction (if we really want to get crazy, we could let students choose the content-based direction, thereby harnessing both the power of prior knowledge and the power of interest). Core knowledge cannot be tapped without unleashing and addressing political issues.

But we should not let disagreements about content scare us away from asserting clearly and effectively that content matters, and that it is properly the foundation of reading. Maybe if we keep insisting, the pendulum will finally swing.

Debates and Tribes

I'm not a huge fan of meta-posting on the subject of navigating the education debates, but it's unavoidable because the ed debates involve such a mish-mosh of tribes.

I agree with what X says about Y, but I'm worried he's just trying to create support for Z-- particularly because he belongs to Organization Q.

I've seen some variation on this a gazillion times-- from more than one side of the debates.

It's understandable because all sides have always consisted of alliances. It's people who can only really agree for half a sentence. For instance, common core opposition included a bunch of people who could start a sentence, "Common Core is a terrible idea, and therefor..." but then would finish the sentence with "...it should be abolished so that public schools can return to their proper mission" and "...it's the final proof that the government can't be trusted to run schools at all."

Sometimes the alliances are barely alliances at all. Lots of folks see the Big Standardized Test as a huge blight on public education, but while some folks want to see those tests abolished, or at least reduced in importance, others see the opposition to the BS Test as a good motivation for jumping into Personalized Competency-Based Learning Education, which is just another heaping pile of corporate reform.

Many people cope with this kind of confusion and tension by simply sticking to their tribe, or focusing on which tribe they oppose. If it comes from our team, it must be good, and if it comes from that other team, it must be bad.

But there are several problems with this.

First of all, tribalism leads to focusing not on the issues or the message, but on the classification of the source. Instead of listening to what Pat says and deciding whether it's bunk or not, I spend my time trying to suss out which team Pat belongs to.

Second, and perhaps worse, it leads to people agreeing with really dumb things just to stay on the right "side." This is the current problem of many so-called conservatives-- they've defined conservatism or "the right position" as anything that makes liberals upset, which has led to sudden bizarre changes in direction like "Russia is swell" and "cheating on your third wife with a prostitute is totally okay."

Ultimately we end up with people hugely overthinking things to their own detriment. We are sitting in the living room of a house that is on fire, and we can see through the front door a swimming pool and an ambulance, and somebody is arguing, "How do we know they didn't set the fire on purpose so that we would go jump into the swimming pool which is actually filed with sharks we just can't see from here? Let's just stay here in the burning house. That will show them." Or even worse, "It was one of their people who yelled that this house is on fire, so I'm not convinced these flames are rea." This is crazy talk. If the house is on fire, get out. Once you're out, check the swimming pool out carefully and make your next choice accordingly.

Pay attention. Watch, listen, and think. Use every available piece of information to get a picture of the road ahead, but don't just make shit up, and don't just take somebody's word for it.

Most of all, keep your eyes on the big picture and the important issues. Don't put tribe ahead of that.

[Update: It seems worth adding that the impulse of many people in the comments section was to try to assign me to a particular tribe and then reject or accept what I said based on that assignment. That tribe assignment included the assumption that if I said something mean about conservatives, then I must belong to the anti-conservative tribe. None of that discussion really considers whether what I said was actually true or not, or if it could be negated by saying something similar about the other tribe.  Though, for the record, anyone who reads regularly would find plenty of criticism of the Democratic brand of misbehavior and the faux progressivism that masquerades as "liberal" school reform. I just happened to pick up the most recent phenomenon of conservatives selling their principles for a ride on the Trump train.

At the end of the day, that's another side effect of tribalism-- in addition to actual conservatives and liberals, we have a lot of people whose only real interest is self-interest, and they'll happily pretend to be a member of whatever tribe they think will further their cause.]

ICYMI: Finally Spring Edition (4/15)

Your reading for the week. Remember to pass along what you value. And don't forget to peruse the blogroll in the righthand column.

Dear Betsy DeVos

Marie Corfield responds to Betsy DeVos's comments about striking Oklahoma teachers. This is what it means to serve the public.

Mute the Messenger

From the Texas Observer-- here's what happened when a Texas professor spoke out against standardized testing. Pearson came after him.

What Teacher Strikes May Teach Democrats about Education Politics

The Democratic Party abandoned public education and the teachers who work there a while ago-- will the current wave of teacher strikes make them rethink their strategy?

Now Watch Republicans Blame Obama for Failed Education Reform

Ed reform isn't quite as beloved as it once was, and one sign of that is the GOP folks trying to pin it on Obama. Jeff Bryan as always brings the research and the insight.

Put Out To Pasture

Nancy Bailey walks us through all the Secretaries of Education, the policies they pursued, and where they are now. It's a useful, but not encouraging, history lesson.

Time To Ditch the SAT

From US News, another compelling argument against the old SAT warhorse.

Why the School Spending Graph Betsy DeVos Is Sharing Doesn't Mean What She Says It Means

Betsy DeVos has been using an exhibit from the Museum of Bad Graphs to make her point. Matt Barnum explains why it is misleading.

Why American Students Haven't Gotten Better At Reading in 20 Years

A compelling piece from the Atlantic making the not-new point that reading can't be taught as a set of discrete skills free of content or context.

Remembering the Holocaust

If there was ever an education story, this is it. A new study suggests that a whole bunch of Americans simply don't know about the Holocaust. Not deny it-- just don't know about it.

Billionaire Offers To Buy High School

If you missed this story, Valerie Strauss has the details. A Pennsylvania billionaire offered his alma mater $25 million-- if they'd put his name on the school, let him set some curriculum, and keep it all secret. It's a truly bizarre tale.

Wakanda Schools

Jose Luis Vilson finally saw Black Panther, and it gave him some thoughts about education and shared prosperity.

The Testing Charade: Buy This Book

This is probably going to be a long post, so let me get the most important parts out of the way first. The books is The Testing Charade: Pretending To Make Schools Better.

Daniel Koretz has published a book that gathers between two covers all the things wrong with the test-centered accountability under which we all currently suffer. His explanations are clear, and his illustrations are vivid. If you want to clarify your thinking about testing-- if you know something is wrong, but it's hard to wrap your head around-- you need this book. If a loved one or a colleague or an administrator asks you, "So what's the big deal, anyway? Why get upset over some simple standardized testing and accountability?" you need to hand them a copy of this book.

Buy this book.

Other things you should know up front. Koretz is the real deal. He is a widely recognized testing expert and scholar based at the Harvard Graduate School of Education. He is not some guy I found who presents my point of view; long time readers know that I am a standardized test radical (kill them all with fire) and Koretz is not-- he sees valuable uses for the big tests done right, and he pushes some ideas I don't agree with. Nevertheless, I'm telling you to buy this book.

Okay.

There are fourteen chapters in this book. I'm going to talk about all of them, with quotes, because Koretz says a bunch of important stuff. If any of this lights a fire in your brain, my advice is simple. Buy this book.

1: Beyond All Reason

Pressure to raise scores on achievement tests dominates American education today.

That's the first sentence of the first chapter, which looks at where we have arrived in the bizarre and extreme pursuit of school improvement via test scores. Koretz provides multiple examples of how this accountability approach has gone off the tracks, including tales of excellent schools given "failing" grades. The evidence of the failure of this system has, Koretz says, "been accumulating for more than a quarter century. Yet it is routinely ignored-- in the design of education programs, in public reporting of educational 'progress,' and in decisions about the fates of schools, students, and educators."

Don't make the mistake of thinking that these problems will disappear now that NCLB has finally been replaced. Test-based accountability was well established in this country before NCLB, and it will continue now that ESSA has replaced it. It is true that NCLB was a very poorly crafted set of policies-- a train wreck waiting to happen, some of us said when it was enacted-- and it did substantial harm....ESSA continues the basic model of test-based accountability, while returning to states just a fraction of the discretion they had in implementing this model before NCLB was enacted.

 Koretz notes that he believes that standardized tests-- properly used-- can be valuable. And while he is going to damn the current accountability system, he is not arguing against all accountability.

2: What Is a Test ?

Everyone knows a test when they see it. However, understanding tests is very different from recognizing them, and unfortunately, many of the people with their hands on the levers in education don't understand what tests are and what they can and can't do.

Koretz offers the useful analogy that a test is like a poll-- a small sampling of a much larger domains, and only useful if the small sample is properly related to the larger area. "The items on a test matter only to the extent that they allow us to predict mastery of the larger subject area from which they were sampled." Standardized tests are not good tools for measuring full mastery, and Koretz lays out the reasons.

First, standardized testing has inherent limitations. Things like complex analytical thinking and problem solving aren't best assessed with a standardized test. Second, test authors must make large numbers of decisions about what is and is not included in test items. These decisions, some deliberate and some not, narrow what the test actually tests. This point, for Koretz, is huge. The sampling decisions introduce error (as in "margin of" and not "oopsies"), The samples will be incomplete measures. And perhaps most importantly, the sampling creates perverse incentives.

High stakes testing creates strong incentives to focus on the tested sample rather than the domain it is intended to represent. If you teach a domain better-- say, geometry-- scores on a good test of that area will go up. However, if you directly teach the small sample measured by a particular test-- for example, memorization of the fact that vertical angles are equal-- scores will increase, often dramatically, but mastery of geometry as a whole will not improve much, if at all.

This is as good an explanation as I've seen of why teaching to the test is a bad idea.

3: The Evolution of Test-Based "Reform"

You know we're going back to 1983 and A Nation at Risk,  as Koretz tries to lay out how we arrived at the place where tests are the most central part of any school's everyday life. He notes that nations used as examples of test-based education are actually trying to back away from it, and that even the highest high-stakes testing countries don't do what we do, don't test so often, don't use test scores to evaluate teachers and schools.

He offers the term "measurement-driven instruction" which is shorthand for a world in which "improving performance on the specific test was to be the explicit goal, and higher-quality instruction would be the consequence. This was the tail wagging the dog."

This approach did not start with NCLB, but NCLB gave it national scale. And folks who thought Obama would provide relief "were quickly and sorely disappointed." Arne Duncan's big contribution was to get states to tie individual teacher evaluation to test scores, leading to "some of the most ludicrous uses of test scores." ESSA isn't going to help, other that in allowing some states to add broader measures to the mix.

Koretz gives test-based reformsters credit for good motives, but "Whatever their motives. the proponents were wrong. The reforms caused more harm than good."

Although he'll spend the rest of the book exploring the details, he gives the broad strokes here to explain why test-based reform has failed so badly. First, it "rewards far too narrow a slice of educational practice and outcomes." Second, it is too high pressure. And third is that it left almost no room for human judgment.

...teaching is far too complex a job to evaluate without any judgment, and many of the things we value most in schools aren't captured by tests.

4: Campbell's Law

The more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processe it is intended to monitor.

Well, you knew Campbell's Law would come up. But what you might not know (well, anyway, I didn't) is that Campbell actually predicted how the law would play out in educational testing:

But when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways.

For folks who don't quite get how Campbell's Law works, or why it applies, Koretz has some great examples of the law in action (my favorite: the old Soviet shoe factory story). And he tempers the observations about the Law in action in education by arguing that we have an obligation to consider the balance; some approaches will be more heavily hurt by Campbell's Law than others, and Koretz wants us to make sure that there isn't more damage than benefit.

5: Score Inflation

This was a helpful concept for me. For Koretz, "grade inflation" doesn't mean blowing up scores or moving the cut score goal posts-- it means raising test scores by techniques that do not actually raise the level that the tests purport to measure.

Koretz talks at length about the problems of trying to study test score inflation-- turns out that people who are politically invested in "good" results on the Big Standardized Tests are not interested in letting scholars study how inaccurate those good scores are.

Koretz addresses inflation throughout the book, and his view is nuanced-- on one end of the scale is flat-out cheating, where on the other end we have coaching. At one point, my bosses expected me to put up a poster of the "anchor standards" that would be tested. Would that count as an inflation factor? I'd say yes, because it was aimed at preparing for the test and not aimed at improving reading, writing, speaking and listening skills.

Koretz says that all you need to get inflated scores is a test that is predictable. If, for instance, you are poring through banks of old test questions in order to narrow down what kinds of questions your students need to be prepared for, that's causing score inflation. And if your administrators are requiring you to do that...well, they are part of the reason that this test-driven system is a failure. High stakes matter as a motivation to do test prep, but even in lower-stakes settings, Koretz found a culture of "applied anxiety" about test scores.

Beyond corrupting the scores of individual students through things like test prep, Koretz points out you can corrupt the group results as well. One way is to focus on bubble kids-- the ones who can be dragged across the cut score line. Another is to game the system by rigging which students are tested. And of course there's plain old cheating.

Koretz explains a less-obvious problem of inflation-- it varies from school to school and classroom to classroom, which means "we are identifying the wrong teachers, schools and systems as successes and failures." And it's also worth noting that low-income schools have a higher incentive to inflate, which in turn means that the students who arguably most need an "improved" education are instead getting the education most corrupted by the testing process. IOW, they most need a "real" education, and they are most likely to get a battery of test prep instead.

6: Cheating

By the end of NCLB, as many of us noted, there were only two types of schools-- schools that were failing and schools that were cheating. Measurement-driven instruction has created enormous incentive to cheat, and created some thorny ethical dilemmas. After all, which is worse-- cheating on the student score for a single BS Test, or allowing bad educational policy to take that child's school away from her?

Koretz has a whole batch of cheating tales here, and they are depressing, but necessary if we're going to understand what high-stakes testing pushes people toward.

7: Test Prep

There is no doubt that test-based accountability has resulted in a huge commitment of time and effort to test preparation, I can't be more precise, in part because people don't agree on the dividing line between test prep and regular instruction.

In fact, the blurring of that line is one of the sobering parts of this chapter,

Koretz lists three types of bad test prep-- reallocating time between subjects, reallocating time within subjects, and coaching.  Schools shift time, both between classes and within classes, to things that count on the test. No band for you, Pat-- you need to spend a period in ELA test remediation. Today, class, we're going to skip over chapter five, because that stuff is not on the test. Coaching is teaching tricks that have no use except for taking the test. For example, process of elimination, which seems harmless enough, but allows a student to get a correct answer when the student would never have been able to come up with the correct answer on her own. Life is not multiple choice; process of elimination is a skill that is only useful for taking the test.

The most alarming part of the chapter addresses the idea that test prep is corrupting the idea of good teaching. Koretz has a disturbing bank of anecdotes about teachers who are told their is not to teach their subject, but to raise test scores in it. And like anyone else who has encountered new teachers in the last decade, he's met young teachers who were taught in college that test scores are their main purpose.

They were telling me that I was missing the boat by seeing test prep as something that competes for time with good instruction. In their experience, raising scores had become the end goal, the mark of a "good" teacher. To an alarming degree, they had been taught that test prep and good instruction are the same thing.

There are so many reasons this is bad, bad news, but here's one that Koretz spots

For the present, it indicates that one of the few checks against inappropriate test prep-- teachers' own understanding of the differences between test prep and good instruction-- has been eroded.

8: Making up Unrealistic Targets

Yikes. This is a depressing chapter. Koretz notes that the "trust most people have in performance standards is essential, because the entire system now revolves around them." And yet that trust is wildly misplaced.

Koretz outlines several methods of setting performance standards, but the important thing to remember is this:

There is another, perhaps even more important, reason why performance standards can't be trusted: there are many different methods one can use, and there is rarely a really persuasive reason to select one over the other.

And yet those different methods can produce vastly different results.

There are other problems.

A primary motivation for setting a Proficient standard is to prod schools to improve, but information about quickly teachers actually can improve student learning doesn't play much, if any, of a role in setting performance standards. When panels set standards, they are not given information about practical rates of improvement, and for the most part they are not asked to consider them.

In other words, they are set up to be the educational equivalent of an agricultural board that declares, "What we need is wheat that we can harvest six hours after it's planted. Do that!"

Also, standards are set mainly on the assumption that all kids are the same, with a goal of radically reducing the variability of achievement. I'm just going to sum up on this part: that's a dumb assumption.

9: Evaluating Teachers

Koretz is pretty brutal about the VAM-sauced test-based teacher evaluation systems. He allows that the old system had a problem in that everyone ended up looking great, but the new system surprised him because it solved problems with solutions so dumb it had never occurred to him that anyone would actually use them (e.g. evaluating teachers based on the scores of students they have never met).

This chapter includes Koretz's account of his visit to the Department of Education in Duncan's early days. You'll want to read that yourself.

How have we screwed up test-based accountability for teachers. Let Koretz count the ways:

* Taking test scores out of context (a "deliberate goal" of reformers, but "one of the main reasons the reforms have failed)
* Trying to use tests to explain, not just describe
* Using "Value-Added Modeling to evaluate teachers (here's as good an explanation as you'll find for why VAM can't possibly work)
* Rating teachers with the wrong test
* Teachers ratings are inconsistent across tests
* Teacher test scores are unstable over time (there are charts and specifics here that drive home how bad this effect is).

10: Will the Common Core fix this?

In a word, no.

Want a longer explanation?

It's not just the Common Core that has been dropped into schools wholesale before we gathered any evidence about impact; this has been true of almost the entire edifice of test-based reform, time and time again. I'll argue later that putting a stop to this disdain for evidence-- this arrogant assumption that we know so much that we don't have to bother evaluating our ideas before imposing them on teachers and students-- is one of the most important changes we have to make.

Koretz also argues that CCSS tests have actually been created in a way that makes them more predictable, and therefor more susceptible to all the Campbell's Lawian Test Preppery he just spent several chapters eviscerating. Koretz dismisses the one size fits allness of the Core, noting that one official once told him that the Core eliminated the distinction between career readiness and college readiness. "Rhetorically, perhaps, but not in actuality" responds Koretz.

It is a new flavor of the same old failed approach, gives one size fits all a "grandiose rhetorical wrapper." Underneath all the noise

the basic failed model of educational improvement remains unchanged: set arbitrary performance targets on standardized tests; apply them uniformly, without regard to circumstances; and reward and punish. Whatever its other virtues and vices, the Common Core hasn't changed this. This approach hasn't worked before, and it won't work with the Common Core.

11: Did Kids Learn More?

We know less about this most fundamental of questions than we probably should.

Why not? Partly because the test data is so subject to inflation that it can't be trusted. But also-- and I know I just ran a similar quote, but this point is important

There is a second reason for the dearth of information, the blame for which lies squarely on the shoulders of many of the reformers. Time after time they declared that they had figured out what would work, and they imposed it on students and teachers on a mass scale without taking the time to evaluate their programs first. It's analogous to a drug company saying that they have figured out, based just on their own beliefs and logic, which drugs will be effective and safe, so they can skip the time-consuming and expensive burden of actually gathering some evidence before selling it to you.

Koretz spends some time crunching some numbers, asking if students learned more, if they learned it because of test-centered accountability, and if what they learned justifies the "huge" costs of these policies (including costs like the corruption of instruction on a broad scale).

The answer is pretty clearly, "No."

12: Nine Principles for Doing Better

Here they are--

Pay attention to other important stuff.

Monitor more than student achievement.

Set reasonable targets.

Stop just kicking the dog harder.

Don't expect school to do it all.

Pay attention to context.

Accept the need for human judgment.

Create counterbalancing incentives.

Monitor, evaluate and revise.

13: Doing Better

Here Koretz works out his nine principles into a more specific action plan. He considers at length some of the usual models-in-other-countries. And he lays out some ideas.

Measure what matters. Okay, no bonus points for originality here, but then one of the subtexts of this book is that apparently you have to point out the obvious to people in education  reform because some of them rush straight into doing things that are obviously stupid. Here also he and I disagree on a role for well-used well-made standardized tests. That's okay; I know I'm out in left field on this. But I do like this:

This is the first and one of the most difficult tradeoffs we face: to measure learning well and to give teachers better incentives, we will have to use measures that have serious drawbacks-- in particular, potential inconsistency from classroom to classroom and school to school.

Yes. The more perfectly something is universally standardized, the useful it is to an individual teacher.

We need to measure "soft skills" well. This will be hard, and there will be disagreement. Yup-- that's already happening. And he has a useful insight-- part of the reason we got standardized testing "was the notion that educators can't be trusted to evaluate schooling or other educators." But the soft skills measures will have to "give a substantial role to the judgment of professionals." Not standardized SEL tests.

We need a sensible accountability system. Most interesting detail here-- we need various measures that are NOT too closely aligned with each other. If everything's aligned together, everything canm be wrong together.

Use tests sensibly. One test cannot do everything.

Provide support to teachers. Monitor and make midcourse corrections (you know-- the way Common Core was specifically designed not to do).

Still here? Good for you.

Chapter 14 is a wrap-up and I'll skip that, just as I've skipped over many specifics and explanations. Really, if I had time to be a better reviewer, I would have given you a much more compact look at what Koretz has created here, which is nothing less than a scholarly, thoughtful, accessible explanation of how test-based reform has taken education into the weeds. You should read this book, and you should pass it on to other folks who care about education and want to both understand the problems and envision some solutions. This is a valuable work, and I'll be coming back to it again and again, and you should, too.