MD: State Super Gets Writing Lesson

Les Perelman is one of my heroes. For years he has poked holes in the junk science that is computer-graded writing, bringing some sanity and clarity to a field clogged with silly puffery.

We are all indebted to Fred Klonsky for publishing an exchange between Perelman (retired Director of Writing Across the Curriculum at MIT) and Jack Smith, the Maryland State Superintendent of Schools. Maryland is one of the places where the PARCC test now uses computer grading for a portion of the test results. This is a bad idea, although Smith has no idea why. I'm going to touch on some highlights here in hopes of enticing you to head on over and read the whole thing.

The exchange begins with a letter from Smith responding to Perelman's concerns. It seems entirely possible that Smith created the letter by cutting and pasting from PARCC PR materials.

In response to a question about how many tests will be computer scored, Smith notes that "PARCC was built to be a digital assessment, integrating the latest technology in order to drive better, smarter feedback for teachers, parents and students. Automated scoring drives effective and efficient scoring" which means faster and cheaper. Also, more consistent. No word on whether the feedback will actually be good or useful (spoiler alert: no), but at least it will be fast and cheap.

In responding to other points, Smith repeats the marketing claim that computer scoring has proven to be as accurate as human scoring, that there's a whole report of "proof of concept," and that report includes three whole pages of end notes, so there's your research basis.

Perelman's restraint in responding to all this baloney is, as always, admirable. Here are the points of his response to Smith, filtered through my own personal lack of restraint.

1) Maryland's long use of computers to grade short answers is not germane. Short answer scoring just means scanning for key words, while scoring an entire essay requires reading for qualities that computers are as yet incapable of identifying. Read about Perelman's great work with his gibberish-generating program BABEL.

2) Studies have shown that computers grade as well as humans-- as long as you are comparing the computer scoring to then work of humans who have been trained to grade essays like a machine. Perelman observes that real research would compare the computer's work to the work of expert readers, not the $12/hour temps that Pearson and PARCC use.

3) The research is bunk. The three pages of references are not outside references, but mostly the product of the same vendor that is trying to sell the computer grading system.

4) Perelman argues that no major test is using computers to grade writing. I'm not really sure how much longer that argument is going to hold up.

5) The software can be gamed. BABEL (a software program that creates gibberish essays that receive high scores from scoring software) is a fine example, but students need not be that ambitious. Write many words, long sentences, and use as many big words as you can think of. I can report that this actually works with the $12/hour scorers who are trained to score like machines as well. For years, my department achieved mid-nineties proficiency on the state writing test, and we did it by teaching students to fill up the page, write neatly, and use big words (we liked "plethora" a lot). We also taught them that this was lousy writing, but it would make the state happy. Computer scoring works just as well, and can be just as easily gamed. If one of your school's goals is to teach students about going through ridiculous motions in order to satisfy clueless bureaucrats, then I guess this is worthwhile. If you want to teach them to write well, it's all a huge waste of time.

6) There's evidence that the software has built-in cultural bias, which is unsurprising because all software reflects the biases of its writers.

It remains to be seen if any of these arguments penetrate State Superintendent Smith's consciousness. I suppose the good news is that it's relatively easy to teach students to game the system. The bad news, of course, is that system is built on a foundation of baloney and junk science.

It angers me because teaching writing is a personal passion, and this sort of junk undermines it tremendously. It pretends that good writing can be reduced to a simple algorithm, and it does it for the basest of reasons. After all, we already know how to properly assess writing-- you hire a bunch of professionals. Pennsylvania used to do that, but then they sub-contracted the whole business out to a company that wanted a cheaper process.

And that's the thing. The use of computer assessment for writing is not about better writing or better feedback-- the software is incapable of providing anything but the most superficial feedback. The use of computer assessment for writing is about getting the job done cheaply and without the problems that come with hiring meat widgets. This is education reform at its absolute worst-- let's do a lousy job so that we can save a buck.

OK: An Example for All of Us

Oklahoma has taken its share of lumps in the ed debates. Their legislature is not quite as determined to burn public education to the ground as are the legislatures of North Carolina or Florida. It's not quite as committed to cashing in on the charter revolution as Ohio. But Oklahoma remains in the grip of reformster baloney, and teachers are tired and frustrated. The word 'frustrated" comes up rather a lot. And teachers are doing something with that frustration.

Word has been spreading since April-- teachers are running for elected office.

Meet, for instance, Kevin McDonald, an English teacher at Edmond Memorial High School.

“Its becoming apparent to more and more educators that to be heard we need to be in the conversation, not outside of the conversation trying to talk at people,”says McDonald.

“Teaching is what I want to do,” he said, “But I’ve come to a point where my ability to teach is being compromised by legislative decisions.”

So he's running for Senator Clark Jolley's seat. Jolley is a third-term congressman who won his last election with 79% of the vote. He serves as a member of the education committee and chairman of appropriations. It is entirely possible that he is not ripe to be unseated. But at a bare minimum, challenging the GOP senator has given McDonald a chance to put school finances in the election discussion.

And finances are a touchy subject in Oklahoma. Well, the Common Core was one source of many bunched-up panties, and they pushed back hard on VAM, but they have had a bad several years when it comes to teacher pay and oddly enough, teacher recruitment and retention. John Croisant, a sixth grade in Tulsa, cites it as a reason for running for an open House seat.

“For me, it’s personal,” said 39-year-old Republican John Croisant, a sixth-grade geography teacher in Tulsa Public Schools who said he’s seen several of his colleagues leave Oklahoma to take teaching jobs in neighboring states for more money. “It’s not that we don’t want to teach. They’re going across the border and they’re able to make $10,000 more each year for their families.”

The teachers have held public sessions to promote their candidacies and their issues, and they have been assertive about spreading the word about candidates. For instance, if you want to run down some information about the education-positive candidates, you can search through the pages of noted OK blog Blue Cereal Education for a host of candidate profiles (marked with the hashtag #OKElections16).

It's a lesson for all of us. As much as teachers tend to shy away from politics and dream of just closing the door and ignoring the world outside, it's politics that set the rules that increasingly intrude on our classrooms. Simply making a contribution to the political action committee of the union is not enough (or, in the case of some unions and some races, not even helpful). We have to speak up. We have to promote the folks that stand for what we value. We have to do our homework and make hard choices (perfect candidates, it turns out, show up as often as perfect humans).

Oklahoma's primaries are next week. My best wishes to the education candidates-- I hope they do well. But even if they don't do well, they have already done good by making the education discussion part of the political discussion. That in itself is an achievement; as we've seen in the last year, no matter how important we think education issues are, getting politicians to talk about them is like trying to get my labrador retriever to talk about existential angst and third world monetary policy.

We have complained for decades that education discussions are being held without any teachers in the room, and we are right to complain. But it is not enough to keep waiting for our invitation to arrive-- we need to get out there and shoulder our way into the arena. Thank you to the teachers of Oklahoma who have worked to do that.

School Accountability Camps

Now that ESSA has opened the door (maybe, kind of) to new approaches to school accountability. What are the possibilities?

Mike Petrilli of the Fordham Institute has outlinbed four possibilities in a list calculated to help us all conclude that only one of the possibilities is really legit. The list apparently grows out of the Fordham contest to design a compatibility system; I entered that competition but was not a finalist, though I swear, I'm not bitter. Someday some rich benefactor will give me a stack of money, and I'll start my own thinky tank, and then I'll have ed system design competitions all the time. so there.

Anyway. let's look at the four camps that Petrilli sees. He labels them by their supposed slogans.

Every School is A-OK!

Petrilli says that the proponents of this view are the teachers unions and "other educator groups," and it does not bode well for his list that he opens with a Gigantic Person of Straw. I read a lot of writing by a lot of people, and I'll be damned if I can think of a single person who says that every school is a-okay. So we can skip past this camp because there is not a single tent pitched there.

Attack the Algorithms

Petrilli says this camp is anti-test, but pro-accountability. "It seeks a system that uses as much human judgment as possible and captures a full, vivid, multifaceted picture of school quality." These tend to base their system on school inspection. It's more common in Europe than the US, which could be in part because the last decade or two of US law has specifically centered on bone-headed high-stakes test-based formulae that do their best to root out any trace of human judgment (except, of course, the human judgment that produced the tests, the scoring of the tests, and the algorithms for crunching the test results). I won't lie. This is my camp.

Living in the Scholars’ Paradise

Yeah, Petrilli doesn't have his poker face on, either.

This approach uses sophisticated, rigorous models to evaluate schools’ impact on student achievement, making sure not to conflate factors (like student demographics or prior achievement) that are outside of schools’ control. 

He's wrong, and he's wrong because he is focused on the single output of student test scores. He's willing (at last) to admit that there are factors that influence the test scores that are outside the school's control, but he's still just counting test scores, and that's an unacceptably low bar for what measuring what schools do. Actually, it's not a bar at all. It's like judging pole vaulters by checking their foot placement at the point of lift-off and ignoring everything else. And, to flog the simile further, under our current testing regimen, it's like judging that foot placement by comparing it to an idea about foot placement cooked up by someone whose only expertise is that several years ago they watched a couple of pole vaulters on television.

Petrilli also wants to include, in fact give most attention to, value-added measures, and at this stage of the game, backing VAM as a good measure of school quality makes even less sense than arguing that Donald Trump should be President because of his great statesmanlike qualities. VAM is simply indefensible as a measure of school or teacher quality. Petrilli has displayed flexibility in the past; why he remains devoted to this slice of junk science is a mystery to me.

NCLB Extended, Not Ended

NCLB is gone but not forgotten. Or maybe it’s not exactly gone, in the mind of folks who yearn for Uncle Sam to mandate accountability models that obsess about achievement gaps and give failing grades to any school with low proficiency rates for any subgroups.

As always, I tip my hat to Petrilli's ability to craft a sentence to stay polite and yet still make clear that he thinks someone is full of shit. Game recognizes game.

Anyway, his portrayal of this camp is interesting in the context of the ongoing reformster battle over whose reform it is, exactly, anyway. Here he says that those lefty civil rights types just want to measure raw achievement scores so as to highlight schools that are failing non-wealthy students of color. As with category #1, I'm not sure these are people who actually exist.

Tents He Forgot To Pitch

One of the critical questions that is repeatedly overlooked in accountability discussions is:

Accountable to whom?

In other contexts, reformsters will talk about being accountable to parents, that in their perfect world parents get all sorts of information that allows them to select the most awesomest school available. Yet none of Petrilli's camps, least of all his preferred one, spends any time at all asking parents what they want the schools to be held accountable for. And everything we know about parents (and personally, I'd like to consult all non-parental taxpayers as well, because, public schools) tells us that "high scores on standardized tests" does not top anybody's list. And look-- reformsters already know that, which is why they insist on talking about "student achievement" when they mean "test scores." They know that if we just told folks about "test scores" they would be unimpressed and unmoved, so we call it "student achievement," which brings to mind a whole wide spectrum of accomplishments and skills-- even though test scores is all that's really on the plate.

And of course, since we're talking about ESSA, what we're really talking about is accountability to the feds and (depending on how far rogue the USED is willing to go) the states. So while talk about accountability to parents sounds nice, that's not really what we're trying to do anyway. We can say that the bureaucrats at the capital are fine proxies and watchdogs for parent and taxpayer interests, but let's get real-- the suits at the capital have a whole list of political concerns (which are in turn tangled up with all sorts of lobbying and money concerns) that have nothing to do with what Mom and Dad are concerned about when they send Chris and Pat off to school.

Bottom line-- the school inspection, human-based accountability gets you the kind of information that parents and taxpayers (who are, by and large, human beings) want and find useful. The scholar's paradise is really a bureaucrat and policy wonk's paradise that gives primacy to the interests, concerns and policy goals of the wonks and bureaucrats over the interests and concerns of parents and taxpayers.

Ask a parent (or taxpayer)-- Do you want to know everything you'd like to know about your local school, or would you like to be confident that the bureaucrats at the department of education know everything they want to know? I have a thought about what the answer would be.

And underlining all of this? Congress and the USED still have to finishing arguing about what the department is supposed to (or allowed to) hold accountable for. Can't wait to see where that leaves us. 

Do Interim Tests Help?

You know the drill. We have to take the Big Standardized Tests in the spring, so in the fall and winter, maybe multiple times, we're going to take the pre-test, or practice test, or interim test, or testing test test.

The plan is that this will get the students ready for the BS Test (because it is such an artificial, inauthentic task that it doesn't resemble any other activity except taking similar inauthentic pre-test practice interim testy test tests). Even more importantly, in some schools, it will let us target the students based on how likely they are to make us look bad come test time.

Procedures vary by school, but a popular approach is to sort students into three categories: Don't Have To Worry About Them, Hopeless, and Maybe If We Really Hammer These Kids We Can Get Them To Squeak Through. The first two groups get little or no attention, and the third group gets "extra attention" which may take the form of anything form extra drill in math and language class all the way up to being pulled out of non-test classes so that their whole day can be devoted to test prep.

There is a cottage industry in pre-test practice interim testing tests. My district used to use the 4sight tests, until we noticed that their ability to predict BS Test results was only slightly better than reading the bumps on a dancing toad under a full moon. These days we're dabbling in NWEA voodoo, so we'll see.

Well, maybe we'll see. After a decade of interim testing, plenty of teachers have an opinion about how well it actually works. Now there's a piece of research that looks at the usefulness of interim testing. 

The verdict? At best, it doesn't make BS Test scores drop. At best.

As always, the research comes with lots of caveats. The study covers about 30,000 students at 70 different schools in Indiana. Two interim testing programs were studied. The study used data from 2010-2011. The write-up includes sentences like "Two-level models were used to capture the nesting in the data."

Nevertheless, the answer to the question "Do pre-test practice interime testing testy test testicles help" is, according to this paper, "No, not at all." Specifically, in the case of the 3-8 grade students, the students with interim testing did no better than the students without. In the case of K-2 students, the students who were interimly tested actually did worse than those who were left alone to play and learn and all the other things that students can do in school when they're not wasting time on test-based test prep for their big test (which as K-2 students, they shouldn't be taking in the first place, but of course we have to get them taking those tests early on so that when the BS Tests start really counting in third grade, the entire process has been normalized-- in other words, the K-2 tests are basically pre-test practice interim tests for the 3-8 tests).

I'm not going to hold this small piece of research up as a final word on the subject, but it is a data point, and the data point says what many of us in the classroom already believed-- the pre-test practice interim testing test is a waste of time and money.

Intangible Greatness

You may have heard by now the satisfying news that the Supreme Court spanked Abbie Fisher in a decision that provides, as Salon put it, "A massive blow to mediocre white people coasting on their racial privilege." Fisher (and her lawyer, notorious affirmative action combatant Edward Blum) argued that as a mediocre white person, she should automatically get preference over a mediocre black person (I'm paraphrasing). Oh, and that the 14th Amendment should get lost.

The Supremes said not so much, and conservative Anthony Kennedy was the guy who helped them do it. You'll be reading about that decision all over the place-- but I want to focus on one excerpt from Kennedy's decision:

A university is in large part defined by those intangible qualities which are incapable of objective measurement but which make for greatness. Considerable deference is owed to a university in defining those intangible characteristics, like student body diversity, that are central to its identity and educational mission.


Emphasis mine. Damn straight, Justice Kennedy. And not just a university, but the individual human beings who make it up. In fact, all individual human beings. "I owe my success and my achievements in life, my whole rewarding existence, my greatness, to a set of qualities that are easily measured and quantified objectively," said nobody, ever. I am sure there will be many days to come on which I disagree with Justice Kennedy, but today is not that day.

Can we just plaster this up some place that it will be visible on the day that the court hears a case involving the use of narrow Big Standardized Tests of math and English to judge teachers, schools and students? Can we make a big meme and plaster it all around the intertoobz? Can we remind reformsters every chance we get about the intangible qualities that make for greatness, but evade objective measurement? And can I have a poster of this in my classroom, to remind myself and my students that greatness, achievement, happiness, and a rich and rewarding life are built out of qualities that defy measurement even as they are the foundation of every worthy and wonderful life that a human being has ever lived?

Attacking the Public in Public Education

Many parts of the attack on US public education have not been subtle or hard to detect. The refrain "our schools are failing" has been so steadily repeated for the past few decades that it is now accepted uncritically, independent of any evidence other than "Hey, I keep hearing people say it, so I guess it must be true." Now we hear it just tossed off as an aside, an assumption-- well, of course, public schools aren't any good.

In addition to attacking the reputation and quality of public schools, we've also heard an unending explicit and implicit attacks on the reputation of our nation's teachers. They're dummies with low SAT scores. They have the worst preparation of any college students. We'd be better off giving an ivy league grad five weeks of training and plunking them in a classroom.

All of these are an attack on the "education" part of "public education," a steady drip, drip, drip that tells us that the system that is supposed to educate is not doing a very good job of educating.

But there has been another steady attack, more subtle but increasingly successful, on the "public" part pf "public education."

The reformster refrain that the money should follow the student is one such attack-- it cuts the public out of the system, removing the voice of any taxpayer who doesn't have a child in school. The whole argument that choice-voucher systems should put all decision-making in the hands of parents makes a foundational assumption that education is not a public good, maintained by the public in the public space in order to deliver benefits to the public. Instead, it re-imagines education as a consumer good, created by a vendor and then handed off to the student while money changes hands. Where education might once have been viewed like air or water or other shared public resources, we're now encouraged to see it like a pizza or a toaster.

We can now start to see some of the side-effects of this view. When a public school is closed these days, it's not necessarily seen as a blow to the community, like the loss of a park or the pollution of a water supply. Instead, it's treated like a store closing, as if we just lost the Taco Bell on the corner, or the local K-Mart was closed up. It's a business decision made by someone who doesn't answer to the community, really pretty much out  of our hands, right?

More troubling, it gets us to a place where the community no longer feels the obligation, the assumption, that it has a responsibility to provide schools.

And so we can see a major school system like that of Erie, Pennsylvania, considering that maybe they should just stop offering high schools at all. Or the governor of New Jersey can suggest that we just spend the identical bare minimum on all students, and if that means that poor kids don't get any real schooling at all. Granted, that move was so outrageous that even longtime Christie apologist Tom Moran condemned it, but still-- to even suggest that we don't really need to bother making sure that we as a community, a state, a nation are getting the job done.

Which is another part of the assault. We don't need to make sure we're providing a public education because those charter schools will do it (maybe, for a while, for some students).

As with all things in public education, this has not happened in a vacuum. As a nation, we have moved steadily away from community responsibility for the public space. We don't want to live next to Those People, and we don't want to pay for their health care or their food or their housing. Politically, we are increasingly united by a belief that some people shouldn't count and torn apart by disagreements about who exactly the not-really-American don't-really-count people are. We want to privatize profit and move risk into the public sector.

Simply put, we don't want to take responsibility for any people except Our Own. You get what you can get, and if you can't get much, well, you should have thought of that before you decided to be poor (or not white).

So it should be no surprise that the most damaging attack on public education has been on the very idea of an education provided by the public for every member of the public, the better to maintain and improve the public space we all share and care for the public resource for which we share responsibility. We have instead been encouraged to think of education as a new couch or a bag of cheese doodles, a commodity that you get access to (or not) depending on the deal that you work out on your own.

Charterista Faux Teacher Programs Make ESSA Demands of Feds

The damndest things turn up on Twitter some days. Take this document. 

Entitled "Joint Statement Calling for Transparency of Outcomes to Improve Teacher Preparation and Better Serve Students and Districts," this is a fairly transparent demand by several "alternative path" teaching programs that new regulations give them a better advantage in the Brave New ESSA World.

Urban Teachers—along with Aspire Public Schools, Blue Engine, Boston Teacher Residency, Match Teacher Residency, National Center for Teacher Residencies, Relay Graduate School of Education, Teach For America and TNTP—join together to request the Department of Education and Congress create clear guidance for state education agencies as they attempt to act on this opportunity and improve the quality of teacher preparation at the state-level.

And by "improve" of course they mean "make more profitable for folks like us." Specifically, these folks would like the feds to "provide states with specific guidance around developing systems where all teacher preparation programs are accountable for collecting and publicly sharing outcomes data on the success of their programs, participants, and graduates."

Yes, it's one of the Top Ten Dumbest Reform Ideas Ever, back for another round of zombie policy debate. The same VAM-soaked high stakes test scores that has been debunked by everyone from principals to statisticians to teachers, the same sort of system that was called arbitrary and capricious by a New York judge, the same sort of system just thrown out by Houston-- let's use that not just to judge teachers, but to judge the colleges from which those teachers graduated.

Why would we do something so glaringly dumb? The signatores of the letter say that consumers need information.

Without the presence of concrete outcome measures, local education agencies and potential teacher candidates are hard-pressed to compare the quality of teacher preparation programs. Thus, it is a gamble for aspiring educators to select a teacher training program and a gamble for principals when hiring teachers for their schools

Yes, because everyone in the universe is dumb as a rock-- except reformsters. Just as parents and teachers will have no idea how students are doing until they see Big Standardized Test results, nobody has any idea which teaching programs are any good. Except that, of course, virtually every program for teaching (or anything else, for that matter) has a well-developed and well-known reputation among professionals in the field.

As for the "gamble" of hiring-- are these folks really suggesting that administrators can sit and interview teacher applicants, watch them teach sample lessons, and look at their work, decide that thecandidate looks really good and turn out to be wrong because they didn't have hard data about the college program. "My recommendation for the history job? Well, I don't know. She seems smart and capable, her sample lesson was engaging and exciting, and she has a kick-ass portfolio of great ideas. She seems knowledgeable and the exact sort of personality that would fit with our staff here. But I don't have any hard data about her college ed program, so I guess I'm just shooting blindly in the dark. You wanna just flip a coin?"

What sample metrics should we use for this collection of hard data that will be so much more useful than human judgment?

Sample metrics include teacher retention and attrition, principal satisfaction surveys, teacher evaluations, teacher performance on state exams and student achievement gains. 

Because whether a teacher keeps a job or stays in teaching or not is totally the fault of the college program. I don't know why we're collecting principal satisfaction data, since principals are apparently as dumb as rocks. But boy, let's just grab all that BS Test data-- in fact, let's double it up by counting it both in teacher evaluations and by itself.

Oh, but I do have to correct myself. Teacher retention can be controlled by the teacher program if the program is, like the Match system, a program that prepares faux teachers made to order for the charter system that runs the teacher prep program, or like TFA, "prepares teachers" for jobs that it has been contracted to fill. So that's a win for these guys. Remember-- select the right data, and you can always make yourself look better.

The signatores also tout the advantage of data-driven improvements to programs, because that's good for the "customer" (a word that turns up plenty in this three page letter). And of course it will also let states decide which programs to "support." Ka-ching.

Finally, this is just the first of a series of letters to the feds telling them what the people in charge of the nation's shadow network of privatized faux teacher trainers. So there's that to look forward to.

Look, it's not just that this is a terrible terrible terrible TERRIBLE system for evaluating teacher programs, or that it's a bald-faced attempt to grab money and power for this collection of education-flavored private businesses. These days, I suppose it's just good business practice to lobby the feds to write the rules that help you keep raking it in. It's that this proposal (and the other proposals like it which, sadly, often com from the USED) is about defining down what teaching even is.

It is one more back door attempt to redefine teaching as a job with just one purpose-- get kids to score high on a narrow set of Big Standardized Tests. Ask a hundred people what they mean by "good teacher." Write down the enormous list of traits you get from "knowledgeable" to "empathetic" to "uplifts children" to "creative" and on and on and on and, now that you've got that whole list, cross out every single item on it except "has students who get good test scores."

It's the fast foodifying of education. If I redefine "beautifully cooked meal" as "two pre-made patties cooked according to instructions, dressed with prescribed condiments, and slapped on the pre-made buns" then suddenly anyone can be a "great chef" (well, almost anyone-- actual great chefs may have trouble adjusting). These are organizations that specialize in cranking out what non-teachers think teachers should be, and their thinking is neither deep nor complicated, because one of the things a teachers should be is easy to train and easy to replace.

When excellence is hard to define and difficult to achieve, just redefine excellence so that it is low bar shallow narrow mediocrity-- voila! Excellence is now easily within your grasp. Not only that, but the people who keep pursuing actual old school excellence will either knuckle under or be cast aside (silly status quo embracers). Ka-ching!

But first, be sure to write a letter to the appropriate government agency so that they can help you with your redefinition.