A Field Guide To Testing Advocates

Now that we have entered testing season, it's time to track the annual explosion of testing advocates as they spread far and wide in their attempt to stem the opt out tide and support the test manufacturing industry. While testocrats and test advocates may all seem indistinguishable, they actually come in many shapes and forms. Here's your guide to identifying the particular brand of test pushers in your neighborhood.

The Tests See All. The new standardized tests can measure deep insight, critical thinking, broad understanding, blood sugar, and future happiness. It may seem that the test reports just a give a number that's just a digital letter grade, with prescriptions like "The student should read better," but if you stare at these reports very hard, you can see every drop and detail of the students' mind.

The Tests Will Reveal What Was Previously Unknowable. Teachers, parents and students would never have the slightest idea how the students were doing in school if not for the Big Standardized Test. Parents and teachers are dopes who think that spending time with the students actually gives them some special information, when of course only a test manufactured and scored by people who have never laid eyes on the students could possibly reveal what those students know or can do.

The Tests Bring Social Justice. By giving non-wealthy non-white students the same test as wealthy white students, we will erase the achievement gap, because testing teaches students to do better. Once the achievement gap is erased, poor students will attend college and from there they will all become VP's of major corporations, minimum wages will rise, systemic racism will disappear, and social justice will rule the land. None of this can happen if you don't take the test.

The Tests Force Politicians To Solve Problems. Once we have actual data showing that this poor school in a poor neighborhood gets poor test results, politicians will have no choice but to get the school the support it needs. Or they could take support away from it and give that support to some charter school that works with a few of the original students. Or they could close it. So by taking the test, your community's school definitely gets a chance to get the help it needs to face its problems, unless it just gets kicked in the teeth instead.

The Tests Will Shape Up These Lazy Kids. Kids these days are slackers and weaklings and have been protected by their mommies. Not taking the test is just more coddling; they need to suck it up and learn about the part of life that involves pointless bureaucratic tasks. These kids whine too much about everything. Not only should they go ahead and take the damn test, but we should probably punch each one in the mouth while the test is going on.

The Tests Will Root Out Those Awful Teachers. Look, the only reason there is an opt out movement is because teachers hate education and students and don't want to get caught sucking, so they have suckered hundreds of thousands of parents who are now just acting as the union's puppets. Anyone who opts out is just protecting a crappy teacher. So your kid suffers a little and wastes some of her time. It's worth it to find proof that teachers are awful and need to be fired and the union broken and all of them just put in their place.


The Tests Are Preparation for the Real World. Hey, life is all about tests. Remember that standardized test you had to take before you could start dating your current spouse? Or how at your workplace you stop working every week or two to take a standardized test that shows how well you're doing your job? Remember how your mom used to make you pass a standardized test before she would feed you, read you a bed time story, or give you a hug? Life is all about the standardized testing.

The Tests Are Just Like Doctors' Tests. When you go to the doctor's office, she gives you a test. She uses her knowledge of your history to choose from among the thousand of tests developed and tested by other trained physicians to develop a picture of exactly what issues you are facing and how to treat them. This is exactly like that, except instead of thousands of tests, there''s just one test used for everything from heart disease to knee injuries to cancer, and instead of that test having been extensively tested, it's just experimental, and instead of coming up with a treatment plan, the authorities close your doctor's office and replace it with a doctor's office filled with healthier patients. So, basically the same exact thing.

The Test Is Required By Law and Everyone Must Take It or Else and I'm Not Kidding Young Lady-- You Get Started Right Now Or I Will Turn This Car Around, and Don't Think I Won't. The law says everyone must take the test or else the school will lose all funding and the school will be shut down and locusts will rain down from the heavens, cats and dogs living together, mass hysteria. How many schools have been punished for low test participation, you ask? Never you mind that. This is serious, I tell you. Serious.

The Test Is Really Important, Isn't It? I mean, it's a standardized test, so it must be legit. Surely they wouldn't put it out there unless it was legit. And if my child doesn't take it, won't that go on their permanent record?


You will find many of these species more often out in the wild during this season. Keep your eyes open and your field guide ready, though you will generally be able to find them if you just follow the sound of their outrage. 

SAT & ACT Go for the Gold

Well, the New York Times wrote about it this week, so it's now officially a thing-- the SAT and ACT are mounting what Scott Marion, the executive director of the Center for Assessment, calls a "land grab."

The ACT and SAT, America's long-standing "you have to take that if you want to get into college" tests have been sliding steadily over the past few decades. Fairtest lists over 850 colleges and universities that no longer require the tests, and research has repeatedly shown that high school grades-- not the venerable tests-- are the best predictors of college success. SAT and ACT scores correlate best to family income, not academic skill or achievement.

The College Board, under new jefe and Common Core architect David Coleman, has aggressively remarketed itself as more student-friendly, more real-worldy, and packed with more free (in exchange for all your marketing and tracking data) test prep. Well, not test prep, exactly, because the New! Improved! SAT is also supposed to be impervious to the kind of coaching that is historically part of SAT success (and if you believe that, keep your eyes peeled for Coleman's announcement that the College Board will soon be selling the Brooklyn Bridge). 

But that is peanuts, because a much larger opportunity has presented itself. Thanks to ESSA (the new body of federal education law) every state must give a Big Standardized Test. And over the last few years, states have become disenchanted with the cost and quality of the PARCC and SBA tests (developed, it should be noted, with lots of tax dollars). So out with the old tests and in with the...?

"Hey," say the SAT and ACT folks. "You need a Big Standardized Test! We've got a Big Standardized Test. I'll bet we could help each other." And now, as chronicled by the NYT, the Big Two of the BS Test business are working their way into the state-level market. Hey, kids-- excited and relieved that the University of Delaware no longer requires you to take the SAT to get in? Bad news-- the state of Delaware is going to require you to take the SAT anyway.

It is impossible to overstate just how lucrative this is for the test manufacturing companies. Just imagine-- every high school student in the country forced to take the SAT and every taxpayer in the country forced to pay for it.

But it is also impossible to overstate just how large a multilayered dumb cake this is, complete with a big stupid cherry on top of big dope icing.

First-- understand what is happening. States have to give a BS Test because the feds want to know that every student is college and career ready. Colleges and universities are announcing one by one that the SAT/ACT cannot predict if a student is ready for college. So states will now adopt a test widely known to not measure college readiness in order to test all students for college readiness.

The SAT/ACT are designed to show if a student is ready for college level work. There is no reason in the world to believe that those exact same tests can determine if a low-functioning student with learning disabilities is reading, writing and mathing at grade level.

The SAT and ACT are norm-referenced tests. That means they do not measure students against some objective, pre-determined standard, but simply rank students from the highest to the lowest. That means that a certain percentage of students will always fail. Always, as in we know before the test is even given. (If that made a horrified light bulb go on for you, I have bad news-- the current crop of BS Tests do exactly the same thing).

And here's the second part of that stupid-- the SAT and ACT have always been given to students who considered themselves college bound. The SAT/ACT folks have never had to crunch numbers for the entire student population of a state. They have never had to build their grading scale with all of the non-college-bound student results folded in. Will that flooding of the bottom end of the bell curve create what amounts to score inflation on the top end? Does anybody at either company have any idea what effect the new score base will include?

Will states buy into the whole package? Will SAT states also buy up the PSAT and the new PPSAT, the 8th grade PSAT-- and if so, will they also buy into those pre-tests' prodigious data-collecting features. The PSAT starts with students volunteering a ton of data, and results have now been tweaked to include recommendations for AP courses (another product of the College Board). In short, will the state now be selling both the students' personal data as well as access to the students for marketing other products?

This is the kind of decision made by bureaucratic amateurs, people who still harbor the childlike belief that any standardized test is, you know, a standardized test, and the results can be used for any purpose because it's a standardized test, and standardized tests are like magical unicorns that can look into a student's soul to see whatever is hiding there. If they were mechanics or carpenters, they would be the ones who said, "Hand me a tool," on the theory that whether the tool is a hammer or a screwdriver or a drill, it's still, you know, a tool, and you can Do Stuff with it.

In the hands of these folks, a standardized test is just a blunt tool to club students over the heads. And there are salesmen just lined up to tout the magical properties of these testing unicorns because having to market to fifty state capitals for the entire possible market is a hell of a lot easier than having to market to every single set of parents, one by one-- particularly when the colleges that have always backed your play have finally decided to stop helping you run the con.

This sort of foolishness is going to keep happening, because there is no smart way to implement a stupid idea-- and the idea that we can give a single standardized tests to every student in the country and by its results know how the state education system is doing, how the school is doing, how the teacher is doing, and whether or not the student is ready for college and career-- that is a supremely stupid idea, and so every attempt to implement that idea is going to be stupid, too.

This particular stupid idea is just hugely profitable for the SAT and ACT folks. You would hope that somebody in power could hear the thudding sound of this big fat dope of an idea slamming into high school students across their state, but I'm afraid that sound is being drowned out by the sound of cash registers. Ka-ching.

NC: Spellings Sinks To a New Low

Margaret Spellings, the queen of NCLB, the career politician that followed Bush II from Texas to DC, the Secretary of Ed who pushed policies that she had watched fail in her home state, has added one more loathsome act to her resume.

You may recall that Spellings was installed at the once-admirable University of North Carolina in a piece of purely partisan politics. As part of the ongoing work to stamp out every progressive, liberal or humane molecule in NC, Governor Pat McCrory and the GOP legislature have been stomping on public education in an unrelenting series of attacks. 

But they really needed to bring UNC to heel, so to that end, they canned the previous well-respected president and installed, by means that failed any test for transparency, Margaret Spellings.

The list of reasons to be unexcited about her crowning was long. There was the whole NCLB thing-- a massive policy failure that she has continued to defend past the point of any sense or reason. She has had ties to the predatory for-profit college industry. She has hung out with the Boston Consulting Group, and she has generally supported the notion that education is a like any widget or consumer good, except that the education system needs to have its inefficiencies wrung out so that people will do more with less. As I've written before, Spellings either doesn't know baloney when she sees it, or she is determined to make a living selling it.

Oh, and there was that time, as Secretary of Education, that she called up PBS and told them to get that lesbian couple off the air.

So Spellings' response to North Carolina's new law legalizing LGBT discrimination is no surprise.

As president of a major university, she could have done any number of things. She could have taken a bold and brave stand and said something along the lines of, "You can pass a stupid law like this, but you can't make me enforce it on my campus." She could have taken a somewhat less brave, more bottom-line practical stance and said something like, "We are a national university and I won't have the legislature's backwards law reduce our stature and interfere with our ability to recruit the best and the brightest of students from all across the country, regardless of their gender orientation."

She could even have taken a not too brave stance by saying, "Well, let's wait until the court challenges are done" or just pretending not to notice the law and just going about her business.

Spellings did none of those things. Instead, she sent this directive to the university chancellors, laying out how she expected them to comply with the ridiculously titled Pubic Facilities Privacy & Security Act.

University institutions "should take" the following actions:

a. Designate and label multiple-occupancy bathrooms and changing facilities for single-sex use with signage

b. Provide notice of the Act to campus constituencies as appropriate.

c. Consider assembling and making information available about the locations of designated single-occupancy bathrooms and changing facilities on campus.

The memo notes that there is a lawsuit currently pending against the governor, the attorney-general, and the university. Plaintiffs include a student, a faculty member, and a staff member from a UNC institution. The university expects to work with the AG office to get counsel and fight the lawsuit. In the meantime, they will uphold the law.

The memo also provides direction if anyone in the system is contacted "by a federal regulatory agency concerning the Act and its implementation," (something that could occur because, I don't know, the Act is pretty damn illegal). At any rate "if your institution is contacted by a federal agency with a question about the Act, please notify the Division of Legal Affairs at UNC General Administration." So if the feds come sniffing around, don't you dare talk to them-- refer them to the UNC Office of Official Stonewalling.

Is it a surprise or disappointment that Spelling shows little moral fiber or leaderly backbone? I suppose not. But it certainly is one more instance in which she is firmly on the wrong side of history, decency and basic American values. She has found one more way to bring shame to a once-proud UNC.

College Debt, Regret, and Readiness

One third of millennials say they would not have gone to college at all if they had known the debt burden they were going to be left with.

That's one of the big takeaways hit by Bloomberg today, reporting on a survey conducted by TNS "on behalf of" Citizens Bank (I love that "on behalf of," like TNS just felt like doing the bank a big ole favor).

The rest of the picture isn't any better. The average student debt for the group of 18 to 35 year olds is $41,286.40 (a spectacularly specific number). This matches up with numbers from other reports which also say that A) about two-thirds of millennials are carrying that kind of debt and B) it is messing up their lives.

TNS reports that millennials don't all have a firm grasp of exactly what's happening with their debt. 15% of those polled weren't sure exactly what their balance was, and over a third didn't know their interest rate. Almost half don't understand the intricacies of private vs. federal loans.

I can report that all of this is right in line with my personal store of anecdotal data. I gave my millennial children the gift of paying for their college educations, and by the time my first child was halfway through, I had eaten through all my savings and commenced with the loan out-taking. My second child has a degree that is entire a product of Parent Plus loans. I've been at this game for a while (and it will be a while before I'm done).

My loans have been pretty straightforward, relatively speaking, but these reports often forget that for many millennials, the rules on much of this changed along the way. While my son was in school, for instance, the rules shifted and we went from dealing with a well-trained by anonymous functionary of a financial institution to dealing with a shell-shocked employee of the school's financial aid office. At numerous points my loans were passed off to a new institution, which resulted in changes in amounts and loan numbers (at one point, I confess, I discovered that I had been double paying on one loan and was months behind on another).

But mostly, I look at the amounts I had to borrow, and I cannot begin to imagine how my children-- and other people like them-- could possibly have had a live at this point if they were paying these loans on their own. They struggled through most of their twenties in minimum wage jobs; one happened to be in love with and finally married to a young man with a well-paying gig, but the other periodically had to borrow from the Bank of Dad just to make ends meet. I remain painfully, acutely aware and angry at the many ways that corporations like banks and utilities and the various phone companies find ways to gouge money out of people are doing their best but can barely manage. What kind of conscience-deprived business plan involves the grown-up equivalent of knocking down the little kids and taking their lunch money?

Some folks offer their idea of a practical solution-- only go to college for degrees that will immediately result in lucrative jobs. This is bad for the humans getting the degrees, and bad for a society that depends on a full range of talents, skills, and jobs to make civilization possible, but pays very poorly for many of those jobs. And if you don't buy my larger philosophical objections, consider the practical one-- if every student at Bodacious University drops their old major and switches to the two that lead to well-paying insta-jobs, that job market will be flooded, and those jobs will no longer pay well.

I don't think I know any millennials who regret going to college, but what does it say for our grand and glorious dreams of national college attainment that so many debt-ridden college grads (and drop outs) are out there saying, "Don't go, kid. It's not worth it."

And why can we move heaven and earth and try to rewrite the entire K-12 education system in pursuit of academic college readiness for all, but we can't lift a finger to promote the goal of financial college readiness for all. Why have we focused so much fierce attention on making sure that Chris can pass college algebra, but so little attention on whether or not Chris can afford to pay for it.

I have no doubt that this is neither an easy problem to diagnose or to solve, but this survey drives home once again that we have an entire generation of Americans for whom college costs are the biggest problem in their lives. They can't afford to put money into the economy. They postpone buying homes and having children. They struggle with the stress and strain of living under the shadow of huge debt. How can the fact that some are required to take remedial college courses be a huge issue that must be screamed regularly from the rooftops, but the house of debt that has been dropped on them (amidst promises that college would be their passport to te middle class) merit barely a mention?

How can we pretend to talk about making students college and career ready and not talk about the crushing cost of college?

Do Charters Make Graduates Richer?

Courtesy of the Journal of Policy Analysis and Management (and not any kind of education related journal) comes a new piece of headline-generating baloney research built to make charter schools look good.

"Charter High Schools' Effects on Long-Term Attainment and Earnings" comes to us from researchers at Georgia State University, Vanderbilt University, and Mathematica Policy Research (always a reliable source of Gates-funded/friendly research), with funding from the Joyce Foundation, and appears to be a revisit of some earlier Mathematica research-shaped product. Their conclusion, coming soon to a headline near you, is that charters lead to more college attainment and more money.

How, you may wonder, can anybody actually research such a thing. After all, the big problem of any research on human stuff is finding a control. We can say that Chris went to a charter school, then went to college, then got a great job as VP of Widgetary Development at World Wide Widgets. But none of that tells us what would have happened to Chris if Chris had attended a public school instead. And unless we can find an exact doppleganger of Chris to follow along an alternate path, or a means of slipping into an alternate universe, we have no way of knowing. Which means we have no way of knowing.

The researchers acknowledge this by opening their methodology section with the phrase "Determining the impact of charter high schools is not easy..." which is true. It is also the last thing they will say in plain English throughout the entire methodological description. Seriously-- I just spent my lunch hour with Les Perelman's BABEL nonsense generator, and this seems ike it might have come from that same source.

Okay, maybe it's not that bad. They do write this:

The fact that the charter students and their parents actively seek an alternative to traditional public schools suggests the students may be more motivated or their parents may be more involved in their child's education than are the families of traditional public school attendees. 

They probably should have quit right there and called it a day. But they didn't. They tried to correct for selection bias, and here's some of what they had to say about that. Here they are rejecting one method:

Two recent studies (Furgeson et al., 2012; Tuttle et al., 2013) have demonstrated that longitudinal analyses of test score impacts that control for pretreatment test scores can closely replicate randomized experimental impact estimates for the same students. But this approach cannot be used to measure long-term outcomes such as graduation, college enrollment, college persistence, and employment, because those outcomes do not occur before a student's enrollment in a charter school.

They talk about how to generate a strong comparison group, which involves looking at charter eighth graders, because reasons. This creates some external validity problems, but in their opinion, the sacrifice is worth it for increased internal validity. They seem to think this is good because it catches students before the transition to high school. Does this not make sense yet? Well, this should settle it for you:

To further deal with potential endogeneity, we also use a matching approach popularized by Rubin (1977) and Rosenbaum and Rubin (1983). While matching procedures can take many forms, we use a one-to-one nearest-neighbor Mahalanobis matching approach (also referred to as a covariate match) in which we match on observable characteristics to create a control group. We then examine difference in student outcomes between those in treatment relative to this counterfactual control group. 

Also, there is math.

The actual data used came from Florida, which covers both high school graduation and, for anyone who has unemployment insurance records, data about employment earnings. The research centered on four cohorts in eighth grade between 1998 and 2002. So this research should be very meaningful, because, really, not much has changed in education in Florida in the last 15-18 years, right?

I tried to answer that, but much of Florida's charter info only goes back ten-ish years. Florida's modern charter law was passed in 2002 (Jeb! Bush was governor from 1999-2007) replacing the first version from 1996. In the 1998-1999 school year, there were a total of 67 charter schools in Florida, and only 20 of those had an eighth grade. Total charter students-- 9,135. By 2001-2001, charters had ballooned to 176, with over 40K students. But still-- I'm wondering just how large a sample the researchers were able to pull out of that.

On top of that, charters were relatively small potatoes, which means that charter students would have been a not-at-all-average group. I'm not a statistician or scholarly researcher (nor do I play one on TV), but I can't escape the notion that the same kind of parental support and push and resources that would get a student into a charter school (particularly back then) would be the same kind of parental support and push and support that would get a student through high school and into college.

In other words, I am once again inclined to conclude that a lot of very fancy researchers and scholars do a lousy job of distinguishing between correlation and causation.

I will gladly accept input from anyone who is actually a trained statistical design scholar, but until someone I can trust tells me otherwise, I'm going to conclude that this is high-priced baloney served on a silver platter.

Segregation, Choice and Education

The National Education Policy Center just released a research report from William J. Mathis and Kevin G. Welner on the question, "Do Choice Policies Segregate Schools?"

Spoiler alert-- the short answer is "Yes."

A review of the research and literature led Mathis and Welner to conclude that while some choice schools are integrated, charters largely are very segregated. That segregation can be by race, poverty, dual language learners (ELL), and students with disabilities. While black students are generally either under-represented or over-represented in charter schools, the researchers found that poor, ELL, and students with disabilities are under-enrolled in charter schools. Within a choice system, both segregation and the achievement gap grow.

However, before charter foes leap on those results, there is this to be considered:

Even without school choice, America's schools would be shockingly segregated, in large part because of housing policies and school district boundaries. School choice policies that do not have sufficient protections against unconstrained, segregative choices do exacerbate the problem.

In other words, charter-choice systems may be making things worse, but they certainly didn't create the problem. Or to put it yet another way, when anti-charter folks say that charters are creating massive segregation problems, they are correct. And when charter fans say that housing-based choice is creating segregation problems, they are also correct.

There may be valid arguments in favor of some charters in some situations, but the "we will fix segregation and close the achievement gap" argument is not one of them. Charters clearly do neither of those things.

Public schools aren't getting it done, either. But in order to look for real solutions, we need to stop pretending that fake solutions are actually working.

 

Grading Good-faith Gibberish

Les Perelman is one of my heroes for his unflinching exposure, time and time again, of the completely inadequacy of using computers to assess writing.

Perelman and his grad students create BABEL, (the Basic Automatic B.S. Essay Language Generator) a program that can generate brilliant gibberish. Diane Ravitch, education historian and activist, took a stab at using BABEL and got, in part, this:

Didactics to subjugation will always be an experience of humankind. Human life will always civilize education; many for diagnoses but a few of the amanuensis. Myrmidon at absurd lies in the search for reality and the realm of reality. From the fact that denationalization excommunicates the denouncements involved of civilizations, humanity should propagate absurd immediately.

This scored a mere 4 out of 6. Apparently Ravitch, as an older Americam, suffers from being the product of our earlier status quo education system. If only she'd been exposed to the Common Core.

The software that scored her essay is PEG writing, and the site has some lovely FAQ items, one of which Ravitch highlighted.

It is important to note that although PEG software is extremely reliable in terms of producing scores that are comparable to those awarded by human judges, it can be fooled. Computers, like humans, are not perfect.

PEG presumes “good faith” essays authored by “motivated” writers. A “good faith” essay is one that reflects the writer’s best efforts to respond to the assignment and the prompt without trickery or deceit. A “motivated” writer is one who genuinely wants to do well and for whom the assignment has some consequence (a grade, a factor in admissions or hiring, etc.).

Efforts to “spoof” the system by typing in gibberish, repetitive phrases, or off-topic, illogical prose will produce illogical and essentially meaningless results.

In other words, PEG knows it doesn't work. It also assumes a great deal in assuming that students writing pointless essays on boring subjects for baloney-filled standardized tests are "motivated" writers. Can the software accurately score motivated gibberish? Can the program distinguish between frivolous garbage and well-meant garbage?

Probably not. As noted in PEG's response to the question of how the software can evaluate content:

However, analyzing the content for “correctness” is a much more complex challenge illustrated by the “Columbus Problem.” Consider the sentence, “Columbus navigated his tiny ships to the shores of Santa Maria.” The sentence, of course, is well framed, grammatically sound, and entirely on topic. It is also incorrect. Without a substantial knowledge base specifically aligned to the question, artificial intelligence (AI) technology will fail to grasp the “meaning” behind the prose. Likewise, evaluating “how well” a student has analyzed a problem or synthesized information from an article or other stimulus is currently beyond the capabilities of today’s state of the art automated scoring technologies.

PEG bills itself as a "trusted" teaching assistant that can help relieve some of the time pressures that come from having many, many essays to grade. But I can't trust it, and it's unlikely that I ever will.

This is the flip side of Common Core reading, an approach that assumes that reading is a batch of discrete behaviors and tricks that are unrelated to any content. Here we assume that writing is just a series of tricks, and it doesn't really matter what you're writing about, which is a concept so bizarre that I can barely wrap my head around it. Use big words-- even if they have nothing to do with the topic of the essay. Use varied sentence lengths-- but don't worry about what the sentences say. 

PEG, like other similar services, offers as proof of its reliability its closeness to human-rendered scores. But that happens only because the human-rendered scores come from a rubric designed to resemble the algorithm of a computer, not the evaluative processes of a human writing teacher. In other words, you make the computer look good by dumbing down the humans used for comparison.

Pearson's continued fascination with AI-directed education, as well as the news that PARCC will use computer essay grading in four of its six states-- these are Bad News, because computer software is simply not up to the job of evaluating writing in any meaningful way. BABEL is just one more demonstration of how completely inadequate the software tools are.

P.S. My favorite line from my own BABEL efforts:

Charter, frequently to an accusation, might innumerably be grout for the allocution.