More Bogus Research from Rocketship

When you need some "research" to pump up your ad copy, what better than to just hire it yourself. In fact, why not hire the guys who have been pumping it out reliably for you for years.

Rocketship Academy is patting itself on the back over a new "research" report that it commissioned from good, old SRI International, a group that just happens to be heavily invested in technology-based education. SRI used to stand for Stanford Research Institute, but separated from Stanford in 1970 and changed its name in 1977 (which suggests the split was plenty amicable). That was long before Preston Smith tacked his Teach for America credits onto his resume and looked to enter the lucrative world of edubusiness, but since then, SRI has teamed up with Rocketship to show the awesomeness of the Rocketship Academy product.

Way back in 2011, SRI did a super-duper study of the K-5 Rocketship to show that the Dreambox program (Dreambox because it's what you bury dreams in?)  raised scores on the NWEA MAP test by a couple of points. Dreambox is a company that Reed "Elected School Boards Stink" Hastings (Netflix) bought for the Charter School Growth Fund, a fund that also invests in Success Academy and Rocketship Academies. We could play "Follow The Incestuous Privatization Ties" all day, but we'll pass for the moment. We'll also let the usefulness of NWEA results ride for a moment (though you may recognize the name from the successful boycott of the test by Seattle teachers). SRI was back with another glowing report in 2014 and were touting further awesomeness in 2015.

It's almost as if SRI was a corporate partner of Rocketship rather than an "independent nonprofit research center," and indeed, way back in 2010 we find Rocketship then-CEO John Danner  (now CEO of Zeal-- more in a second) explaining in an interview that they are teaming up with SRI

 Next year, with SRI’s help we’re going to instrument Rocketship to be a test lab where we can measure the effectiveness of every online curricula for elementary schools.

Funny side note. Zeal started out as one more adaptive instructional software start-up, with money from NewSchools Venture fund and other folks interested in a good ROI on their edubiz dollars.Visit their site today and you'll find them headlining "live, on-demand coaching" from "real coaches." So I guess Danner has lost a little of his faith in soft-ware based education.

Fine. Whatever. What about that New Research?

So Rocketship's use of SRI as an independent evaluator is about as fishy as the "volunteer" pulled out of the audience by the bad magician at the company picnic. How about the actual research? Does it look plausible anyway?

Short answer: not so much.

What the research claims is that Rocketship middle schoolers have gained a full year of learning over their peers.

After controlling for demographic differences and other key variables, SRI found that Rocketship alumni outperformed their classmates on NWEA MAP, a nationally norm referenced assessment, by approximately one year of academic growth in both math and reading. This independent evidence from a world-renowned research institute is a powerful validation of our pioneering personalized learning model. Our incredibly dedicated teachers, deeply engaged parents and purposeful approach to technology are delivering significant gains for our Rocketeers that extend into middle school.

Let me pause a moment to nod at my usual ridicule of the notion of measuring learning growth in years, and especially in parts of years. Which part of the year? Did the student gain an April or a September? The notion that a year of learning represents some sort of homogenous tofu-like spread of steady increase only make sense to someone who doesn't know many human beings. It's a statistical construct, like saying that one car is stolen every fifteen minutes, as if car theft is a regular ticking event by which we can measure time. "How long have we been trapped in this dungeon, Chris?" "I'm not sure, but I think Pat's learning has grown about this much, so I figure around three months."

But let's look at the experimental design.

The study population included Rocketship alumni enrolled in one of seven participating charter middle schools in the San Jose area between the 2012–13 and 2015–16 school years, totaling 625 students. The comparison group of non-Rocketship peers was made up of students who did not attend any Rocketship elementary school and who attended the participating charter middle schools in the same grade level and in the same academic years as the Rocketship alumni in the study, totaling 1,294 students.

There's some other baloney about controlling for demographic characteristics "using propensity score weighting and multiple linear regression" (I just hope the reset the turginator on the framistan before they did all that), but the bottom line here is that they took a bunch of Rocketship middle schoolers and compared the ones who had attended Rosketship elementary school to the ones that hadn't.

From this we can conclude that students who attended Rocketship Academy elementary schools were better prepared for Rocketship Academy middle schools than students who did not.

Other research also indicates that at most Rocketship Acdemies, the sun rose to the East of the school. Early working papers also suggest that water is wet.

Also, grit!

Just in case you think Rocketship is only focused on the quality of their test prep for a standardized math and reading test, they'd like you to know that they are also looking into the hearts of Rocketshippiteers.

Of course, test scores are only one indicator of future success. That is why we embed our core core values, socio-emotional curriculum, and positive behavior interventions and supports throughout our Rocketeers’ learning experience. And while these skills are harder to measure, they still matter tremendously, especially for the student population we serve.

Emphasis mine. And the emphasis is just in bolding because I don't have an emoticon for a dropping jaw. "Especially for the student population we serve"??!! You mean poor minority deficient defective kids? Good behavior, grit, socio-emotional qualities-- the rich white kids don't need that? On the one hand, there's a bit of a point here. We have living proof that in our society, a person born into privilege can display every socially undesirable quality known to humankind and still have a shot at being both successful and also Presidential material. On the other hand, it kind of looks like Rocketship needs to work on its savior complex and its deficit thinking model for approaching non-white, non-wealthy students. But then, wealthy kids are not the Rocketship market, nor are the privileged crowd lined up for a chance to partake of this educational awesomeness. The whole model reeks of a low-cost edu-product that's good enough for Those People. Every once in a while they just forget to avoid saying so out loud.

Also, where Smith wrote "hard to measure," I think that's a typo. "Impossible to measure" is undoubtedly what he actually meant.

Bottom Line?

It's more charter marketeering fluffed up with science-flavored PR filler. It's dishonest and not very useful in adding to a real conversation about meeting educational needs or evaluating the actual impact of charters in general and the Rocketship blended plunk-kidsd-in-front-of-computers model in particular. I'm sure we'll see this thing passed around in the weeks ahead. Do not be fooled.

Rot and The Yearbook Biz

Being a yearbook photographer was one of the great joys of my high school career, so I naturally gravitated toward yearbook advising as a teacher. I've been at it for about sixteen years, and I've watched the business change.

Some of the changes are not what folks might have predicted. Technology has been about to destroy yearbooks for years, but there has always been a problem with that-- when I first started, our publisher was trying hard to sell us the newest innovation, and that was a "digital yearbook" housed entirely on a cd-rom. Many schools jumped on that bandwagon, and I'm sure those cd-roms now make excellent coasters and scarecrow decorations. Meanwhile, I can go into our archive and look at fifty, sixty, even seventy year old books and still find that the operating software (my hands and eyeballs) is still working just fine.

Meanwhile, students still love pictures. We have to provide something a little cooler than the selfies and snaps that they share daily on their phones, but as a percentage of the student body, our yearbook sales have stayed pretty solid.

Most of the significant changes are changes the yearbook reader can't see. We produce the book digitally on-line at tremendous savings (when I started, we were spending thousands of dollars annually on photo printing), and my students get practice and training in layout and design. I am one of the fortunate few to have yearbook as a scheduled class during the day, and we build our book up from scratch-- no templates or clip art or anything not produced by the students themselves. The computerizing of all this has created a hugely rich and rewarding educational experience for my students.

But other changes in the biz are not so rich and rewarding-- not for students, anyway. I'm doing one of my every-so-often look at the market to see whether it's time to change companies again, and I notice a common thread among the big companies. Let's see if you can spot it, too!

Taylor Publishing

Founded in 1938 by the three Taylor brothers, who had become interested in yearbook publishing as boys, bought an engraving company, explored some different printing techniques, and finally settled on lithography books which helped them to explode into one of the major players in the business. At times, they were bringing in customers faster than they could serve them. By the seventies the now-public company was suffering from growth without modernization. In the eighties they struggled with Chapter 11, but by the nineties had emerged from that with a strong footing in computer-aided printing and were doing quite well.

Then in 1999 the company was scarfed up by Castle Harlan Partners III, a division of Castle Harlan, Inc, a merchant bank. Castle Harlan is a "Global Leading Private Equity Firm" which is "distinguished by disciplined focus on investing in control positions in middle market private companies."

So Taylor Publishing is no longer in the yearbook business, but is instead in the making money for investors business.

Herff Jones  

Back in 1920, Harry Herff and Randall Jones started an insignia company that specialized in school hardware like letters and bars and trophies. They didn't get into the yearbook biz until the 1960s and were perhaps a little late to the digital production party, but they were an employee-owned company, which gave them a nice homegrown feel. They did try an on-line digital version of the book, but that was not such a hit (who knew that people don't like to pay for online content--oh, no, wait-- everybody!). They rebranded themselves a Varsity Brands, with the yearbook as a division of that conglomerate. In 2014, under a new CEO, they sold off several divisions (Lifetouch photography, the ubiquitous church directory outfit, is a Herff-Jones offspring).

Then in December of 2014, the whole company was bought by Charlesbank Capital Partners.

Charlesbank Capital Partners is an investment group with a "broad focus," and they have "A consistent track record of creating value in middle-market companies."

So Herff Jones is no longer in the yearbook business, but is instead in the making money for investors business.

Jostens 

Jostens is the 800 pound gorilla of the yearbook biz. In the late 1800s, Otto Josten looked to expand his jewelry business into school memorabilia, and by 1906 he was in the class ring business, which grew into a hugely lucrative business of Otto. In 1950 they set up the American Yearbook Company which quickly stomped all over the marketplace. The next few decades paralleled the whole industry's story-- trouble with management succession in the seventies, financial problems in late eighties-early nineties, expansion into computer stuff. Bought by Bain Capital, then soon bought themselves back and tried to make Wall Street happy, with mixed results.

Then in 2015, Jarden bought up Jostens. Jarden was a "spin-out" company of Ball Corporation that owned a variety of companies including Yankee Candle, Rival and Mr. Coffee. Jarden itself has just completed a merger/absorption by Newell Rubbermaid, putting Jostens in the same pen as Sharpie markers. When contemplating the move, Bloomberg offered this quote:

“This deal is about creating scale,” Chris Ferrara, an analyst for Wells Fargo & Co., said Monday in a research note. “Jarden is a portfolio to which Newell can apply its blueprint of integration, cost savings and selective reinvestment for growth.”

So Jostens is not in the yearbook business, but in the making money for investors business.

So what?

The effects of all these corporate moves have been visible on the local level. My school has dealt with all three of these companies at one point or another over the past few decades, and all three have been marked by a steady decrease in quality of service accompanied by an increase in cost. We dropped Taylor years ago because they increasingly made mistakes with our book. Jostens was at first a nice fit with a production plant right in State College, but that facility has been shut down. All companies have billing and business departments that appear to have no working relationship with the sales or production departments at all.

Meanwhile, longtime sales reps are pushed to get out of the field to make room for newer, younger and presumably cheaper replacements, all of whom are pushed to sell, sell, sell. None of this helps the rep's relationship with the local yearbook adviser, which is rough on everybody because those reps are the point of contact between the company and the customer. How much can you trust someone to look out for your interests when they are busy trying to protect their own back?

Choices?

There are other smaller companies out there, but without naming names (because that would just be mean) some of them just don't create a very good product or offer very good tools. Jostens, for all its corporate faults, has a design product co-developed with Adobe, meaning students can get real preparation for digital publishing in the real world. That's not nothing. But beneath all of it is a sense that there is roiling and mess beneath the surface. I've had two seasoned reps pushed into retirement. I've met reps who just finished jumping from one company to another. My last rep quit suddenly this fall after just two years after leaving his previous company. Now my new rep is a guy whose LinkedIn profile tells me that he is also a recent refuge from another company.

The bottom line is that the more I read, the more I understand that none of the Big Three are as interested in helping me produce a school yearbook as they are in using me to get their numbers up. Maybe that dose of the real world is a useful lesson for me and for my students, but it's a little depressing and it makes the choosing that much more difficult.

It's also a reminder that there's free market and there's free market. All three always existed in the open market, subject to competition and market forces, but braving that sea with ships built of quality yearbooks. Now they're just the patches on ships that are sailing the free market in search of quick profits and ROI. Surviving the free market by serving customers is way different from serving investors. It's a reminder that it's not just the idea of free market schools that is problematic, but the idea of free market schools that are meant to serve investors and hedge fundies.

It's an ugly, destructive rot, and it's spreading everywhere.

Familiar Miracles and Ed Tech

I started typing this a few thousand feet in the air somewhere over Illinois. I worked all day the day before to get things ready to go, got up well before dawn, drove myself to Pittsburgh, where I boarded the first of two planes that will transport my to Seattle for the next several days. Now, after spending many days on the other side of the country, I am finishing up this post thousands of feet in the air, connected to the internet on a computer-based device.

Every step in that process is a small miracle. I could travel to the airport in about 90 minutes because of the big, smooth interstate highway system, as well as the technological marvel that is my car. Now I'm in an airplane, off to try to help out with my daughter's family, which recently upgraded from one nearly two-year-old to one nearly-two-year old and a new-born infant. I am going to miss my wife a great deal, but we can exchange pictures and words any time we like because smartphones. And I'm writing about this on an airplane that has Wi-Fi. Before we took off, I used a pocket-sized, wireless network enabled, computer to tell my family in several parts of the country how my travels were progressing.

We live in an age of miracles, but then, human beings usually do. Henry David Thoreau's family was in the pencil, a technological marvel that allowed people to write or draw with a handheld piece of wood that had ben implanted with just the right amount of material to make easily controlled marks. Thoreau thought the world was already moving too fast for people to properly appreciate it. And it was Thoreau's friend Emerson who noted that if the stars only came out every few hundred years or so, they would strike humans with overwhelming wonder and awe (and, added Isaac Asimov, panic and hysteria).

Miracles are a relative thing. On this plane I'm surrounded by people reading cheap, readily available books.  The fact that a million identical copies of a book can be printed and bound so that the very same reading experience can be available to just about anybody-- that's miraculous, but it's also been around for centuries, so very few people feel very much excitement contemplating a book.

Modern familiar miracles give us many responsibilities as teachers. For one, I consider it part of my job to remind my students that they live with miracles, many of which are relatively new. My eighth grade social studies teacher, Mr. Confer, frequently regaled us with stories of his boyhood, and we thought we were getting out of class every time he opened his mouth. Only later in life did I realize he had taught a hugely difficult lesson-- the world was not always the way it is now, and it will not always be this way.

It's also part of our modern gig to remind students that new miracles mean new possibilities. When miracles become familiar, they become transparent, and we forget to look and see and consider what else they can do. I-79 runs down to Pittsburgh, but it runs many other places as well, and if I think of it as just "the road to Pittsburgh,": I miss that. My students carry their own pocket computers, but for some these miraculous devices are simply their Snapchat machine, or their Instagram Access Device. They absolutely forget that other uses are possible.

Reminding these students what miracles can be achieved can also help us remember that devices are only as interesting as their uses. Familiar miracles don't look like miracles at all, and boy do we forget that. Since the first computer landed in a classroom right up until today, this minute, there have been folks who believed that a banal, bland, boring worksheet becomes an explosion of educational awesomeness if we run it through a computer.

It does not. Imagining that your stupid educational idea (let's have a really huge bank of worksheets and hand each student a specific individual one based on how she did on the last one we gave her) will be an awesome idea because you're doing it On A Computer -- that's dumb. Even dumber than saying, "This lesson will be super-engaging if we print it in a book!"

Tools are conduits, no better or worse than the pedagogy we send through them. A speedy delivery system can deliver crap as quickly as it delivers gold, but it doesn't transform one into the other. The fact that we find modern computer tools miraculous in and of themselves really means one thing-- we are old. Each generation experiences its own miracles, and each generation gets to see its miracles become dull, commonplace, familiar.

Yes, we should work hard to preserve and share a sense of wonder, an appreciation for miracles. But in the classroom we must recognize that our favorite miracles may retain little power to amaze in and of themselves (well, unless you're teaching the little ones-- then everything is new and amazing). That's the challenge-- to remain familiar but not jaded, amazed but not foolish. And now I could say more, but here comes the food cart. I will see you all later, back on earth.

NY: Commissioner Struggles with Teacher Shortage

Maryellen Elia was brought in to the Empire State to help clean up the mess that former education honcho John King left behind.

Her results have been mixed. She invested a lot of energy in defending the Big Standardized Test, even issuing a handy propaganda kit to help push down those opt out numbers. It didn't work.

Last Friday, Elia talked to the NYS Association of Teacher Educators and the NY Association of Colleges for Teacher Education about some of the issues tied to the looming (or possibly not, or possibly already here, depending on who's talking) teacher shortage. As reported by the Times-Union, some of what she had to say was on point, and some of it indicates that she still doesn't understand the situation.

On the plus side, Elia is one policy leader who understands that more than decade of beating up on teachers and the teaching profession has not exactly made the field attractive to folks.

But folks at the meting also talked about a side-effect of test-driven accountability that has been long-predicted and now visibly concerning. Placing student teachers is becoming increasingly difficult in a world where high stakes testing season runs the schools. How can I possibly turn my class over to a pre-rookie when so much about the future of my school and my own professional career is riding on what's happening in my classroom. And the focus at this meeting was on student teachers who are hard to place at al-- that's before we even get to student teachers who get a placement, but who are not allowed to do much of anything because of concerns about the BS Tests.

Elia said that the department is working on "several efforts to boost district cooperation." But the department also made efforts to boost district cooperation and participation with the BS Test-- and they failed.

Elia addressed the exceptionally confusing mess in NY regarding subject area tests, which are transitioning except when they aren't, and which allow students to take one of a couple of options, their choice best based on judicious use of a Ouija board, since even education professors have thrown up their hands in confusion. There is no report on whether anyone specifically called Elia out on New York's use of the giant baloneyfest that is the expensive-but-useless EdTPA tests.

Elia was also called on the defend the new system by which out-of-state teachers can waltz right into a NY classroom with no additional tests or paperwork, meaning that the absolute best path for someone who wants to teach in NY is to go get certification in some other state.

Teacher education schools are understandably grumpy-- this is a direct assault on their market base. Elia defended the practice by pointing to the teacher shortage, and in her defense, I should point out that NY's solution is still far better than the many states flirting with or embracing the practice of simply issuing teaching certificates to any upright hominid with a pulse. That's what we've come to in education-- any discussion of bad policy can eventually lead us to observe that it could be worse and, in fact, somewhere (probably Florida or North Carolina) it is.

However, "not the worst policy around" is a low hurdle to clear. While from the standpoint of a Pennsylvanian who knows that our teacher ed programs are having trouble rounding up students, I think New York's policies are just swell. But as someone who cares about US public education, I'm not so impressed. Maybe as a veteran of some giant Sunshine Stare disasters, Elia can't help but cast a rosy gaze on New York policies, but I would encourage her to shoot for a more ambitious slogan than "New York education-- At Least We're Not Florida." That is not how to recruit folks for the profession.

ICYMI: Reading in the Seattle Dew (10/23)

I'm far from home, but there's still plenty to read on the interwebs.

Our Stupid Questions

Teacher Tom with a striking twist on the old "no such thing as a dumb question" shtick.

How Joy Became the New Grit

Jennifer Berkshire gives us a look at how the world of grit is giving way to something even worse

Talk about Passion

Kristen Perkins guest-blogs at Blue Cereal Education with a piece about passion in teaching.

Reflections from a Nasty Woman

Nancy Flanagan with a brave and honest post about the price women pay for coming forward in matters of sexual harassment. If you're only going to read one item on this list, this is the one.

What New Challenges To The Charter School Industry Reveal 

These were supposed to be the salad days for the charter business. Jeff Bryant takes a broad look at why that's not happening.

The Value-Added of Teacher Preparation Programs

Audrey Amrein-Beardsley takes a look about what new research tells us when it comes to using VAM to evaluate teacher education programs

You Can't Teach Writing from the Side of the Pool

Did we just talk about this last week? Sure we did, but it's a point worthy of repeating. Russ Walsh hosts a guest post from Cynthia Mershon.

More About Attrition Rates in Boston

Mark Weber (Jersey Jazzman) is back to once again cut through the smoke and mirrors of the Massachusetts charter debate.

The Best-Laid Plans of Grown-Ups

This after noon we took the grandsons to a playground. It's a lovely playground, one of many, many lovely playgrounds available in Seattle. Here's a look at just some of the cool playground stuff available there.

And here is how my oldest grandson spent a good chunk of his time.

It's a well-flogged truism that children will throw away the toy and play with the box, that they will reject the finest plastic construction that the toy industry can muster in order to play with ordinary household objects. I suppose that somebody could have forced my grandson to drop the stick and play "properly" but why, unless they were intent on imposing adult will and plans on a child. "I planned on you playing on that jungle gym over there. Now put down that stick and go have fun, dammit, or else."

The bottom line is that children have instincts and interests and involvement of their own. Adults can go nuts trying to direct that, and they can twist children's brains up by hammering them withy messages about what they are "supposed" to do. 

It is certainly true that there is room for adult direction and guidance. My grandson played with some of the equipment and played with his father, who did not try to tell my grandson what to do, but joined wholeheartedly in helping my grandson tap into his transcendent joy over swinging.

But if you go to the playground armed with an adult agenda that allows no room for the voice of the children, you are on the wrong path. The damage is evident by the time students land in my eleventh grade classroom and have trouble writing well because they are more concerned about what they are supposed to write-- what they are supposed to do to meet the requirements of the grown-ups' agenda-- instead of tying to get in touch withy what they actually think.

It is easy as parents or teachers to get caught up in the desire to see the tiny humans make the safest, wisest, best decisions. But that process has to include their own voice, their own aims, their own intentions and inclinations. That's not just how you honor their existent as thinking, feeling, sentient, individual human beings-- it's how you create future entrepreneurs, leaders, creators, makers, employees, employers, and people who are not inclined to elect raging tyrants out of desire to have "strong" leaders who will tell them just what they are supposed to do.

Yes, the world needs a certain amount of order and sense, and I am not advocating unleashing wild anarchic chaos on the universe (not today, anyway). But attempting to impose adult best-laid plans on every minute of children's lives is both evil and foolish. Evil, because every human's voice is a precious thing no matter how young. Foolish because-- well, I will give my grandson the last word with his ideas about how to use carrot slices.

Nobel-Winning Evaluation Advice

Sometimes it takes a Nobel Prize-winning economist to confirm what many of us in education have been saying for years-- reformsters are doing teacher and school evaluation all wrong.

The writer pointing this out if Derek Neal (University of Chicago), over at Education Next of all places, The prize-winner he's talking about is Bengt Holmstrom, a Finnish economist at MIT.

This guy has more Nobel Prizes than you

Holmstrom has done pretty much no work on education. What he has done is work with contract theory, particularly with contracted with incentives in areas "where worker actions are hard to observe and worker output is difficult to quantify." So, totally education.

Neal focuses on two major elements of Holmstrom's work, providing two insights into how an incentive pay system can go wrong.

1) Good incentive systems use all the data that provides more clarity and detail about employee performance. So, not just some sliver of data, and not data that doesn't really improve the picture of job performance.

2) There must be alignment between the performance task and the actual desired task.

Both which tell us that the test-based evaluation that has been favored since NCLB was a pup is seriously off the mark.

So, if out of the full range of teacher behaviors, you collect only data about how students do on a narrow reading-and-math test (which is also measures a mess of data unrelated to teacher performance), you cannot build a good incentive system on that data.

And if, for instance, you are not measuring how well students read, but rather how well they answer multiple-choice questions, your data is not suitable for creating performance incentives.

Basically, Holmstrom provides a fancy explanation of why Campbell's Law is a thing and how it works. When you measure the wrong thing and/or measure the right thing incompletely, you incentivize the wrong behavior, and you get lousy results.

Holmstrom also notes that in some settings, collecting enough of the right data can be really prohibitively expensive. So what to do instead?

In these settings, the best approach may be to adopt hiring procedures that identify workers who will perform well in order to satisfy their own personal norms and the norms espoused by the organization, and then pay these workers a fixed salary.

What?? Hire competent, self-directed people with solid training and then just pay them well?! That's crazy talk, you Nobel-winning loon.

Neal also notes that reformsters are more focused on doing evaluation than on doing it correctly, but that the data from the past decade or three shows the systems put in place are not working so well.

Many voices in current education policy debates are advocating an end to all forms of assessment-based incentives. These reactions are understandable given the evidence gathered over the last two decades or more, but we do not yet know how educators would respond to well-designed incentive schemes that incorporate the theoretical insights of Holmstrom and others, as well as the empirical insights produced by decades of research on the use of incentive systems in schools, government agencies, and businesses. We do know that, if policymakers continue to ignore these insights, those who oppose all forms of assessment-based incentives will continue to gather evidence that lends support to their cause.

It's not rocket surgery. If policymakers continue to design incentive systems based on measuring the wrong things badly, on measuring based on tasks that are not aligned withy the results we actually want, then these systems will continue to stink.