Do Rising Charter Tides Lift All Boats?

So lately we've been getting the charter-pushing return of the notion that the rising tide lifts all boats. Here's Fordham Institute's head honcho Mike Petrilli at The Hill, throwing in a side of "follow the science" because Petrilli is great (and I say this with all sincerity) at working the angles. 

The "science" that he's referring to is a 2019 study-ish thing that Fordham put out (with Walton financing), authored by David Griffith, entitled Rising Tide: Charter School Market Share and Student Achievement. The idea he sets out to prove is that when charters open up, the resulting competition causes all students in all schools--both public and charter--to do better. The rising tide lifts all boats.

Griffith's background is a BA in Politics and Philosophy; later on he went back to school for a Master's in Public Policy. He worked as a political aid for a couple of years, worked briefly as a "research assistant" at two different outfits, did some TA work after grad school, taught for one whole year at a DC charter prep school, then landed a job at Fordham as a Research and Policy Associate. Make of that what you will.

The "study" depends, as always, upon Big Standardized Test data. Its finding included a claim that "high charter market share is associated with" big gains for Black and Hispanic students, but no real gains for white or rural students.

I could pick apart this piece of charter marketing for you, but as it turns out, that job was already done at the time, by the folks at the National Education Policy Center. The NEPC review was conducted by Yongmei Ni; she's an actual professor, the department chair of education leadership and policy at the University of Utah, who has produced a whole bunch of actual peer-reviewed studies. 

Her review of Griffith's "study" can be found here. Here's the short form of her findings:

One should interpret the findings and conclusions with extreme caution because of major issues surrounding the data and methods, including the measure of charter market share, the sample selection criteria, and the overreliance on results based on a small number of districts, especially the ones with over 95th percentile of charter market share. Overall, the findings have little use to policymakers because of these issues with data and methods and because the report does not probe beneath the surface.

The problems with Griffith's work are many. 

The rationale for the conclusions are the result of treating correlation as causation. The positive results (all student achievement rising across the district) only appears in certain samples, and there is no evidence offered that the effect is caused by charter school presence. 

The research literature cited in the report is carefully cherry picked. The market share measurement is inaccurate. The data is inaccurate. And the selection of samples is arbitrary and "very puzzling." And there's some stuff about cubic spline regression that can be a little confusing for laypeople, but I can fully understand the point that ultimately, Griffith bases his findings on data from about six or seven districts.

So a tiny, hand-picked set of samples gets him the results he wants, if you squint and don't examine the data too thoroughly, and if you don't want to talk about how such a thing could happen.

Yongmei Ni finds that the study is not useful for guidance of policy and practice, but apparently we're going to do just that by trotting it out again to claim that the rising charter tide lifts all education boats. It may not matter, as GOP choice fans seem to have deserted charters in favor of voucher programs, but here's your reminder that following the science will not lead you to any boat-lifting charter school revelations.