Five Great Questions For "Evidence-Based" Programs

At EdWeek, Larry Ferlazzo ran a guest post by Rene Grimes that focused on the "science of math" and the programs that are supposed to be based on it. Between the EdWeek part and the "math" part, some folks may have missed the really good set of questions that Grimes proposes at the end of the piece:

*What question did the original research actually investigate?
*Which students, settings, and outcomes were studied?
*Does the evidence directly support this recommendation, or is the recommendation partly an interpretation?
*What limitations or unanswered questions should shape its use?
*What evidence will show whether this approach is working for our students?

What a great batch of questions.

Imagine, for instance, if anyone backing Common Core had tried to answer these questions. Blanks all around.

What question did the original research actually investigate?

So much education "research" asks only one question-- if we do this thing, will students get a higher score on the Big Standardized Test this year? Lots of folks really want to use that as a proxy for asking questions like "Will students have a higher level of reading comprehension" or "Will students be able to have an increased ability to solve complex problems." But those aren't the same question at all. Sloppy and inexact questions lead to sloppy conclusions.

Which students, settings, and outcomes were studied?

I remember the days when it seemed like every time you peeled back a piece of education research that said "this is how students learn" you found a "study" performed on a dozen college sophomores at a private university. This has always been a problem for education because educators are correctly reluctant to use students as guinea pigs. 

Some private and charter schools make claims about their effectiveness, but these are once again bits of research done on a very select group of students. That research isn't nothing, but it doesn't apply to a general student population.

Does the evidence support this sales pitch recommendation, or is the recommendation partly an interpretation?    

Boy is this one an issue. Every single piece of evidence based on Big Standardized Test scores involves interpreting those test scores to mean all sorts of things that they don't necessarily mean.

What limitations or unanswered questions should shape its use?

It is always a red flag when neither limitations nor unanswered questions are even mentioned. One of the Common Core red flags was that there was no mechanism at all in place for talking about limitations or questions arising from classroom deployment. 

There is no bit of legitimate science or evidence-gathering that declares "This question is now definitively solved and we don't need to look at it or question it ever again." And yet how rare is it that a new program is launched in a school and the Powers That Be say, "Now, here are some issues we want you to keep an eye on, and some questions that we are still looking to answer." In education, it's far more likely that every great new program is meant to be the end of the conversation, and uppity teachers who want to ask questions get a big round of administrative stinkeye.

What evidence will show that this is working for our students?

Again, the all-too-common default is that evidence will not be collected because it might not provide the evidence the Powers That Be want to find. No administrator wants to hear, "We've been trying your new program for a year and we have no evidence that it is working for our students." Should any teacher make that comment to admin, or any educator make that comment to a policy maker who mandated the Great New Thing, and they will get that old familiar refrain-- "You're implementing it wrong."

I love these five questions, and I hope more educators will use them more often-- including for every damned AI doodad that comes down the pike.  

                                                                

Does AI Tutoring Help?

There's a whole wing of the edu-wonk world pre-occupied with "high-impact tutoring," whatever that is presumed to be. The field serves as a successor to the notion of "accelerated learning," the idea that learning gaps could be closed by teaching the students who were behind by teaching them more, faster, than the other students-- an idea that stunk for a variety of reasons.

"High-impact tutoring" is also a successor for "two-sigma tutoring," another idea that rested on a single very small, narrow, and largely debunked study. 

What all of these ideas have in common is that they are super-appealing to techbros anxious to sell some version of AI tutoring. Lauraine Langreo over at EdWeek took a look at some studies about the various ways AI can be deployed for tutoring, and she opens with what has been the guiding principle for all of these operations. 

High-impact tutoring has been heralded as an evidenced-backed solution for addressing the significant academic gaps that schools are facing. But scaling and sustaining these programs are challenging for schools because they require a lot of human tutors, time, and money.

"Has been heralded" is doing a lot of work here, but the idea that it takes a lot of humans spending a lot of time (and wanting to be paid for it) is spot on. That crucial fact about tutoring smells a lot like money to AI companies. "However," as she notes, "research on the effectiveness of AI tutoring models is limited and still emerging."

Well, that's kind. Sal Khan has taken a good hard look at his own products and discovered that-- surprise--when you plunk a young human in front of a screen and say, "Go ahead and use this program to teach yourself math," by far the most common response is "I wonder if I can get regular Youtube on this thing." Students simply don't use it.

Langreo is looking at a "brief" from Stanford that considers how AI might be used.

SCALE, part of Stanford's Accelerator for Learning (yes, that again), has a brief out that looks at the ways AI can be used, and well...tutoring by a human has a "robust evidence base." After that, every evidence base is "emergent," a research term that means roughly "there is no actual evidence of this in sight, but we kind of smell the possible concept of a piece of the beginning of a sprig of evidence just about to appear, but it's not here yet."

SCALE likes a human tutor with AI supporting the tutor behind the scenes, which they call "potentially as effective, or more effective" as an actual unaided human. This would, of course, depend on the human. But if Human plus AI gets you the same results as Human alone, that suggests at the very least that the extra cost of AI is a waste of money. If the problem your leaders are trying to solve is "How do we incorporate AI into our program," maybe this looks not so bad, but if the problem you're trying to solve is "How do we best tutor these young humans," then adding AI seems like a non-starter (even before we start considering, as SCALE does not, the various negatives associated with AI.

"AI tutoring with human support," which has been the model of computer-powered schools, from Summit to Altschool to Alpha School-- the emergent evident here suggests that implementation matters, because it's only a shade removed from AI-only tutoring, which SCALE declares has "unknown effectiveness." Sad news for everyone who thought they could just swap out an actual human tutor for some AI program and get "high impact" results.

Look, we know the most effective means of tutoring-- one teacher, one or two young humans, repeated a few times a week. It's not a mystery. But there are two longstanding responses to this. One is the eternal education response from leaders and taxpayers: "That sounds nice and I can see it would make education better, but I don't want to pay for it." The other is corporations looking at it and thinking, "I bet we could make money pretending to fix this problem."

But if you want to get some actual tutoring done, you are going to have to hire a knowledgeable human being to do it, not a bot. 

One More Reason for States to Avoid Federal Vouchers

The Trump administration has ratcheted up its attack on discrimination in schools-- by which they mean, of course, discrimination against white guys.

They have hung their interpretation on Students for Fair Admission v. Harvard, arguing that it means that there can be no programs or benefits that white guys can't have. This has been the rationale for a widespread attack on all manner of college and university DEI wokitude.

Now they would like to come after high schools. Here's the opening paragraph from the IRS press release:

The Department of the Treasury and the Internal Revenue Service (IRS) have issued proposed regulations to end federal tax-exempt status for private schools that engage in racial discrimination, delivering on President Trump's Executive Orders ending discrimination and restoring merit-based opportunity.

But they are also coming after elite public high schools, as covered by Liam Knox and Erin Hudson for Bloomberg. The threat here is to come after admissions practices that discriminate based on race (i.e. preferences to anyone but white guys). 

The threatened punishment is to cut off federal funding if the school is found to be violating the administration's funhouse mirror of civil rights law. 

But there's a problem with that. As Knox and Hudson point out:

But public high schools receive far less federal money than most colleges, giving the administration less leverage over their admissions practices without action from the courts. Bronx Science, for instance, received around $44,000 from the federal government last year, and Stuyvesant got a little more than $160,000 — less than 1% of the budget at both schools.

To really get leverage over public schools, the feds need strings to pull. Specifically, purse strings. For "regular" schools, Title I and IDEA provide an opportunity (though not if they successfully convert them all to state block grants) to control local schools, particularly those who serve many non-wealthy or special needs students. 

So federal school vouchers provide another opportunity for the feds to get tighter control over local schools, including those that aren't so dependent on Title I or IDEA funds. Particularly if they adopt some of the ideas of Marguerite Rozas and use those voucher funds to finance important programs. Imagine if Bronx Science was used to taking in a couple million in federal voucher funds.

And for the MAGA crowd that thinks this would be just swell, I ask you to imagine what the power of that program might do in the hands of President Hilary Clinton or President Zohran Mamdani. You didn't like how the feds used federal purse strings to push Common Core (neither did I)-- do you really want to see that movie again?

Federal vouchers represent, among their many shortcomings, a real chance for the federal government to gain more leverage for controlling local school districts. It's a bad idea.

ICYMI: Oh, September Edition (9/6)

I can always see the big May crunch coming from a mile away, but September usually just sneaks up on me and suddenly--BAM-- I am up to my ears in One Damn Thing After Another. A show. A presentation at a writing festival. The Board of Directors' cross-country season. A friend I haven't seen in roughly fifty years coming to town. The ramp up to the biggest festival in my community. And this year, a conference that I usually never miss that I will have to miss (the Network for Public Education gathering in Texas) that includes a lot of great stuff and some good people I usually only get to see once a year. Also, the lawn needs mowing, but it has rained a gazillion days in a row. Retirement was supposed to be quieter than this.

But there is still reading to do. Anyway, here we go.

Parents Demanding Tech Opt Outs Take Their Fight to Federal Court

Lawsuits are after tech opt-outs for parents. I'll be curious to see where this one lands.

Butts on Fire

Phillip Maciak at City Chicken has been watching the robots crash and burn at the robot olympics, and it has made him feel better about his AI-in-school stance.

‘Kids literally disappeared’: Minnesota schools lost at least 1,700 students following Operation Metro Surge

Becky Dernbach and Cynthia Tu of Sahan Journal take a look at just how badly the ICE attacks disrupted education in Minnesota.

Books Can Help Teens Navigate Tough Topics in Ways Their Parents Can’t

Rebecca Stafford in People, of all places, explains why even books about Naughty Topics are important for children.

I Spent Three Days Inside Ryan Walters' Fake National Teachers Union

You may recall that Oklahoma's education dudebro-in-chief left the state for a gig leading the Teacher Freedom Alliance, a project of the union-busting Freedom Foundation. They held a confab this summer, and teacher Micah Cash was there, skeptical but open to listening. He has written a long deep dive into the event for The Pickup, and it is every second it takes to read it. 

The Nation’s Public Schools Endure Threatened and Turbulent Federal Funding

Jan Resseger explains some of the many ways that the Trump administration is screwing with funding streams, and how they may cause trouble for schools.

The Five Questions You Should Ask Before Believing the ‘Science of Math’

Larry Ferlazzo offers a guest post from Rene Grimes, and even if you just skip to the five questions at the end, it's worth burning a free visit to EdWeek.

Dan Patrick's book ban isn't dumb. It's worse than that. 

Loren Steffy points out the dumb (including Patrick's call to ban a work on the new Texas mandatory reading list). Houston Chronicle.

Private school tuitions seeing huge jump across Florida with voucher expansion

Surprise, surprise. Vouchers make windfall revenue for private schools. In other news, sun rises in West tomorrow.

Decades of Poor Transparency and Accountability in the Charter School Sector

Shawgi Tell looks at the many ways that charter schools have proven untrustworthy.

Unsubstantiated Things People Say about Public Education

Nancy Flanagan takes on a baloney-filled article written a politician.

Move Slow, Remake Things

Audrey Watters on what meaningful change looks like.

A Child’s Citizenship Does Not Depend on a Parent’s Paperwork

The administration's latest attempt to chase Black and Brown children out of the country. Bruce Lesley explains.

More than 1 in 10 Ohio Charter Schools suspected of “testing anomalies”

Ohio has a problem with "testing anomalies," and the problem is turning up in a high proportion of charter schools. Stephen Dyer breaks down some numbers.

No One Was Blaming Parents

Sue Kingery Woltanski published column from the Orlando Sentinel, entering a debate about where the money goes in Florida's multi-tiered education system.

Report: Public Libraries Face ‘Existential Threat,’ Putting Books, Information and People at Risk

A new report from PEN America details how the attack on books has extended to public libraries.

At Forbes.com, I look at a new report that suggests using AI to edit writing is a bad idea.

Here's your reflection on the end of summer.

    

Subscribe! It's free and always will be. 

Should Schools Out LGBTQ Students

Donalt Trump has been spewing the baloney sex change operations in schools for years now (here he is repeating the falsehood in 2024), so lying about it a couple of weeks ago (in front of actual young humans, and aided in the lying by Linda McMahon) is par for the course (particularly for a guy who beats par on the course by lying ). Meanwhile, Attorney General Todd Blanche has declared a "pandemic" of gender transitions in schools.

The notion that schools where there aren't enough classrooms or books or staff, but somehow the reading teachers is taking time out of her class schedule to perform sex change operation in a fully functional OR is so stunningly absurd that it defies satire. Of all the things that never happened, secret sex change operations in schools didn't happen the most.

However, there's a real issue behind this absurdity, and whether Dear Leader has distorted it by passing his partial understanding through the funhouse mirrors of his brain or whether he's deliberately distorting the case, his secret school ORs run the risk of distracting people from a real issue being tossed into the courts.

Out in California, brought by a couple of teacher-parents, Mirabelli v. Bonta made it to the Supremes, and another in New Jersey, brought by Moms for Liberty was just filed. Mirabelli is probably what Trump was referring to. 

These cases get complicated, but they revolve around one basic question-- do public schools have an obligation to out LGBTQ students to their parents, or do they have an obligation NOT to do so? 

For the Parental Rights crowd, it's a simple question. They have should have complete control over all aspects of their children's lives at all times. The view of the extreme members of this crowd is that parents have rights and children have none. When they say, "My child is not the property of the government," the unspoken rest of the sentence is "My child is my property" and not, unfortunately, "My child is a living, breathing human being who does not belong to anybody."  

Cases like the New Jersey case invoke the parental right "to direct their children's religious and moral upbringing." 

I wish they would also invoke their parental responsibilities or their children's rights, but I really do understand their point. I would certainly prefer, for instance, that my children's teachers not deliver instruction on how LGBTQ persons are delusional or mentally ill. I am absolutely responsible for the upbringing in all its aspects, and I really don't want the school working against me.

If my children do encounter ideas that run counter to what I want for them, the major tool their mother and I have is an open line of communication. In fact, as they get older, that's pretty much the only real tool I have. 

If your child is using a new name at school and you don't know about it, there are two parties that are not telling you. If your child has come out as LGBTQ and you don't know about it, the only reason is because they have chosen not to tell you.

If a child comes out to a teacher or other school staff, that adult has a responsibility to encourage the child to talk to a parent about it. But they also have a responsibility to listen to a child who says their home will become unsafe if they come out there. LGBTQ children are a disproportionate part of the homeless population, more likely than their straight peers to be thrown out of the house. To insist that a school must out an LGBTQ student no matter what is irresponsible. 

Both schools and parents have a responsibility to look after the health and welfare of the child, and a child's health and welfare is best served ideally by a strong relationship with their parents. It helps if the parents and schools communicate regularly and openly. At the heart of all this should be care and concern for the education, health and welfare of the child-- not absolute authoritarian power of the parent-- or the school-- over the child.

Of course, many policy makers don't care a bit about parental rights. Witness the states, like Texas, where if the parent wants the school to recognize their child's trans identity, the state says "too bad." Sometimes these battles aren't about parental rights at all-- they are about policy makers' desire to erase LGBTQ identities, a stance that puts the health, welfare, and education of the child absolutely last. 

The group for which I have the least patience or sympathy is those teachers who don't want to call students by their chosen name. Grow up, get over yourselves, and call the student whatever the student prefers to be called. Spare me your religious objections; if your religion is defined by the people you are called to reject and deny, the students you think Jesus wants you to tell, "Your identity is not welcome here," you are following a brand of christianism I do not recognize.

The one thing we are not called to do as teachers is to tell students who they are in defiance of who they say they are. 

As a parent, I often think of the words of  Russell Barkley, who argues that we do not engineer children, but serve as shepherds. We don't beat them into the shape we prefer, but guide and support and nudge them along, and if they don't grow up exactly as we ordered them to, that is not a sign that someone has surreptitiously out-engineered us. 

If your child feels unsafe talking to you about highly personal stuff, it will not aid your cause to make sure that they can't trust any adults at school, either. It will just ensure that your child will feel very isolated at a very vulnerable and scary time, which opens them up to the kinds of problematic influences that you were worried about in the first place. 

Parental rights are real and important, but when they are "protected" and extended at the expense of children's rights, that's bad for everyone. Children deserve acceptance and guidance, support and safety, autonomy and guardrails, and providing those things requires a tricky balancing act by all the adults in their lives. It's a balancing act that's best managed by centering on the rights of the child and not hamfisted lawsuits, policy declarations, and press conference bloviating.

FL: State Wants To Run Church Business

The Catholic Church and Florida Attorney General James Uthmeier are in the midst of an argument about who runs Catholic private schools.

Catholic private schools, says Uthmeier, should allow students to enroll without vaccination if they plead religious exemption. Uthmeier expressed his thoughts publicly, coupled with a demand that the Catholic church explain itself within seven days. And have waved threateningly in the direction of Catholic School eligibility for Florida's taxpayer-funded school vouchers. "Nice funding stream we gave you there. Be a shame if something happened to it." 

Michael Sheedy, the executive director of the Florida Conference of Catholic Bishops, responded on August 7. Here's the opening of that letter. See if you think Sheedy is intimidated:

As explained in detail below, the Florida Conference of Catholic Bishops (FCCB) and the Catholic bishops in Florida do not have to justify to your office their position on Catholic teaching and their reasons for the current vaccine policies. They will, however, do so out of respect for you and your office and with the understanding and expectation that your office will respect the Catholic bishops’ constitutional rights and religious freedom. 

While we appreciate the opportunity to present the rationale for school vaccine policies, we wish we could have met and discussed these concerns with your office and perhaps avoided all the public and legal posturing created by your correspondence requesting a response in seven days.   

There's irony all around here. Nobody has pushed harder for policies that funnel public dollars to private schools than has the Catholic Church. But they much prefer the model in which the state hands over the money and then leaves the private school alone. Many states have carefully written voucher laws to require the state to stay hands off, but Florida's HB 1, the 2023 voucher bill, somehow skipped that part. 

So now the Catholic Church finds itself in a completely predictable position; when you break down the wall between church and state, the church may get public money, but they also lose protection from the interference of zealous public officials who are more than happy to tell church folks how they should properly go about the free exercise of their religion (see also: every state that passed a law requiring schools to post the official state version of the Ten Commandments).

Folks on the right have also predicted this and railed against it. Government money means government interference along with a reduced ability to resist. Or, as I always say, when you mix religion and politics, you get politics. 

Or as Sheedy put it, "Government officials may no more determine the content of Catholic teaching than Church officials may determine the content of Florida law." Which is, again, ironic, because the church officials actually worked really hard to determine the content of Florida law when it comes to taxpayer education money. Three years ago, the Florida Conference of Catholic Bishops thought this law was awesome.

On the other hand, if Uthmeier rubbed three of his MAGAbro brain cells together for two seconds, he would realize this is not a fight he wants to win. There isn't anything anywhere in HB 1 on which to hang his demands, so he's left with "Private religious schools that accept vouchers must do what the state tells them to or else," and I'm thinking there a lot of folks on the conservative and pro-voucher side who aren't ready to support that. 

But Uthmeier is a certified real piece of work. He's been a US Department of Commerce senior counsel and senior advisor in the Trump administration, general counsel for Ron DeSantis, and the DeSantis chief of staff until appointed AG. He whipped up a new "Free State of Florida" seal, proposed Alligator Alcatraz, called surrogacy "modern day slavery," and went after a teenage transgender girl in a national women's Irish dance step competition. He chaired the committee that illegally used the Hope Foundation money for political purposes. And he drew a contempt citation from a US District Judge for ignoring her ruling to shut down an illegal Florida immigration law. Uthmeier stated "If being held in contempt is what it costs to... stand firmly behind President Trump's agenda on illegal immigration, so be it." 

Also, part of his argument against vaccines is that they were developed using fetal cell lines. Which is not true.

And yes, as you may already have guessed, this MAGAbro who wants to tell the Catholic bishops what they believe and how to act on it is, himself, a Catholic.

So, yeah. He's that guy. So it's not hard to imagine that his principle here is "I'm allowed to tell the church what to do, but people I disagree with are not." In the Free State of Florida, that may be enough to get by.

"Any insinuation that a government official can dictate to the Catholic Church (or any religion) how to apply her tenets and dogma, violates well-established constitutional religious freedom rights," writes Sheedy. He did not conclude "Now give us our free taxpayer money and shut up." 

This is just another version of the same fight that keeps breaking out right at the very spot where choicers busted a hole in the wall between church and state. There's nobody to root for here. We'll see if Uthmeier backs down or if the whole thing ends up in court. Grab your popcorn and stay tuned.

TX: Alpha School and Voucher Grift-- It's Synergy!

At last we are seeing some of the press that Alpha School actually deserves. Lexi Churchill, in a joint Texas Tribune and ProPublica piece, shines a spotlight on the Texas emperor's wardrobe. You should absolutely read the article, but I want to underline and expand on a couple of points.

First, the switcheroo. As it has in every other state (except, of course, Arizona), Alpha failed to get approval for its attempted charter school launch in Texas. That's Texas, where the state board of education crew is not known for its rigorous standards. This is not a surprise; Alpha School's educational bona fides are still limited to puffed up unproven baloney. As Churchill notes, several journalists have now tried to verify Alpha's claims of magical success with just two hours of screen time a day, and so far, evidence has proven elusive.

But to get into the Texas voucher program as a vendor, a school just needs approval from the state comptroller's office. For that approval, a school need only show that it operates in Texas, that it’s accredited by an approved organization, that it’s been in operation at least two years and that it administers annual standardized tests. Alignment with state education standards? Proven student success? Evidence-based program administered by qualified personnel? None of that is needed.

It's worth remembering that Alpha is by no means the only school playing this game of switching back and forth between charter and voucher status based on whichever category will let them keep the money flowing in with the least possible accountability. If your state has both charter schools and vouchers, I guarantee that there are schools picking whichever designation will keep their owners happiest.

And if you wonder what exactly is the aim of Alpha school and the husband and wife team of MacKenzie and Andrew Price, there's this reminder.

At least two state education boards also took issue with Alpha’s plan to keep Price and her husband on the charter’s oversight board while the school simultaneously paid their companies, which provide services like the two-hour learning platform. The Prices acknowledged the conflict of interest and repeatedly promised to correct it.

When the Price's made their half-baked amateur-hour bid to open an Unbound cyber charter in PA, there were multiple companies connected to the enterprise.

There's Unbound Academy, which is the charter arm of Alpha School itself. Both tout the 2 Hour Learning program, which is set up as separate business. The proposal in the PA Unbound application said “In support of its operations, Unbound Academy will collaborate with 2hr Learning, Inc. to deliver its adaptive learning platform, while Trilogy Enterprises will manage financial services, and Crossover Markets, Inc. will assist with recruiting qualified virtual educators." 

Trilogy Software is owned by Joe Liemandt, the billionaire techbro who backed all of this and is currently the principal of Alpha School; Andrew Price is the CFO. Crossover (an internet employment hub currently advertising a job as Director of Academics fore 2 Hour Learning at $400K) is supported by ESW, Joe Liemandt's personal investment company; their main client is Trilogy. Crossover has a proprietary tool called WorkSmart, which surveils employee work on their computer and was promised for the PA Unbound school. Crossover Markets and 2 Hour Learning share the same Austin address. It appears that Andrew Price was also linked to Crossover, but I find no clear current evidence.

The PA application also noted that the general and administrative service provider would be YYYYY.LLC, a company whose director, president, and principal are Andrew Price.

So Alpha School comes with a whole big web of companies that it "hires" to run the various aspects of the operation, and which in turn funnel all of that money back to the Prices. As Dan Meyer noted regarding that Arizona Unbound charter, "Ultimately, MacKenzie Price, Unbound Academy’s board president, will need to decide whether MacKenzie Price, founder of 2 Hour Learning, Inc, has set a fair price for those curriculum and resource materials. Arizona taxpayers await her decision."

Also worth noting-- Alpha School has ridden a wave of buzz mostly generate by MacKenzie Price's willingness to plug her operation to any media outlet that was willing to be impressed. And yet, according to Churchill

None of the Alpha leaders named in this story, nor the company’s press office, agreed to interviews, nor did they answer most of the newsrooms’ questions, including about their test scores. An Alpha spokesperson provided an emailed response only to questions about previous reporting by other news outlets.

 So, feeling a little shy suddenly.

But hey-- so much synergy. Synergy between Alpha's lazy charter school and Texas's lazy voucher rules. Synergy within the web of money-scrubbing corporations. Who cares about weaving a tangled web if it's a profitable one.