Audrey Watters bills herself as “an education writer, an independent scholar, a serial dropout, a rabble-rouser, and ed-tech's Cassandra.” Her Hack Education blog is required reading for anyone who cares about technology in education. Since founding the blog in 2010, she has provided a meaty, thoroughly researched and well-thought-out end-of-year assessment of ed tech. This year, she looked back over the entire span with “The 100 Worst Ed-Tech Debacles Of The Decade,” and it deserves a spot on every list of education end-of-decade lists.
So as we roll into the new year, let me suggest eight resolutions for journalists who cover the ed tech beat.
1) Ask what this person knows.
As you interview the latest hot new entrepreneur, probe his history. Does he have any experience or training in education, or has he spent his whole life in computer tech or investment banking? If he has no actual education in his background, a good follow up question might be, “What makes you think you have any idea what the heck you’re doing?” Note: two years in the classroom does not make someone a teaching expert.
2) Check the history of the idea.
Hardly a quarter goes by without the announcement of software that will make it possible to grade student essays. That promise has been made repeatedly, and even though some states are using versions of such software, there’s still no reason to believe that anyone has actually delivered the real thing. Les Perelman has made an entire side-career out of debunking essay grading software (including fooling computers with other computers). The vast number of ed tech “innovations” are not actually innovative at all. When confronted with the latest “new” idea, journalists should be studying up on the history of that idea and then, at a minimum, asking, “What is different about your iteration of this oft-attempted idea?”
3) Check the entrepreneur’s track record.
Ed policy is a field in which many somehow continue to fail upward. What is this innovator’s previous record? If his last three attempts at Changing The Face Of Education all failed, consider that a significant part of the current story. Also, when looking into the causes of his previous failure, you might want to talk to someone other than the innovator himself.
4) Cultivate a team of actual teachers.
Every single time a piece of ed tech tanks, every single time the Next Big Thing has to close up shop, in classrooms across the country there are teachers saying, “Well, duh. I could have told you that junk was doomed.” Every ed tech journalist (just plain education journalists, too) should have a group of actual classroom teachers that she trusts. When the Next Big Thing lands on her desk, she should get ahold of her teacher squad and say, “Could you take a look at this and tell me what you think?” Interviewing teachers who have been hand-picked by the ed tech company does not count.
5) Get an explanation, in plain English, of how the tech works.
A line-by-line look at proprietary programs is not necessary. But the tech company should be able to explain, in plain English, what the tech actually does. “Computer magic” is not a legitimate answer. “Sophisticated machine learning growth algorithms detect biometric cues which, through deep data large authentic engagement learning-centric analysis, provide flexible insights into individualized personal AI-selected metrics of the student’s systemic growth,” is not a legitimate answer. “We measure how long it takes the student to push the button because we think that correlates with how engaged she is,” is a legitimate answer. Follow up questions should include, “Is it even possible to actually do that” and “What reason is there to believe that the thing you’re actually measuring is a good proxy for the thing you want to measure?”
6) Remember that numbers aren’t automatically science, and science isn’t automatically correct.
The fact that the tech reduces something to a numerical value does not mean that it has actually measured anything worth measuring. Also, scientists once scientifically measured brain activity in dead salmon. Some science is bad science.
7) Correlation is not causation.
The marriage rate in Kentucky correlates with the number of people who drown after falling out of a boat, but the number of people who drown in a pool correlates to the number of films that Nicolas Cage appears in. Every time an ed tech pitch includes the claim that A correlates with B, go to this website (or buy the book) and remind yourself that ending Cage’s film career would not reduce drowning deaths in the U.S.
8) Be skeptical.
Talk to people who have knowledge and expertise, but no personal interest in the endeavor you’re writing about. Ask questions. In particular, ask questions such as “Can they really do that? How?” and “How would this benefit actual teachers and students in actual classrooms?” Journalists might even get extra rigorous and ask questions such as, “Should I believe these people?” Most of all remember that an ed tech declaration “This is what the future will be” is more about aggressively hopeful marketing than legitimate insightful prediction.
Ed tech will continue to be a field full of hustling entrepreneurs and hopeful investors, and that’s fine. But it would be helpful if ed tech journalism was a little less breathless gee whizzery wrapped around the uncritical echoes of PR spin, and a little more carefully analytical skepticism.
4) Cultivate a team of actual teachers.
Every single time a piece of ed tech tanks, every single time the Next Big Thing has to close up shop, in classrooms across the country there are teachers saying, “Well, duh. I could have told you that junk was doomed.” Every ed tech journalist (just plain education journalists, too) should have a group of actual classroom teachers that she trusts. When the Next Big Thing lands on her desk, she should get ahold of her teacher squad and say, “Could you take a look at this and tell me what you think?” Interviewing teachers who have been hand-picked by the ed tech company does not count.
5) Get an explanation, in plain English, of how the tech works.
A line-by-line look at proprietary programs is not necessary. But the tech company should be able to explain, in plain English, what the tech actually does. “Computer magic” is not a legitimate answer. “Sophisticated machine learning growth algorithms detect biometric cues which, through deep data large authentic engagement learning-centric analysis, provide flexible insights into individualized personal AI-selected metrics of the student’s systemic growth,” is not a legitimate answer. “We measure how long it takes the student to push the button because we think that correlates with how engaged she is,” is a legitimate answer. Follow up questions should include, “Is it even possible to actually do that” and “What reason is there to believe that the thing you’re actually measuring is a good proxy for the thing you want to measure?”
6) Remember that numbers aren’t automatically science, and science isn’t automatically correct.
The fact that the tech reduces something to a numerical value does not mean that it has actually measured anything worth measuring. Also, scientists once scientifically measured brain activity in dead salmon. Some science is bad science.
7) Correlation is not causation.
The marriage rate in Kentucky correlates with the number of people who drown after falling out of a boat, but the number of people who drown in a pool correlates to the number of films that Nicolas Cage appears in. Every time an ed tech pitch includes the claim that A correlates with B, go to this website (or buy the book) and remind yourself that ending Cage’s film career would not reduce drowning deaths in the U.S.
8) Be skeptical.
Talk to people who have knowledge and expertise, but no personal interest in the endeavor you’re writing about. Ask questions. In particular, ask questions such as “Can they really do that? How?” and “How would this benefit actual teachers and students in actual classrooms?” Journalists might even get extra rigorous and ask questions such as, “Should I believe these people?” Most of all remember that an ed tech declaration “This is what the future will be” is more about aggressively hopeful marketing than legitimate insightful prediction.
Ed tech will continue to be a field full of hustling entrepreneurs and hopeful investors, and that’s fine. But it would be helpful if ed tech journalism was a little less breathless gee whizzery wrapped around the uncritical echoes of PR spin, and a little more carefully analytical skepticism.
9) Do you have any teacher testimonials that your shiny new thing works well with an overcrowded class of high needs 7th graders?
ReplyDelete