Spotting Bad Science

This item comes via my old friend and Chicago nursing queen Deb Burdsall. Its original source is Andy Brunning at chemistry site Compound Interest, but it certainly brings to mind some of the "science" that floats around the education world.

So here is the rough guide to spotting bad science:

 

1) Sensationalized headlines are not always the fault of the researchers when their work is glommed up by the media, but when headlines like "Good teachers are as important as small class size" or "Calculus can make you rich?" are not a good sign.

2) Misinterpreting results. How many times have you followed up on an piece of research only to find it doesn't actually prove what the article says it proves.

3) Conflict of interests. As in, funding research specifically to prove that your pet theory is correct. Just google Gates Foundation.

4) Correlation and causation. This one is everywhere, but nowhere has it been more damaging than in all the policy decisions by the current administration deciding that since low standardized test scores and poverty go together, low test scores must cause poverty. And more research that concludes that teachers cause low test scores.

5) Speculative language. Again, we are living with a boatload of policies based on how we think things ought to work. The infamous Chetty study about future earnings is loaded with suppositions.

6) Sample size too small. Is this still a problem? I just remember looking up studies in college and discovering the "research" was performed on thirty college sophomores. 

7) Unrepresentative samples. Chalk this one up for every piece of "research" that proves the effectiveness of a charter school.

8) No control group used. A built-in limitation of education research. We can't really assign a group of tiny humans to have no education so we can see what difference a teacher makes.

9) No blind testing used. Also a limitation. I'm not sure I can even think of how to use blind testing of educational techniques. Blind teacher? Students wrapped in plain brown paper? We get a pass on this one, too.

10) Cherry-picked results. Well, yes. Easiest to do if you're doing charter research and you cherry-pick the test subjects to begin with.

11) Unreplicable results. Sort of like the way VAM scores never come out the same way twice. In fact, VAM fills the bill for most of these indicators of bad science.

12) Journals and Citations. My favorite thing about thinky tank "research" is how it provides nice citation pages filled with references to other papers from thinky tanks. Or this ACT report with footnotes from other ACT reports.

VAM is perhaps the leading source of junk science in the education field, but there are so many fine examples. Print out the handy graphic above and keep it nearby the next time your are perusing the latest in educational "research." 

Also, I am going to use this as an excuse to post this picture (one of my faves)