Five Hard Truths About The Big Standardized Test

Christopher Tienken (Seton Hall) has done some especially valuable research work with the Big Standardized Test, that state-level test that supposedly tells us how well students are learning and teachers are teaching. 

In one of my favorites, from way back in 2016, Tienken and his team showed that with just three pieces of demographic data-- percentage of families with income over $200K, percentage of people in poverty, and percentage of people with bachelors degree-- you can pretty accurately predict what the BS Test results for a district are going to be. If you want solid evidence that BS Tests primarily measure socio-economic factors rather than educational ones, Tienken is your guy.

So this piece, which dropped quietly in January, is well wort the look. Tienken has put together an invaluable and brief listicle-- Five Things Educators Should Know About Standardized Tests. I could have just put this on the weekly Sunday digest of Good Things To Read, but 1) it requires a couple of clicks to get there and B) I really, really think you should see this. Also C) it makes me want to add my two cents.

The five hard truths about the BS Test:

1) State tests are not diagnostic

Since Day One of the rise of the BS Test, critics have pointed out repeatedly that a single standardized test cannot be used for a dozen different purposes. A test that is used to measure achievement is not useful for diagnosing student needs. Tienken can explain this in more professional terms, but for laypeople, there are many analogies. You can't measure water temperature or volume with a yardstick. A tool that ranks students according to height does not tell you how tall any given student actually is. If you want a test to diagnose what students need to plug holes in their understanding, test experts can tell you how to design it, and the BS Tests do not meet those design specs.

2) State tests are predictable.

See above. Tienken has repeated and repeated versions of the earlier research in state after state. The fact that test results can be predicted by using demographic factors strongly suggests that, at a minimum, we are spending way too much in time and resources to get information we could easily elsewhere. Also, maybe that information isn't really telling us what we were promised it would tell us.

3) State test results are influenced by family income and background knowledge.

Teachers assess how well students learn and how well they taught by aligning tests to the actual lessons that preceded them. This is not exactly a radical notion. State tests don't do that, and so they favor students who have a bigger background of general knowledge, vocabulary, and reading.

State tests and other standardized assessments have included questions that uses passages, contexts, or situations based on a famous violinist, visits to a state park, pioneer life in the 1800s, ecology and environmental topics, life on the farm, space exploration, travel and vacations, contemporary suburban life, roller coasters, life in Japan, and other contexts and topics that require students to have varied life experiences and background knowledge to successfully navigate and understand the passages and contexts to answer the question.

4) Standardized tests disadvantage English Language Learners

It's not just trying to navigate a second, new, language, but the use of idiom, slang, and "white middle class situations." 

Look at 3 and 4 together this way-- imagine I give you a series of questions about 11th century slavic language development or Central African culture in the 500s, and then declared that the results of those questions showed your reading comprehension skills.

5) Standardized tests disadvantage students with individualized education programs.

An IEP is supposed to mean that you get an education crafted to meet your particular strengths and weaknesses. A one-version-for-everyone standardized test does not do that. As Tienken puts it, "Standardized testing for a student with an IEP makes as much sense as having a left-handed person create a writing sample in cursive with his right hand and then making a determination about the quality and skill of his handwriting and his readiness for college and careers based on that sample."

What standardized tests promise us is a frictionless measure of student skill, achievement, smartitude, whatever. It's supposed to be like giving a runner a clear, flat open track so that we get a "pure" measure of the runners true, best speed. 

But what Tienken and others repeatedly remind us is that the standardized test track is not level, not flat, not smooth, and is littered with all manner of obstacles, so what we end up measuring is not the runners speed, but their ability to navigate that particular set of obstacles. Instead of a frictionless measure, we get a measure of how well students manage the friction itself. That makes them lousy tools for the many purposes for which they've been sold.