If you needed any more reason to be wary of the massive upsuck of student data from the education system, The Weekly Wonk offers more reason to fear our Data Overlords.
"reduce caseloads in an increasingly black box manner."
But even if a system is well-designed,
it can be “garbage (data) in, discrimination out.” A transportation
agency may pledge to open public transit data to inspire the creation of
applications like “Next Bus,” which simplify how we plan trips and save
time. But poorer localities often lack the resources to produce or
share transit data, meaning some neighborhoods become dead zones—places
your smart phone won’t tell you to travel to or through, isolating these
areas into islands of poverty.
It's the linkages that can be the real killers. The criminal justice system now routinely collects DNA from arrested individuals. But your DNA doesn't just identify you like a fingerprint-- it links you to all your relatives. And then there's all the computer devices we use.
Homes come outfitted with appliances that sense our everyday activities,
“speak” to other appliances, and report information to a provider, like
an electric utility company. While it’s presumptuous to say that
retailers or utility companies are destined to abuse data, there’s a
chance that information could be sold down the data supply chain to
third parties with grand plans to market predatory products to
low-income populations or, worse yet, use data to shape rental terms or
housing opportunities. What it boils down to is a lack of meaningful
control over where information travels, which makes it more troublesome
to intervene if and when a problem arises in the future.
Gangadharan and Woolley do not address the data mining of education, but the implications aren't hard to imagine, particularly in a system that not only admits the possibility of data linkage, but embraces it. From way back in the infamous Marc Tucker "Dear Hillary" letter through the Bush/Obama cradle-to-career tracking program, our Data Overlords have said that a fully-integrated data trail is a Good Thing.
But as Gangadharan and Woolley's article suggests, what happens
-- if the data cloud determines that an infant has a genetic marker present in earlier generations who are linked to criminal behavior or recurrent disease or susceptibility to alcoholism?
-- if a corporation decides to use predatory marketing carefully aimed at "low information" customers who are identified by their elementary and high school academic records?
-- if an algorithm selects out groups who can be identified by genetic, family and school records and targets them for discriminatory practices?
-- if you apply for a job and the program sees that you used to do lots of googling for bong suppliers and that your grades and test scores dipped precipitously about the same time, thereby deciding you were a teenage pothead?
-- if a program puts together your genetic record, your googling for "HIV treatments" and your repeated trips to the guidance counselor in high school, and decides you are too high risk to be hired?
It's been just a couple years since Target famously announced to a father that his daughter was pregnant, and we are still only scratching the surface for all the many ways that data trails can be used to jump to conclusions about people and then discriminate against them on that basis. And that's just looking at the accidental discrimination.
Remember when we decided that it is illegal to ask someone they're race on all sorts of job and financial application paperwork? Well, now it will never be necessary again. Remember how juvenile court records are supposed to remain sealed so that a youthful mistake doesn't ruin the rest of someone's life? Well, that's pretty much a joke now.
We don't even know yet all the ways that this data monster can screw with peoples' lives, destroy any semblance of privacy, and make them victims of discrimination carried out by programs that can't even be questioned. Imagine what could happen if we fed that monster everything we know about a person's youth and childhood?