This is one more example of how a chatbot will NOT read your writing and provide useful insights into what you have created. I referenced this piece of mini-research before, but it deserves its own post (partly so I can find it easily when I want to reference it again).
Adam Kucharski is, according to his bio, a mathematician/epidemiologist. He wrote Proof: The Art and Science of Certainty, and his work has appeared in Wired, Guardian, and New Scientist, among others.
Back in May, he published an account of a simple experiment. It started with a simple question-- how good is Copilot at finding insights in a data file.
To test it out, I asked Copilot to look at differences in how people in US and UK expressed emotions in an Excel dataset that contained thousands of survey responses.
Copilot found all sorts of deep insights. The responses differ "mainly in tone, intensity, and wording style," the bot typed. And it went on to provide deeper analysis. "US responses were typically more direct and emotionally amplified, while UK responses were more restrained." or how about "UK responses use more poetic, metaphor-rich phrasing; US responses favor plain, action-oriented wording." And even "UK wording reflects understatement and nuance; US wording emphasizes clarity and emphasis."
Wow, that is some deep and nuanced analysis.
Except that the dataset wasn't real-- and it wasn't different. Kucharski explains his technique:
First, I’d created 2000 free-text responses and labelled them ‘UK’. Then I copied and pasted the exact same 2000 responses but labelled these ‘US’. Finally, I combined them to create a dataset of 4000 total responses, and jumbled them up.
So Copiloty looked at two identical data sets--not made by a human in any country-- and found all these nuanced differences between them. It's almost as if the chatbot is just making shit up.
So he tried it again:
This time, I got an LLM to simulate 200 statements about career aspirations. Then I duplicated the dataset five times, labelling each one ‘US’, ‘UK’, ‘France’, ‘Germany’, ‘Italy’.
Again, the bot found a variety of differences and nuanced distinctions between the five different countries. It even generated a chart that shows the likelihood of people in a particular country pursuing particular careers-- In France, they are twice as likely as the UK to pursue a career in the arts. US citizens are most likely to pursue a career in business, and way more likely to pursue one in medicine.
Again-- all this analysis is based on identical datasets generated by a bot that is not a citizen of any country.
Kucharski's post gets into some specifics and offers links to his datasets. Plus some updated follow-ups by other folks who took a stab at the same experiment. And it addresses a bit the question of "Well, if you used a different process..." You know who doesn't have knowledge of the best ways to trick the bot into doing the thing for real? The kind of folks who think the computer is smart and will help them with their writing.
Any time someone tries to tell you that a chatbot can be a useful tool for analyzing your writing, you should be thinking of this (and many other) actual demonstrations of how badly bots do at this task.
One of the most useful checks I've seen came from anonymous writer online who suggested that the best way to think of a prompt is that you're not asking "Do an X for me" but instead "What would an example of X look like?" The bot is not going to do an actual analysis-- it's just going to extrude an example of what an analysis would look like based one whatever stolen analyses it was trained on.
It is not reading your work. It is not analyzing your work. It is most definitely not your writing partner or editor.
No comments:
Post a Comment