A Dangerous Sea
During the pandemic and the two years that followed, our daughter Elsa spent her free time writing fantasy novels. While other people were making sourdough and working the Netflix back catalogue, she dashed out A Dangerous Sea and two sequels whose names I forget.
I'll never forget our family dinner conversation one night shortly after ChatGPT launched. Our kids had grown up seeing us build an AI company, so they were no strangers to the concept. Our younger two kids were making interesting observations about ChatGPT, Textio, and Grammarly. But all of a sudden, Elsa, 12 at the time, burst into tears.
"Someone could use ChatGPT to write books, but they will never be as good as a real novel. It is unfair and insulting to novelists!"
The tears continued and we had to change the subject. So I was struck when, just a couple months later, the same kid was happily using AI to generate illustrations for her latest book.
When I asked her about the discrepancy, she shrugged. "I don't know how to draw," she explained.
Expertise by any other name
I've told the above story many times, including last week when I keynoted the annual meeting of the American Psychological Association. I think about it constantly. Because when it comes to AI adoption, I see Elsa's story everywhere.
In my tech circles alone:
Reflecting on my own AI use, the pattern holds up. I use AI to analyze my kid's volleyball clips and my knee x-rays, since I have no personal skill in these areas. But in areas where I am confident in my expertise, I am less likely to turn to AI for help; I use AI to help with the ops side of nerd processor, but not the content.
The things we delegate
Given all of the above, I formed a hypothesis: People are more likely to use AI in areas where they are novices. They are less likely to trust AI in areas where they have expertise.
To explore this, I gave 40 people the same AI-written memo. I gave them two tasks:
- I asked them to highlight the single sentence within the memo they thought most needed to change
- After that, I gave them 10 minutes to edit the memo with the goal of making it as strong as possible, with explicit permission to rewrite, add, or delete any content
The AI-written memo was a brief, unedited layoff announcement for a fictitious company, which you can see below.
The 40 people represent five different personas:
- Executives
- HR
- Legal
- Comms
- Entry-level employees in roles other than the above
I wanted to see whether people filtered their response to the memo through the lens of their own expertise.
Spoiler alert: They did.
Layoffs in the eye of the beholder
There was remarkable consensus by persona in picking the sentence most in need of editing. Six of the execs and six of the employees picked the same sentence, and all eight HR, legal, and comms participants did.
You can see which sentence each persona selected below.
Each persona picked a sentence that is aligned with their unique ability to critique and improve, based on the perspective they have from their own role.
The pattern is even more striking when you look the rewrites people did during their 10 minutes of editing. Within each persona, the pattern of changes is remarkably similar:
- Executives: Add the business rationale (why now) and dial back promissory language
- HR: Add concrete detail on severance and support
- Legal: Soften legal and promissory claims; hedge the AI causation
- Comms: Reorder the narrative; cut phrases that become bad headlines
- Employees: More empathy and accountability; less jargon
Furthermore, people generally left content outside their specific focus area intact. In other words, each persona made changes aligned with their expertise, and trusted AI on the rest.
This really needs a longer paper
This is the first data story I've done in a while where I feel like I need the 20-page version to share all the insight I found, walk through the methodology, show the detailed rewrites, and explore why the data looks the way it does. I haven't written that paper yet, but should I? Since I presented this at APA, maybe I will.
For now, I'll just note that Elsa was on to something back in 2022. Most of us are least likely to delegate to AI in places where we're experts, and we trust it everywhere else.
Kieran
If you liked this story, why not subscribe to nerd processor and get the back issues? If you want to learn to tell data stories of your own, check out my full playbook!
My latest data stories | Build like a founder | nerdprocessor.com