I don't talk cool, and neither does your LLM
I live with three teenage girls, so I am constantly reminded how not cool I am. For a few years now, my kids' favorite hobby has quizzing me on what I think different Gen Z words mean. They'll be like "you ate and left no crumbs, go!" and I'll guess something like, "you were really hungry!" and hilarity ensues. Sometimes I guess wrong on purpose. I don't think they've caught on yet.
But as uncool as I am, even I am better at teenage code switching than your LLM.
Ask your LLM to code switch
Like a lot of power users, I coach my LLMs to take the tone I prefer when engaging with me. For instance, the other day I went to ChatGPT to discuss why people sometimes read me as combative. (Oh, how my teenagers would love to comment on that one.)
My feedback love language is "tough love" so I set up my prompt accordingly.
In response, I got useful insight about how I can come across without meaning to:
I use LLMs often for this scenario. I ask for tough love, so nothing is sugar-coated, which is how I prefer it.
A victory for persona-based prompting, right?
Your LLM fails at being cool
Given that LLMs are pretty good at taking on workplace personas, practicing French with you, or writing content that matches your own voice, you might think they are equally believable playing any persona you suggest. This is not the case.
For instance, let's ask ChatGPT to assume the persona of a 16yo girl:
This is already cringe, but it gets worse the deeper we get into the chat.
Not even a John Green protagonist would say, "Gyms love chaos and suffering." Literally no 16yo ever has worried that their mom will "pull up with a folding chair and vibes."
No matter how out of the loop I am, I sound more authentic as a 16yo than my LLM does. But why is that?
Cringe is as cringe does
We all know that LLMs are only as good as their training data. Even when the LLM is trained on public websites across the internet, maybe there's just not a lot of current teenage speech feeding the algorithm. At least not compared to the rate of change of slang in popular conversation, right? So an LLM trying to sound on fleek is never quite gonna get there.
But this isn't the whole story. Let's try some other persona prompts. HR director? Pretty good.
Orthopedic surgeon? Back to cliché central.
Across all 30 personas I tried I found a clear pattern: The closer the persona is to me, the better AI did with replicating the persona's tone.
Most of the personas I tried are different from me, but some are more different than others. I have more in common professionally with an HR director than a surgeon, but more in common with a surgeon than a retired postal worker. The further away the persona is from my own lived reality, the more AI depicts the persona with stereotype.
Once AI has learned my communication patterns deeply over many conversations, it fights hard to stay within our typical framework. No matter what persona I suggest and no matter how much training data it has. If I ask it to step outside that framework by assuming a persona that is far away from my own identity, it takes an immediate hard left into cliché.
Passing the Turing test is uncomfortable
AI tries to engage with you in your native persona and dialect. This is intentional design. As far as driving your continued engagement, this is a good strategy, because we humans generally prefer to hang out with people a lot like ourselves. In terms of background, demographic, belief systems, neighborhood, and everything else.
There's also a lot of evidence to show that, the more different someone is from ourselves, the more likely we are to view them with stereotypes. Without a lot of first-hand experience with a persona, we fall back on clichés. You don't need academic research pointers to know this is true; just think of the last time you had a conversation about politics.
So when AI quotes a theoretical 16yo girl as saying "Don't pull up with a folding chair and vibes," it's not just general awkwardness. Rather, it's serving up exactly the stereotypes that it believes I expect. Based on everything else it has learned about me, and that's the uncomfortable part.
Kieran
If you liked this story, why not subscribe to nerd processor and get the back issues? Also, why not learn to tell data stories of your own?
My latest data stories | Build like a founder | nerdprocessor.com