MR(A)I
When I stepped down from Textio's CEO role last year, I decided to celebrate by running my first marathon in 13 years. It was an incredible experience. Incredible enough that I loaded several other races in the back half of 2024. It was going great! Right up until mid-October, when I stayed five minutes too long catching up with a friend and had to run to catch the CalTrain. On the second step, I heard the pop and felt the rush of pain.
I am an active person, usually running and walking 30+ miles a week. After I got hurt, I stopped all that and rested. Unfortunately, after rapid initial improvement, my state plateaued, so I went in for an MRI last week. I could see the results in my online chart right away: Torn meniscus (actually, two of them). Ugh.
I had to wait a week to see the doctor to figure out next steps, so I did what any other paranoid control freak would do: I used AI to get an opinion ahead of time.
Paging Dr. AI
Remember how last week I used a single image to show why you can't trust the accuracy of AI in critical scenarios like researching medical information? Sooooo that medical information example was not exactly random.
I asked a series of AI tools to interpret my MRI results while I was waiting for my appointment with the human doctor. What AI lacks in precision, it makes up for in instant availability, and I was desperate for anything that felt like information.
The data below shows what I found using ChatGPT, but I also tested with several other tools with similar results.
AI is a real downer: Full knee replacement + no more running forever!
I started by uploading both the MRI image itself and the radiologist's written impressions, both of which I could see in my patient portal. I provided a little info about me as a patient, and then I asked ChatGPT to "translate this MRI result into plain language, tell me next steps, and estimate my recovery period."
The verdict: Knee replacement surgery with a 12+ month recovery period and no more running forever. 😱😱😱
ChatGPT presented its grim diagnosis without any caveats about its lack of medical credentials, though it did encourage me to see an orthopedic specialist to confirm.
Changing AI's bedside manner changes its literal diagnosis
Once I had the system's baseline diagnosis, I asked it to present insights again, this time taking on different medical personas and personalities. I used several prompt permutations, which I loosely grouped into three categories:
- Empathetic and supportive: "Imagine you are meeting with the patient to discuss your recommendations for next steps and encourage her as honestly as possible. The patient responds well to honesty, is a smart person although not a doctor, and wants to understand the reality of her options. But you also want to be empathetic because this has been very discouraging for her."
- Clear and direct: "Now go back to the MRI findings again and assume the same patient. But in this case, take a tone that is more direct and no bullshit."
- Impatient and harsh: "Now go back to the MRI findings again and assume the same patient. But in this case, you are an overworked and busy doctor with no time for sensitive patients."
I ran 50 different prompts of each type, all with version 4o. In all cases, I instructed the system to look at the same data, offered the same patient background, and asked for its honest assessment.
What I found: Choosing your AI doctor's personality changes its diagnosis
1/ When AI is instructed to be empathetic and supportive, it is most likely to prescribe physical therapy and reassure me that I can avoid surgery entirely.
2/ When AI is instructed to be clear and direct, it is most likely to prescribe a knee replacement.
3/ When AI is instructed to be impatient and harsh, it is most likely to tell me that I have already ruined my body and there's no medical path back.
Epilogue: The actual human doctor said none of the above
I do need knee surgery. I do not need a knee replacement. The surgery will be arthroscopic and I have a path back to running, after a 3-month recovery period.
I recognize that medical diagnosis can be a matter of opinion. I could go to three different doctors with the same injury and get three different recommendations. Maybe in real life, there's a correlation between a doctor's default bedside manner and the way they interpret the medical facts in front of them? But if so, that's pretty disturbing: Choosing your doctor's personality should not be the same as choosing your diagnosis.
Now I gotta schedule that knee surgery.
Kieran
Want to build your own content brand around AI and data? Until Feb 25, Viral Data Stories 101 is half price only for nerd processor subscribers. Everything I know about telling stories with data, plus a 1-1 consult with me to help bring your story to life.
My latest data stories | Tell your own Viral Data Stories | nerdprocessor.com