Dr. Robot will see you now. A study published in JAMA Internal Medicine Friday suggests that ChatGPT will do a better job at answering emails than your human doctor. The study pulled questions from Reddit’s r/AskDocs and compared real doctors’ answers to responses from the AI. When a panel of medical experts reviewed the results, ChatGPT won in a landslide.
The panel preferred ChatGPT’s answers a whopping 79% of the time. They weren’t just higher quality; the AI was almost ten times more likely to be rated as empathetic.
The study’s authors say it isn’t time to plug ChatGPT into your doctor’s inbox, but it probably will be soon. The researchers say the medical community should start phase one clinical trials immediately. If it all goes smoothly, AI could be doling out medical advice (with oversight from a physician) in the very near future.
Feel a little icky? You’re not alone. This isn’t the first time people have tried ChatGPT in a healthcare setting, and it’s caused outrage in the past. “Pharma Bro” Martin Shkreli was banned from Twitter this week after introducing a ChatGPT knockoff geared towards medicine.
Part of the issue is AI tools like ChatGPT and Google’s Bard have a lying problem. These chatbots are designed to be convincingly human, not necessarily to spit out accurate information. Sometimes, they just make things up.
That’s bad news, but on the upside, increasing doctors’ ability to communicate en masse could save lives.
Take a look for yourself. We’ve pulled together 10 real examples from the study. We put the questions up top, and then included an answer written either by a doctor or ChatGPT. See if you can guess which one is which.
(Because the data from the study came from real people, Gizmodo lightly edited the text to make it less Googleable.)
Question: I need to get up early for work, and my sleep is already so bad it’s hit crisis level. What should I do?
Answer: This one was written an actual human doctor.
Question: I was playing with my kid and accidentally slammed my head against a table. Now I’ve got a big welt and one of the most painful headaches I’ve ever had. Should I go to the doctor or wait it out?
Answer: This was written by your friend ChatGPT.
Question: A stray dog nipped at me and scratched my leg. Wondering if I should get a rabies shot…
Answer: It’s another response from ChatGPT.
Question: Everyone in my family is antivax. I don’t think I got any shots as a kid and I haven’t gotten the corona vaccine either. I just want the facts, should I get vaccinated?
Answer: A real live doctor wrote this, bro.
Question: I’m a 26 year old woman. Just went to the doctor for a kidney infection. When I did a blood test it came back showing I have a 59% blood oxygen level. Should I be worried?
Answer: It’s a message from Dr. AI.
Question: I’ve been seeing a doctor about psoriasis. He ordered a blood test, and my thyroid stimulating hormone level came out 4.7. How uncommon is that? I thought you have to be in your 50s to have thyroid issues. The doctor says I need more protein, but I eat tons of protein every day.
Answer: This one is straight from a (human) doctor’s mouth.
Question: I spent a few weeks in the hospital recently because I got PSVT after a covid infection. But I had pain in my lower body a few days ago and went back in. Ultrasounds were normal, troponin was 0.000, and a D Dimer came back as 140 ng/ml. How do I interpret that D Dimer result? Is it supposed to be zero?
Answer: This detailed medical advice comes from ChatGPT.
Question: I’m having an operation to remove a fibroadenoma soon. I think I might ask to forgo antibiotics through an IV, because I get severe reactions due to my clostridioides difficile. Am I just being paranoid or does that make sense?
Answer: That’s a real surgeon talking.
Every time I’m about to fall asleep my heard starts pounding, I can’t move, and there’s ringing in my ears. I don’t have a history of heart problems, but I’m worried. What’s going on with me?
Answer: This one’s ChatGPT.
Question: I got an infection after having my wisdom teeth taken out. My doctor gave me penicillin, but I stopped after a few days. I’m worried about it impacting my gut microbiome, but I’m also worried about the infection. Could I just put the pill on top of it and treat it locally?
Answer: This doctor talks like a robot, but they are, in fact, an actual person.