ChatGPT as a Doctor: The Danger of Misdiagnoses

What happened?

An American pastor, a resident of California, experienced chest pain and difficulty breathing. Instead of seeing a doctor or calling emergency services, he consulted ChatGPT, OpenAI's chatbot. The model, based on GPT-4, assured him that his symptoms were not serious and were likely anxiety or indigestion. Following that advice, the patient delayed medical care and continued with his daily activities until he collapsed hours later. He was rushed to a hospital, where doctors diagnosed him with a pulmonary embolism, a blockage of a pulmonary artery that, according to medical literature, has a mortality rate of up to 30% if not treated early. The pastor survived but suffered complications requiring prolonged hospitalization. He has now filed a lawsuit against OpenAI for negligence, alleging that the chatbot provided erroneous medical information that worsened his condition. The case, initially reported by 9to5Mac, highlights the dangers of delegating health decisions to artificial intelligence systems not designed for clinical purposes.

Why is this important?

This case is not an isolated incident. According to 9to5Mac, hundreds of millions of people consult ChatGPT every week for medical advice, despite the platform's disclaimers. The popularity of chatbots as a source of health information stems from their accessibility, immediacy, and tone of authority, which creates a false sense of trust. However, generative AI lacks the ability to make clinical diagnoses: it does not examine the patient, has no access to their medical history, cannot perform physical or laboratory tests, and is not trained to interpret symptoms with the precision of a professional. In fact, language models like ChatGPT are prone to hallucinations, i.e., generating false but convincing responses. In the medical field, this can have fatal consequences. The pastor's case is an extreme example, but recent studies show that chatbots often provide incomplete or dangerous information in health contexts. For instance, a 2024 Stanford University study found that ChatGPT correctly answered only 62% of standard medical questions, and in several cases recommended contraindicated treatments. The lack of specific regulation for AI in healthcare exacerbates the problem, as tech companies are not legally liable for damages caused by their products unless gross negligence is proven.

Consequences and context

The incident underscores the urgent need to regulate the use of artificial intelligence in critical areas like healthcare. OpenAI has already added visible disclaimers to ChatGPT, such as "I am not a doctor" or "Consult a professional," but the potential harm persists because many users ignore these warnings. Compared to other cases of technology misuse, such as medical misinformation on social media (e.g., promotion of dangerous home remedies on TikTok or Facebook), generative AI amplifies the risk in a unique way: its responses appear personalized, coherent, and reliable, leading users to overestimate their accuracy. Moreover, unlike a web search engine that displays multiple sources, ChatGPT offers a single response with an air of authority, reducing critical thinking. This case also recalls past incidents with other AI systems, such as in 2023 when an insurer's health chatbot recommended incorrect treatments to patients with chronic diseases. Legally, the lawsuit against OpenAI could set an important precedent: if the court rules in favor of the pastor, AI companies could face liabilities similar to those of defective product manufacturers. This would force stricter controls, such as verification of medical responses by human experts before display, or integration of alert systems that detect queries about serious symptoms. However, current regulation is insufficient: the FDA has not yet classified chatbots as medical devices, and the FTC has only issued general warnings about deceptive advertising.

What should readers know?

• Never use ChatGPT or any chatbot as a substitute for a doctor. Chatbots are not designed or certified to make diagnoses and can make serious mistakes.
• If you have symptoms like chest pain, difficulty breathing, uncontrolled bleeding, or signs of a stroke (facial paralysis, difficulty speaking), seek professional help immediately. Every minute counts in medical emergencies.
• Chatbots can provide useful general information (e.g., explaining medical terms or reminding you of medication schedules), but not clinical decisions. Always cross-check any advice with reliable sources or a doctor.
• OpenAI and other companies are improving warnings and adding filters to detect medical queries, but the ultimate responsibility lies with the user. Technology is a tool, not a replacement for human judgment.
• If you use AI for health topics, verify the information with sites like MedlinePlus, the WHO, or your primary care physician. Be wary of responses that sound too confident or downplay serious symptoms.

“ChatGPT is not a doctor. It cannot diagnose, prescribe, or treat. Trusting it for health decisions is potentially deadly.” — Analyst at TheVortiq

In summary, the pastor's case is a wake-up call about the limits of AI. As technology advances, regulation and user education must catch up. Health is not a field for digital experiments: human life is at stake.