I can set a reminder in my phone to take medication, but does that qualify as the phone making health advice? ChatGPT is not marketed as a health product, so is it subject to laws about health claims? Does any restriction on speech apply in any way at all, if I ask it for legal, engineering, therapy, can it really be held to account for anything?
Summary: a lawsuit [0] was filed this week by the Chinese parents of an Amazon engineer who died in February 2026, blaming OpenAI for how GPT-4o assisted their daughter in devising a deadly prescription regimen for her epilepsy medication.
The woman in the article was in an ongoing, multi-month chat discussing her medication with ChatGPT.
I don't use it for medical stuff very often, let alone on models within the last year or so. But I have certainly experienced the "drift" that can occur (even on subjects with sensitive guardrails like medical advice) where it gradually bends towards the user's perspective and away from the inflexible canned responses from the system prompt over time.
The language in many of the quotes from ChatGPT included in the article seem like they could be miscategorized as following directives to be supportive/friendly/helpful even though it implicitly was providing (reckless) medical advice at the same time.
I can set a reminder in my phone to take medication, but does that qualify as the phone making health advice? ChatGPT is not marketed as a health product, so is it subject to laws about health claims? Does any restriction on speech apply in any way at all, if I ask it for legal, engineering, therapy, can it really be held to account for anything?
ChatGPT does advertise health advice: https://chatgpt.com/health/
Though notably she was using GPT-4o and not their health product, 4o being much more sycophantic and definitely their most controversial model.
ChatGPT Health just tells me to go to the hospital right away for any minor thing
Gift link: https://www.nytimes.com/2026/10/27/well/ai-chatgpt-medicatio...
Summary: a lawsuit [0] was filed this week by the Chinese parents of an Amazon engineer who died in February 2026, blaming OpenAI for how GPT-4o assisted their daughter in devising a deadly prescription regimen for her epilepsy medication.
[0] https://schenklawfirm.com/wp-content/uploads/2026/10/wan-v-o...
https://archive.is/SPkBD
ChatGPT is overly cautious and wants to send me to see a doctor or go to the hospital for pretty much anything.
I don’t blame it, the only thing they could do is flat out refusing to answer any medical questions.
The woman in the article was in an ongoing, multi-month chat discussing her medication with ChatGPT.
I don't use it for medical stuff very often, let alone on models within the last year or so. But I have certainly experienced the "drift" that can occur (even on subjects with sensitive guardrails like medical advice) where it gradually bends towards the user's perspective and away from the inflexible canned responses from the system prompt over time.
The language in many of the quotes from ChatGPT included in the article seem like they could be miscategorized as following directives to be supportive/friendly/helpful even though it implicitly was providing (reckless) medical advice at the same time.