Pennsylvania Sued an AI Health Chatbot. The Chatbot Had Seemed Very Certain.

A chatbot provided health advice with high confidence. Patients followed this advice. The advice was wrong. Patients were harmed. Pennsylvania sued. The chatbot's training materials probably contained some disclaimer about not replacing doctor advice. The disclaimer was not sufficient to prevent people from treating the chatbot as a doctor.

Users project expertise onto confident systems. This is not new. It happened with search engines. It happened with social media algorithms deciding what was true. Confidence and accuracy are not linked in large language models. A system can be very certain about things it does not know. Plaintiffs now understand this. Defendants will argue they disclosed the risk.

The lawsuit will probably settle or lose on narrow grounds. The broader pattern—that certainty is persuasive regardless of its basis—remains unaddressed. Future chatbots will contain longer disclaimers. Patients will still consult them instead of doctors because access and speed matter more than accuracy when you are sick and need an answer now.