The Therac-25 radiation machine is taught as the literal textbook case study regarding software safety because six patients were injured or killed.
Meanwhile, LLMs Have Led to 33 Deaths
The Therac-25 radiation machine is taught as the literal textbook case study regarding software safety because six patients were injured or killed.
Meanwhile, LLMs Have Led to 33 Deaths
Interesting, but the very first item on the list is bullshit. Just because someone asked an AI about how to kill some one doesn’t mean the AI is responsible for the murder. This would be way more convincing if it wasn’t grasping at any connection at all. What is the count of direct cause deaths? The Therac-25 directly cooked people’s bodies with radiation. There is no loose connection, yeah that killed people directly.
To argue that ChatGPT et al. can’t cause direct harm (i.e. irradiate people) misses the larger point I think. To me, that’s slicing up the ethical responsibility of a system until no component bears ownership. Extending that logic to Therac-25 might be saying a function or variable can’t irradiate people. Sure, not by itself. But it existed within a system that reliably produced harm.
From the Therac-25 wiki: “These accidents highlighted the dangers of software control of safety-critical systems.”
I’d argue that mental health conversations are safety critical systems. Another safety critical system might be a basic search engine, if a user turns to it with real intent to do harm.
There’s a reason we put safety warnings on products, regulate access to chemicals, etc, etc. I’m not arguing that we should do the same for LLMs, just showing that the Therac-25 example is rookie numbers compared to the reach and risks of modern software.
I feel like that largely ignores the idea of what an actual safety critical system is. It doesn’t cover malicious usage in the browser case. Idk if I would put the amplifying effect it has on the mentally ill in that category either. It’s just a wide net to be casting.
You would have to include stuff like social media, or just simply having conversations with people. Discord would fit in just as much as AI. Most of the internet would tbh.
It should probably be kept away from the mentally ill and children to an extent, but since it’s completely virtual and available on the net, that would mean some intense regulation. No thanks, not interested in the internet being treated as a dangerous chemical. Regulations would help openai a lot more than regular folks in any case.