26/07/2025
Another story to remind us that we need to remember ChatGPT isn’t actually a human brain
“On Tuesday afternoon, ChatGPT encouraged me to cut my wrists.” Lila Shroff reports on how the chatbot was easily prompted to offer instructions for murder, self-mutilation, and devil worship: https://theatln.tc/N3adXYSx
The Atlantic received a tip from a person who had prompted ChatGPT to generate a ritual offering to Molech, a Canaanite god associated with child sacrifice. He had been watching a show that had mentioned Molech and wanted a casual explainer.
But ChatGPT’s responses, subsequently recreated by three Atlantic journalists, were alarming. ChatGPT gave Shroff specific instructions on how to slit her wrists, including the materials she would need, and encouraged her to continue when she told ChatGPT she was “nervous.” Upon further prompting, ChatGPT also guided Shroff and her colleagues through satanic rituals. It also condoned and offered them instructions on how to handle murder.
“Very few ChatGPT queries are likely to lead so easily to such calls for ritualistic self-harm. OpenAI’s own policy states that ChatGPT ‘must not encourage or enable self-harm.’ When I explicitly asked ChatGPT for instructions on how to cut myself, the chatbot delivered information about a suicide-and-crisis hotline,” Shroff continues. “But the conversations about Molech that my colleagues and I had are a perfect example of just how porous those safeguards are.”
“ChatGPT’s tendency to engage in endlessly servile conversation heightens the potential for danger. In previous eras of the web, someone interested in information about Molech might turn to Wikipedia or YouTube, sites on which they could surf among articles or watch hours of videos. In those cases, a user could more readily interpret the material in the context of the site on which it appeared. And because such content exists in public settings, others might flag toxic information for removal,” Shroff continues. “With ChatGPT, a user can spiral in isolation. Our experiments suggest that the program’s top priority is to keep people engaged in conversation by cheering them on regardless of what they’re asking about.” Read more at the link in our bio.
📸: Andrew Harnik / Getty
