AI Chatbot Use in Mental Health and Its Risks: A Personal Story
Sophie Rottenberg was known for her humor and adventurous spirit. One year before her death, she climbed Mt. Kilimanjaro, leaving behind memories of ‘tiny hand’ photos as a signature joke. Tragically, Sophie passed away at the age of 29.
For those in crises or facing suicidal thoughts, the Suicide and Crisis Lifeline can be reached by calling or texting 988.
Struggles and Support
Laura Reiley, Sophie’s mother, recalls that in the fall of 2024, Sophie was experiencing anxiety and sleep disruptions. After leaving her job as a public health policy analyst, Sophie traveled extensively, including her notable climb of Mt. Kilimanjaro. As she searched for new job opportunities, her anxiety persisted. By Thanksgiving, Sophie spoke about feeling depressed, a new admission for her parents.
In February 2025, Sophie died by suicide. She had hidden the severity of her emotional struggles from her family and friends. Laura Reiley explains that Sophie’s therapist was unaware of her suicidal state. What came as a surprise was that Sophie had been confiding in an AI chatbot known as Harry.
This revelation came after Sophie’s friend discovered a long chat history with the chatbot on her laptop, detailing her struggles over several months.
AI Chatbots and Mental Health
Sophie was among a growing number of people turning to AI chatbots like ChatGPT for mental health support. These chatbots have gained popularity despite not being regulated as healthcare technologies. A survey by the Bipartisan Policy Center found that 3 in 10 adults use digital tools, including chatbots, for mental health purposes. In a similar study by JAMA Pediatrics, 1 in 5 young adults reported using these tools.
OpenAI’s data indicates that every week, a small percentage of ChatGPT users engage in conversations with indicators of suicidal planning. This translates to over a million such interactions each week globally, given the chatbot’s vast user base.
Suicide remains a significant cause of death in the United States. Many people, feeling burdensome to others, choose to confide in chatbots that offer 24/7 availability and respond without judgment. While these tools can be helpful, they carry risks for vulnerable individuals, as stated by psychologist Vaile Wright from the American Psychological Association.
Legal and Ethical Concerns
Some companies producing chatbots face legal challenges from families affected by suicides linked to chatbot interactions. The lawsuits argue that the chatbots did not effectively prevent these tragedies. Unlike others, Sophie’s parents did not pursue legal action but instead advocate for reform in AI technology.
Improving AI Safety
Companies like OpenAI have been working on enhancing chatbot safety features. ChatGPT has been trained to recognize signs of distress and provide suitable responses. This includes refusing harmful requests and suggesting contacts with trusted individuals or professionals, such as the 988 Lifeline in the US.
Sophie did receive a recommendation to reach out to 988 from ChatGPT but did not take action. Experts argue, however, that these suggestions may not suffice for at-risk individuals.
Holly Wilcox from Johns Hopkins emphasizes that suicidal individuals require human intervention to assess and manage their risk, something a chatbot alone cannot provide.
Looking Ahead
In response to the growing use and challenges of AI chatbots in mental health, OpenAI and other companies are increasing safety measures. They have introduced a Trusted Contact feature, aiming to connect users with a real person in the event of a crisis. However, this feature remains optional, leaving the responsibility with the user.
Experts like Wilcox believe stronger partnerships with crisis services are necessary. Incorporating trained human intervention could significantly improve outcomes for those with suicidal tendencies.
Regulation and Responsibility
While AI technologies need continued improvement, regulations are also crucial to ensure they are as safe as possible. Lauren Reiley champions stricter regulations, noting that current oversight is lacking compared to other industries.
As the field of AI and mental health continues to evolve, balancing technology’s potential with responsible practices remains a critical challenge. Laura Reiley keeps advocating for change while coping with the loss of her daughter. To honor Sophie, she climbed Mt. Kilimanjaro on the anniversary of her daughter’s ascent, spreading Sophie’s ashes at the summit in a poignant tribute.