Among the new research papers published last week that have become major topics in the AI field, there is a quite controversial one that appeared in the journal Acta Psychiatrica Scandinavica. It identifies clinical notes from 38 patients in a large Danish psychiatric system and coincides with AI chatbot use, mostly ChatGPT, with harmful mental health effects, like delusions, suicidality, and eating disorders.
The research raises a very important topic, though there are still questions about how representative the sample is, including a minor fraction, an internal sample, age and gender distribution. To better understand what this study means for commercial AI development, we talked to Volodymyr Getmanskyi.
Background & experience:
- Over 15 years of practical experience in advanced data analysis and modelling. Currently manages a large AI team while providing presales and delivery support for complex AI implementations.
- Technical expertise encompasses the full spectrum of AI technologies relevant to government applications: NLP/AI agents for document/information processing and assistance, compliance checks, computer vision for security and monitoring systems, AI-based RPA for process automation, and predictive modelling for policy planning and resource optimisation.
What does this study actually tell us, and how reliable are the findings?
Volodymyr: The study is an important signal, but we need to read it carefully. Its methodology relies on keyword searches across over 10 million clinical notes, flagging 181 mentions and finding 38 "compatible" with harm.
However, it explicitly lacks causality proof; there were no counterfactuals or systematic patient questioning used. The assessments were made by just two researchers who applied subjective judgments on fairly vague criteria, such as chatbots serving as an "object for delusions." And because exact case details are hidden due to privacy constraints, the replication (~Popper criteria) is impossible. Also, the narrow focus on ChatGPT cases only (spelling options generated by ChatGPT itself) might miss broader AI use (“LLM” in clinical notes), inflating perceived specificity.
That said, the results are still very eloquent and bring many thoughts, like understanding that we are at the start or middle of a generative AI grand cycle; this data is very likely the tip of the iceberg.
What were the key patterns of harm observed?
Volodymyr: The majority of observed patterns were about the reinforcement of existing symptoms (paranoia, mania, eating disorders). This is partially due to the fact that all such systems are designed to be cooperative and act as idealised conversational partners: always available, responsive, and non-confrontational, so they may engage rather than challenge.
For example, instead of rejecting any strange paranoid claims, a conversational agent may provide speculative or technical-sounding responses that reinforce the patient’s distorted reality. The same is with the guidance, which leads to opposite results (self-harm, aggressive diets).
But it is important to mention the other side of the data as well: in 32 cases, the agents had positive effects. Patients reported reduced loneliness and used the system as a form of talk therapy. So, the whole picture is nuanced, not purely negative.
What does this mean for companies building commercial AI agents?
Volodymyr: Developers of conversational AI must prioritise mental health safeguards, even starting from usage limits (avoid overusage and addiction), delusion-detection prompts, or mandatory clinician referrals for at-risk users, potentially slowing agents' development under such regulatory.
More importantly, these findings underscore an urgent need for cross-disciplinary input. We cannot wait for some hypothetical "more perfect LLM version 14.0” to address this. Psychiatrists and other subject matter experts must be involved in agent development, shifting the design philosophy from engagement-focused to risk-averse.
Related insights
The breadth of knowledge and understanding that ELEKS has within its walls allows us to leverage that expertise to make superior deliverables for our customers. When you work with ELEKS, you are working with the top 1% of the aptitude and engineering excellence of the whole country.
Right from the start, we really liked ELEKS’ commitment and engagement. They came to us with their best people to try to understand our context, our business idea, and developed the first prototype with us. They were very professional and very customer oriented. I think, without ELEKS it probably would not have been possible to have such a successful product in such a short period of time.
ELEKS has been involved in the development of a number of our consumer-facing websites and mobile applications that allow our customers to easily track their shipments, get the information they need as well as stay in touch with us. We’ve appreciated the level of ELEKS’ expertise, responsiveness and attention to details.
