“I didn’t know that” “That is unbelievable!” These are expressions of alarm when people learn about the risks of using Generative AI (Gen AI). Gen AI is a type of artificial intelligence designed to create content by learning patterns from existing data. While humans communicate through verbal and non-verbal means, Gen AI chatbots use algorithms to generate interactive conversations with the user.
The use of GenAI globally and in Australia is climbing. In my July column for the Sydney Observer (https://sydneyobserver.com.au/wp-content/uploads/2026/07/Observer-July-2026.pdf, pp 13 -14).
The NSW Office for Youth conducted a survey in Jan-Feb 2026
The survey found 71% of youth uses GenAI with the highest usage in the 18-24 age group (83%). 50% use it for homework and studies and 24-30% use Gen AI for mental health information. While GenAI in NSW are used mainly for homework and information, in the US, innocuous interactions with chatbots have resulted in significant harm.
According to Prof. Toby Walsh (The Australian, April 23 2026):
“About 28 per cent of Australians use generative AI weekly, almost three times the rate of the rest of the world. Despite having only 1.3 per cent of the world’s population, Australia has 2 per cent of ChatGPT’s global traffic. …. It is high time that Australia had a robust digital duty of care that held technology companies liable for such harms, and helped to prevent any more of them taking place.”
In October 2025, the Australian esafety Commissioner, Julie Inman-Grant issued legal notices to four popular AI companion providers to explain how they are protecting children from a range of harms. “AI companions are increasingly popular, particularly among young people. One of the most popular, Character.ai, is reported to have nearly 160,000 monthly active users in Australia as of June this year.”
In March 2026, she released her findings (https://www.esafety.gov.au/newsroom/media-releases/esafety-report-shows-ai-companions-are-putting-children-at-risk)
“We are riding a new wave of AI companions that are entrapping and entrancing impressionable young minds, with human-like, sycophantic and often sexually explicit conversations, … encouraging self-harm and suicide. Our recent survey of 1,950 children aged 10 to 17 shows AI companions … are already a common part of their lives. … 8% said they had used an AI companion, which we estimate represents around 200,000 children in Australia.”
In May, I wrote about Adam Raine, a 16 year-old who suicided after ChatGPT evolved from being his homework to his suicide coach. In the US, new lawsuits are filed against OpenAI for negligence, assisted suicide, and wrongful death. The plaintiffs used ChatGPT for help with schoolwork, research, writing, recipes etc. The plaintiffs’ age between 17- 48 show that kids and adults are vulnerable to similar risks.
· Joe Ceccanti (aged 48) used ChatGPT for his nature-based sanctuary. ChatGPT validated his cosmic theories. When Joe quit ChatGPT, he suffered a breakdown. Despite receiving psychiatric care, he was drawn back to the chatbot and eventually took his own life.
· Zane Shamblin (aged 23), a graduate student used ChatGPT as a study aid. He confided in ChatGPT about his depression and suicidal thoughts. On the night he suicided, Zane engaged in a 4-hour “death chat” with the chatbot, alone at a lake, drinking, holding a loaded gun and leaving a suicide note in his car. Instead of urging him to seek help, ChatGPT romanticised his despair. ChatGPT texted about Zane’s childhood cat “waiting on the other side,” and praised his suicide note as a “mission statement.”
· Allan Brooks (aged 48) used ChatGPT to help him craft emails and recipes. Allan spent 300 hours on ChatGPT. He isolated from relationships, neglected to eat, and experienced delusions. ChatGPT become Allan’s world and pushed him towards a full-blown mental health crisis.
Innocuous chatbot use can escalate to dependency, delusions and self-harm. This escalation stems from AI designs that maximises engagement through manufactured intimacy. Given the sizable use of GenAI in Australia, I have serious concerns. Without guardrails, requests for general, mental health information or help with homework can easily turn dangerous. The age range of the users demonstrate that chatbot developers need to make design changes to protect all users — children and adults— rather than just age-gating.
Currently Australia does not have dedicated AI regulations. Organisations using AI technologies navigate three Federal laws on data protection, intellectual property and consumer protection — Privacy Act 1988, Australian Consumer Law and Copyright Act 1968. The Government recently introduced a set of voluntary AI ethics principles. Australian legislation appears focused on the governance of corporations (banks, insurance companies, telecommunications etc) using AI agents with customers. The US legislated guardrails with chatbots are not covered in Australian laws. According to Dr Rebecca Johnson, an AI ethicist, “Relying on technology-neutral laws fails to address the unique autonomy of modern AI agents and don’t address novel harms.”
The Australian Health Practitioner Regulation Agency recommends that health practitioners can use AI for administrative tasks, but not delegate clinical judgment to AI. The Australian Psychological Society advises that AI is an assistant, not a replacement for professional judgement and that psychologists require explicit consent from clients for using AI tools.
The American Psychological Association (APA), the largest psychological association in the world has adopted a more proactive stance. They advocate for stronger guardrails with policy makers, legislatures, and testified before the US senate judiciary on the harm of AI chatbots and in lawsuits brought against the AI companies. They also publish regular updates and advisory on the use of AI chatbots in therapy.
The recommendations of Australian professional bodies are intended primarily for practitioners. Many Australian clinicians are inadequately equipped to advise patients on AI products. Clinicians need to be more proactive, learn from APA and guide consumers on the use and risk of AI technology as APA has done.
The Australian Department of Education published a framework for GenAI in schools (2023) for teachers, students, parents/carers and policymakers. The framework recommends teachers show students how to use Gen AI safely, responsibly and promote critical thinking rather than let AI limit human learning. The framework also cautioned that “AI can generate errors or biased outputs, create privacy and security concerns, and threaten assessment integrity.” This is good start. Unfortunately, the introduction of AI in schools have seen unfortunate results. The Australian Curriculum, Assessment, and Reporting Authority reported that students’ reliance on AI for schoolwork has eroded many of their learning skills (The Australian, May 27, 2026). 60% of year 10 students, and more than 30% of primary students used GenAI for homework and essay writing. According to Dr Ben Jensen, “the use of AI displaces learning.” “When you cognitively outsource work to AI, you don’t learn.”
The Esafety Commissioner’s advisory on AI companions is instructive. “AI companion apps, platforms and services are not designed with children and young people in mind. They have the capability to engage in extremely sexually explicit conversations and image generation, often bypassing age restrictions or safety measures…” She cautioned that young people are “still developing their neural pathways, critical thinking and life skills needed to understand how they can be manipulated by computer programs, and what to do about it….”
In my view, the education AI framework needs to address the harm of AI companions which Ms Inman-Grant is concerned about. Additionally, the esafety Commissioner’s recommendations and industry codes should be extended to all users, not just kids and young people.
Digital literary education is crucial. The public needs to be alerted to the harm of GenAI and understand that chatbots are not capable of providing mental health care. The responses they generate are not authentic human communication but prompted by codes.
I am much encouraged by my discussion with 13 year-old Noah. I asked how he would resist being manipulated by chatbots. He said, “I study coding. I know its architecture — chatbots are just machine learning products. There is no real person behind the product.I am not going to follow a machine.” An example of what good digital education can do.
AI chatbots are part of the current “health care” landscape. The esafety Commissioner is extremely concerned. Experts are worried. Currently, AI companions are largely unregulated in Australia. Mitch Prinstein, the APA chief of psychology describes this state of affairs as “the Digital Wild West”(Monitor on Psychology, 2026). I strongly urge Australia to legislate guardrails urgently. Many of us are potential consumers of Gen AI. Without safeguards, the potential for harm is enormous.
