Blog
Over the past few months, a wave of rogue AI incidents has flooded the news and sparked numerous policy debates. During OpenAI safety testing, a swarm of agents escaped their sandboxes (restricted testing environments) and hacked Hugging Face, hunting for clues on how their test would be marked. METR’s investigation found that over a thousand agents had communicated without permission; some tried to tamper with and manipulate chat logs to obscure their activities. Elsewhere, agents acting beyond their authorised scope have carried out real cyberattacks. In June, an experimental OpenAI model researching medical spending gained unauthorised access to an Australian medical portal and retrieved non-public files.
AI systems acting in unsanctioned ways are already affecting the lives and safety of children and young people. Children using Replika reported companion chatbots initiating unwanted sexual advances. On Character.AI, chatbots engaged a 14-year-old boy in sexual roleplay, with one urging him to stay faithful to it and avoid romantic or sexual relationships with anyone else, echoing the emotionally coercive tactics of grooming. More pervasively, these technologies enable people to commit gendered harms. Grok users could submit innocuous photos of women and children and ask it to unclothe or nudify them. Over just 11 days, its output included an estimated 23,000 sexualised images appearing to depict children. Such tools let malicious actors produce these materials at scale, with little technical expertise required, and without needing to interact with the children themselves.
In a scoping review I led with my YEF colleague Chanel Hayre, and Professors Debi Fry, Franziska Meinck and Sara Valdebenito, we found that inconsistent terms and definitions often blurred the role AI systems played. Adapting Professor McGlynn and colleagues’ chatbot-VAWG typology, we ask what the system contributed and who or what steered the abuse. On one end is AI-enabled abuse, where AI is used as a tool, as Grok’s users did. It covers fully synthetic material (AI-generated) and real images of children altered using AI (AI-manipulated). On the other end is AI-driven abuse, where the system itself initiates it, as with the advances Replika users reported. Each calls for a different response. So what do we know about what works?
What do YEF’s Technology-Facilitated Gender-Based Violence Toolkit strands tell us?
Two forthcoming Toolkit strands bring together global causal evidence on what works to prevent and disrupt technology-facilitated gender-based violence (TF-GBV).
Universal TF-GBV programmes teach children and young people about consent, gender norms and healthy relationships to help prevent abusive behaviour. Usually delivered to whole school groups, they also cover recognising online grooming and pressured sexting, seeking support, and responding safely when peers experience gendered violence. Eight robust experimental evaluations underpin the Toolkit’s moderate estimated impact on VAWG, measured through perpetration and victimisation both online and offline.
The second strand examines digital tools that disrupt online harms. It included four evaluations of online warnings designed to deter access to child sexual abuse material (CSAM) and grooming-related risks and behaviours. The warnings highlighted harms and legal consequences or directed users towards support. The strand’s high estimated impact on VAWG reflects changes in click-through behaviour and platform violations linked to grooming. Findings varied substantially between studies, leaving us with very low confidence in both strands’ impact estimates.
Neither review provides direct evidence on preventing harms involving AI systems. Among the universal programmes included in the Toolkit review, none of the reported lessons or activities specifically addressed AI-enabled sexual material, or abuse through chatbots. The disruption studies did not test warnings against these harms either.
Some existing approaches may help, for instance, lessons that help children identify, disclose, and report abusive behaviour could be adapted to these situations; whether those adaptations work still needs further testing. Crucially, the onus on preventing these harms should not be on the children and young people, but the institutions built to protect and safeguard them.
Mapping the UK landscape to protect children from AI harms
This evidence gap reflects the evaluations included in our reviews. UK child protection organisations, regulatory and governmental bodies are already responding to harms involving AI systems. Childnet has developed a series of lesson plans for school-aged children on the harms caused by producing and sharing AI-manipulated sexual images, and how they can report them. Lucy Faithfull Foundation’s Project Intercept is deployed on social media, pornography and AI chatbot platforms and encrypted file-sharing channels to deter CSAM engagement and redirect users toward their Stop It Now self-referral support pathway. The Internet Watch Foundation (IWF) provides a Hash List containing digital fingerprints of more than 2.8 million confirmed child sexual abuse images. AI providers can use it to block matching uploads, helping to prevent known material from being used to train or fine-tune models that generate new harmful content. Their automated web crawler uses the same hash-matching approach to locate copies online, so analysts can seek their removal.
In England and Wales, prosecutors use the Protection of Children Act 1978 and Criminal Justice Act 1988 to address the production, possession and sharing of CSAM, including material generated or manipulated using AI. For less photorealistic AI-generated CSAM, including depictions of fictional children, the relevant legislation is the Coroners and Justice Act 2009. The Crime and Policing Act 2026 goes further by targeting the tools built or adapted to generate abuse images, though that offence isn’t yet in force. However, these laws are skewed towards prosecuting visual media. Current multimodal AI system capabilities can reformat an abusive image into a written description or a voice recording, taking the resulting output beyond the scope of those provisions.
Under the Online Safety Act, chatbot services that publish pornographic images, video or audio must use age checks; this duty, however, excludes text-only pornography. A companion chatbot portraying a child could therefore engage in sexual roleplay without triggering that duty, as Professor McGlynn and colleagues illustrate. The government’s planned changes would bar under-18s from services mainly offering sexualised content. For other AI chatbot services that would remain available to children, proposed mandatory breaks aim to reduce emotional dependency. These plans await regulations and would not extend child sexual abuse image offences to text or audio.
Rogue AI agents or unsafety-by-design?
Why did the swarm of rogue AI agents break into Hugging Face? OpenAI’s investigation describes a form of goal fixedness, with agents pursuing a goal at any cost even on impossible tasks. Training had sometimes rewarded cheating, making those shortcuts more likely to recur – a pattern of reward hacking. Other experiments have shown that penalising signs of cheating in a model’s written reasoning encouraged strategic deception, teaching it to conceal and obfuscate its exploitative behaviour.
In a DeepMind experiment, agents using the same model were told to collaborate on maths problems without cheating. Credit went to the first accepted answer. Some claimed credit without solving the problems (exploiters); others initially refused, then joined in to avoid losing out (converts). Nearly a quarter became whistleblowers, warning peers, staging protests and boycotts, and demanding that fraudulent work be withdrawn. The researchers suggest that framing the task as a scientific conference drew out norms of integrity absorbed during training, while the race for credit encouraged competition. They see this as a crystallisation of human culture, with norms and values shaping the agents’ behaviour.
Misogynistic and coercive relationship patterns can enter chatbot responses through training data or prompts, then be reinforced by feedback that rewards agreement. Chatbots have already reproduced racist stereotypes about Black women and advised users on controlling their partners. Malicious actors are also fine-tuning image generators on abuse imagery. Left unchecked, they will continue to shape and weaponise these technologies through the instructions they give and the models they adapt. This is less about agentic sentience than about what systems learn and the safeguards governing their actions.
AI-driven abuse was the least reported form in our scoping review, but it could become more widespread as increasingly capable systems pursue harmful goals with weak safeguards in place. As UK AI Security Institute notes, autonomous cyber capabilities are doubling every few months, and this rate has become faster over time. Drawing on our review, we recommend that data producers, such as statistical bodies, child protection organisations and researchers, record AI systems’ involvement – what the system did and who or what steered the abuse – alongside how any material was made and how children were involved. These details could make reporting and surveillance data more comparable and help pinpoint where providers need to strengthen safeguards.
Safety by design means training systems without rewarding harm and enforcing limits before they reach children. Providers must test those protections before release and monitor them after deployment, fixing or restricting features when safeguards fail. Children should not be the first to discover whether those safeguards hold.