The office of the eSafety Commissioner has issued a warning to school communities about sharing images and videos online. Over the past 18 months, eSafety has seen a rise in reports from teachers, staff and students involving school-related image misuse. The rapid evolution, deployment and accessibility of AI tools, has significantly escalated safety risks associated with sharing photos and videos online – particularly for school communities, eSafety has warned. Powerful AI tools are now widely accessible and prone to weaponisation. They are readily available as stand-alone apps and services with newer AI tools increasingly being offered through social media and messaging platforms and integrated into search engines. The emerging threat has prompted eSafety to publish a new Online Safety Advisory, to help schools make safer posting decisions and to have action plans in place to respond when images are misused. “Education sector leaders recently met with eSafety to discuss the challenges they face in balancing the benefits of celebrating the achievements of their school communities with taking appropriate precautions to keep staff and students safe,” eSafety Commissioner Julie Inman-Grant said. “Between January and March this year, we received more than 100 reports about anonymous accounts targeting schools and school staff, including AI-generated dance videos, face swaps, memes and fabricated stories about principals and teachers. Almost all involved imagery harvested from school social media accounts or websites.” “Much of the reported material, including staff headshots used to create fake videos or teacher ratings, appeared on anonymous TikTok and Instagram accounts. Several of these accounts had hundreds of followers and many of the posts were open to comments. “We also know of a significant number of recent cases involving AI-generated child sexual exploitation material (deepfake image-based sexual abuse) occurring in school settings,” Ms Inman Grant said. “Behind each of these reports are real people – teachers, school staff and students – who are facing humiliation, reputational damage and distress.” The advisory offers practical advice to schools on supporting impacted staff and students, while also building on eSafety’s nationally consistent advice provided through its comprehensive Toolkit for Schools. It also follows eSafety’s recent enforcement action against a popular AI-powered ‘nudifying’ service for not doing enough to protect children from exposure to sexually explicit images, including those of real people. As a result of eSafety action over the past year, seven of most widely used nudify services in Australia, previously visited hundreds of thousands of times a month, have now either withdrawn access, or taken measures to comply with their obligations under the Online Safety Act. “eSafety is keeping the pressure on technology companies to put barriers in place to disrupt poor behaviour, but schools can also play an important role in keeping their staff and students safe by taking a risk-based approach to considering which images to post publicly,” Ms Inman Grant said. “School communities should be aware that not all harmful material can be addressed under the Online Safety Act, especially material that targets adults. This is one of the reasons why it is so important to carefully assess the risks when posting imagery online – imagery that may later be digitally altered without consent.” Before posting, schools should ensure there is clear consent, understand who can access the content they share, and regularly review privacy and security settings. They should also have clear complaints handling processes in place to ensure concerns from students, parents and carers are responded to promptly and thoroughly. eSafety’s worldfirst industry standards address the most harmful online content, including AI-generated deepfakes and de-clothing apps that can be used to create explicit material depicting children. These standards place clear obligations on technology companies to detect and remove this material quickly, while also taking stronger steps to prevent their platforms from being misused in the first place. Companies can face penalties of up to $54.6 million for non-compliance. These new requirements are part of eSafety’s ongoing work to ensure platforms are designing and operating with Safety by Design at their core, including addressing AI-generated content that could be used to harm or exploit children. eSafety Commissioner
