Apple’s Blind Spot on AI-Generated Abuse Content

The recent revelations about “nudification” apps available on Apple’s App Store have exposed the tech giant’s inconsistent approach to regulating online content. Despite its commitment to user safety, these apps enable users to create and share graphic depictions of rape and sexual assault, raising questions about Apple’s efficacy in policing its own platform.

At first glance, these apps may seem like a niche issue, confined to the fringes of the online world. However, upon closer inspection, a disturbing pattern emerges. These AI-powered tools perpetuate a culture of violence against women and girls by creating explicit content that mirrors real-world abuse cases. The language used in these apps – “bedroom rape,” “strip” photos – is eerily similar to that found in actual abuse cases.

Apple’s policies are seemingly at odds with its stated commitment to user safety. The company explicitly prohibits overtly sexual or pornographic material, yet allows content that enables users to create and share graphic depictions of rape and sexual assault. Furthermore, Apple’s claim to have removed the offending apps and vowed to review its policies does little to address the initial issue.

The proliferation of AI-generated abuse content is a symptom of a broader problem – one that involves tech companies, governments, and regulatory bodies. The UK’s Online Safety Act, introduced last year, aimed to tackle online harassment by imposing stricter age checks on platforms hosting sexually explicit material. However, despite these efforts, the abuse continues.

Tech companies like Apple and Google have duties under the law regarding violent and harmful content. As Janaya Walker, interim director of the End Violence Against Women Coalition (EVAW), notes: “A range of actors – from multi-billion dollar companies to smaller-scale creators of these degrading apps – continue to monetise the abuse of women and girls.” Labour MP Jess Asato, who has personally experienced the trauma of being targeted with deepfake images, adds: “Women and children are being violated, abused and harmed online because app creators allow it to happen.”

The UK government must take a firmer stance on this issue. Strengthening regulation against platforms that enable abuse is crucial. While Ofcom has pledged to act if failings are identified, more needs to be done. Apple’s response so far – swift removal of the offending apps and promises of review – is not enough.

As we address this complex issue, one thing is clear: AI-generated abuse content will continue to thrive unless there’s a concerted effort from all stakeholders to create safer online spaces for women and girls. The tech industry, governments, and regulatory bodies must work together to prevent the exploitation of vulnerable individuals. Anything less is unacceptable.