I continue to worry about the ease of producing and then disseminating deepfake imagery. Recent reporting from Wired and Indicator, as an example, noted that:
The true scale of deepfake sexual abuse taking place in schools is likely much higher. One survey by United Nations children’s agency Unicef estimates that 1.2 million children had sexual deepfakes created of them last year. One in five young people in Spain told Save the Children researchers that deepfake nudes had been created of them. Child protection group Thorn found one in eight teens know someone targeted, and in 2024, 15 percent of students surveyed by the Center for Democracy and Technology said they knew about AI-generated deepfakes linked to their school.
…
In South Korea and Australia, schools have given pupils the option not to have their photos in yearbooks or stopped posting images of students on their official social media accounts, citing their use for potential deepfake abuse. “Around the world, there have been cases where school images were taken from public social media pages, altered using AI, and turned into harmful deepfakes,” one school in Australia said. “Imagery will instead feature side profiles, silhouettes, backs of heads, distant group shots, creative filters, or approved stock photography.”
The ability to create deepfakes is fast, inexpensive, and increasingly possible offline using open-source models and weights. There are also innumerable online services that facilitate this activity at minimal cost
But this isn’t a ‘technology’ issue alone: the underlying intent to humiliate, denigrate, and exercise social control reflects entrenched social behaviours that cannot be addressed through technical measures alone. Notwithstanding the fact that technology is used to create deepfakes, the solution will not be found in controlling the technology nor even in criminalizing the activity (though both are likely part of a broader solution). Beyond statutory or regulatory responses, we need to invest in interventions that challenge the underlying attitudes that give rise to this behaviour in the first place.
We must recognize, call out, and work to counter the misogynistic norms and underlying attitudes that too often drive their creation. This is something that we owe to children, as well as to one another, and is especially important as we live in an era where more and more images are being made every day of each and every one of us. If we fail in doing so, however, expect to see social responses such as bars on public photography, production of yearbooks, and more as a defensive response to protect children and adults alike.
Excited Pixels 