Auditing Generative Models Without Generating Illegal Content: A Key Breakthrough to Protect Minors

What happened?

An interdisciplinary team from the Massachusetts Institute of Technology (MIT), led by graduate student Vinith Suriyakumar and professors Ashia Wilson and Marzyeh Ghassemi, in collaboration with the nonprofit organization Thorn, has developed an innovative auditing method for generative AI models. The technique allows determining whether a model has been specialized—using techniques such as Low-Rank Adaptation (LoRA)—to generate illegal content, particularly child sexual abuse material (CSAM), without needing to generate such content. In tests, the method identified with 100% accuracy the model variations that had been fine-tuned to produce CSAM, according to results presented at the “Trustworthy AI for Good” workshop of the International Conference on Machine Learning (ICML).

The research, published in MIT News, details that the team examined how the model's internal representations are modified during fine-tuning with LoRA. Instead of generating prohibited content, auditors analyze changes in model weights and hidden activations. This approach avoids generating CSAM, which is illegal in the United States and many other jurisdictions, even for testing purposes. Thorn, an organization dedicated to child protection, contributed its expertise in combating the sexual exploitation of minors in digital environments.

Why is it important?

The rise of open-source generative models has democratized access to AI but has also made it easier for malicious actors to adapt these models to create illegal content. According to data from the U.S. National Center for Missing and Exploited Children (NCMEC), in 2025 more than 1.5 million reports of AI-generated CSAM were received, a dramatic increase from 67,000 reports in 2024. This more than 2,000% increase in one year underscores the urgency of effective auditing tools.

Until now, auditing models to detect harmful capabilities required generating prohibited content, which is not only illegal but also poses a serious psychological risk to human evaluators. The new method removes that barrier by analyzing the model's internal representations without generating any output. This represents a crucial advance in AI safety, as it closes a blind spot that was being exploited by malicious actors. As Suriyakumar notes, “before, we had no way to measure this. It was a huge blind spot that some people were taking advantage of.”

Consequences and applications

The technique opens a new avenue for platforms hosting open-source models—such as Hugging Face or GitHub—to detect and remove dangerous models before they are downloaded and used. It could also help law enforcement identify models used to produce CSAM, facilitating investigations without exposing agents to traumatic material. The authors highlight that the approach is scalable and does not require access to the original training data, only to the LoRA-tuned model. This is particularly valuable because many fine-tuned models are distributed without training data, making traditional audits difficult.

Additionally, the method avoids trauma for human reviewers, who in the past had to expose themselves to illegal content to evaluate models. Thorn has collaborated on the research to ensure the technique is practical and ethical. In a context of increasing regulatory pressure, with laws like the EU AI Act and proposals in the U.S. to require safety audits, this tool could become a standard for model hosting platforms.

What readers should know

• Not a magic solution: The method focuses on models fine-tuned with LoRA, a popular but not the only technique. The authors note it can be extended to other fine-tuning methods, such as full fine-tuning or attention-based adapters. However, for now, it has only been validated for LoRA.
• Privacy and ethics: The audit does not require generating illegal content, making it legal and ethical. However, its implementation on platforms must balance safety with the legitimate use of models. For example, a model fine-tuned to generate medical images could be falsely flagged if it shares internal representations similar to CSAM. The authors are working to minimize false positives.
• Regulatory context: This breakthrough comes at a time of increasing regulatory pressure on AI. The EU AI Act classifies AI systems by risk, and tools for generating illegal content would fall into the unacceptable risk category. In the U.S., the 2023 Executive Order on AI and bills like the “AI Foundation Model Transparency Act” require transparency and safety audits. This method could be a technical response to those requirements.
• Limitations: The method does not detect models that generate CSAM without being specifically fine-tuned (e.g., base models with implicit knowledge). It also does not address other types of harmful content like hate speech or disinformation, though the authors believe it could be adapted. Additionally, the 100% accuracy was achieved in a controlled environment; in practice, false negatives could occur if malicious actors use evasion techniques, such as obfuscating model weights.

What do experts say?

Vinith Suriyakumar, lead author of the study, told MIT News: “This unlocks a new avenue for platforms hosting open-source models and law enforcement to actually test whether a model is capable of generating CSAM. Before, we had no way to measure it. It was a huge blind spot that some people were taking advantage of.” Ashia Wilson, co-author and associate professor at MIT, highlighted the importance of collaboration with Thorn: “Working with child protection experts allowed us to understand the real problem and design a solution that respects legality and ethics.”

External experts, such as Dr. Ben Zhao, professor of computer science at the University of Chicago, have praised the approach: “This method is a big breakthrough because it avoids the need to generate illegal content. However, it will be crucial to test it in real-world scenarios and extend it to other forms of fine-tuning.” The research was presented at the ICML workshop “Trustworthy AI for Good,” a forum recognized for its focus on AI safety and ethics.

Next steps

The MIT team plans to explore detecting other types of harmful content, such as hate speech or graphic violence, and work with platforms like Hugging Face to implement the technique at scale. They are also investigating how to extend it to models that do not use LoRA, including full fine-tuning and multimodal models. Additionally, they are developing methods to protect the privacy of legitimate models during auditing, preventing sensitive internal representations from leaking.

For now, this tool represents a promising advance that closes a critical gap in generative AI safety. As Suriyakumar notes, “we hope this can be widely adopted to prevent the spread of harmful models.” The combination of technical rigor, collaboration with child protection organizations, and ethical focus positions this work as a milestone in the fight against malicious use of AI.