Abstract
The rising number of concept unlearning techniques for text-to-image (T2I) diffusion models has produced a fragmented evaluation landscape. Methods are assessed under heterogeneous experimental conditions making principled cross-method comparison difficult. We present eval-unlearn, an open-source Python library providing a unified, reproducible benchmarking framework for concept unlearning in T2I Diffusion models. eval-unlearn integrates twelve published unlearning techniques spanning fine-tuning, closed-form model editing, and inference-time intervention, alongside nine complementary evaluation metrics covering erasure efficacy, adversarial robustness, generative quality, and concept retention. Its plugin architecture lets third-party techniques and metrics self-register without modifying the core framework, and its streaming, batched pipeline supports efficient evaluation of both standard NSFW concepts and arbitrary general concepts. As a further contribution, we release a public leaderboard on HuggingFace along with an interactive tool for real-time evaluation of unlearning techniques. The leaderboard compares nudity concept erasure case study across all twelve techniques, exposing significant accuracy-quality trade-offs that are obscured by heterogeneous evaluation. eval-unlearn is released under the MIT license; the package, code, leaderboard, and documentation are all available at https://eval-unlearn.readthedocs.io.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Mansi, Raghavan, N., Huang, Z., Ong, K. S., Lim, J. S., Ling, B. S. X., & Leofante, F. (2026). eval-unlearn: Benchmarking unlearning in Text-to-Image Diffusion Models. https://omanscience.com/en/articles/eval-unlearn-benchmarking-unlearning-in-text-to-image-diffusion-models
MLA 9
Mansi, , et al. "eval-unlearn: Benchmarking unlearning in Text-to-Image Diffusion Models." https://omanscience.com/en/articles/eval-unlearn-benchmarking-unlearning-in-text-to-image-diffusion-models.
Chicago (author–date)
Mansi, Nikhil Raghavan, Zixia Huang, Kai Sheng Ong, Ji Shen Lim, Brandon Siao Xiang Ling, and Francesco Leofante. 2026. "eval-unlearn: Benchmarking unlearning in Text-to-Image Diffusion Models." https://omanscience.com/en/articles/eval-unlearn-benchmarking-unlearning-in-text-to-image-diffusion-models.
Harvard
Mansi, Raghavan, N., Huang, Z., Ong, K. S., Lim, J. S., Ling, B. S. X. and Leofante, F. (2026) 'eval-unlearn: Benchmarking unlearning in Text-to-Image Diffusion Models', Available at: https://omanscience.com/en/articles/eval-unlearn-benchmarking-unlearning-in-text-to-image-diffusion-models.
Vancouver
Mansi, Raghavan N, Huang Z, Ong KS, Lim JS, Ling BSX, et al. eval-unlearn: Benchmarking unlearning in Text-to-Image Diffusion Models. https://omanscience.com/en/articles/eval-unlearn-benchmarking-unlearning-in-text-to-image-diffusion-models
IEEE
Mansi, N. Raghavan, Z. Huang, K. S. Ong, J. S. Lim, B. S. X. Ling, and F. Leofante, "eval-unlearn: Benchmarking unlearning in Text-to-Image Diffusion Models," https://omanscience.com/en/articles/eval-unlearn-benchmarking-unlearning-in-text-to-image-diffusion-models.