Abstract
Evaluation of large language models (LLMs) for safety, security, and privacy (SSP) relies heavily on static benchmarks, which suffer from score saturation, data contamination, and aggregation artifacts, and fail to capture sensitivity to linguistic variation. As a result, models that perform well on fixed test sets often fail under semantically equivalent rephrasings. We introduce SSP-Bench, a dynamic benchmarking framework that generates evaluation instances on demand while preserving domain consistency. The framework ensures label validity through externally grounded sources, enforces scope via service-specific validation, and calibrates difficulty using a multi-model steering panel. Benchmark construction is formulated as a multi-objective optimization problem over difficulty, separability, novelty, and diversity. Across 24 models and four SSP services, SSP-Bench reveals systematic failures of static evaluation, including near-zero correlation in safety rankings due to construct mixing, strong safety--over-refusal coupling, and hidden within-family regressions. These results show that static benchmarks can misrepresent model behavior, motivating dynamic, deployment-relevant evaluation.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Deniz, F., Boshmaf, Y., & Khalil, I. (2026). SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation. https://omanscience.com/en/articles/ssp-bench-a-hybrid-data-generation-framework-for-safety-security-and-privacy-evaluation
MLA 9
Deniz, Fatih, et al. "SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation." https://omanscience.com/en/articles/ssp-bench-a-hybrid-data-generation-framework-for-safety-security-and-privacy-evaluation.
Chicago (author–date)
Deniz, Fatih, Yazan Boshmaf, and Issa Khalil. 2026. "SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation." https://omanscience.com/en/articles/ssp-bench-a-hybrid-data-generation-framework-for-safety-security-and-privacy-evaluation.
Harvard
Deniz, F., Boshmaf, Y. and Khalil, I. (2026) 'SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation', Available at: https://omanscience.com/en/articles/ssp-bench-a-hybrid-data-generation-framework-for-safety-security-and-privacy-evaluation.
Vancouver
Deniz F, Boshmaf Y, Khalil I. SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation. https://omanscience.com/en/articles/ssp-bench-a-hybrid-data-generation-framework-for-safety-security-and-privacy-evaluation
IEEE
F. Deniz, Y. Boshmaf, and I. Khalil, "SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation," https://omanscience.com/en/articles/ssp-bench-a-hybrid-data-generation-framework-for-safety-security-and-privacy-evaluation.