Abstract
Existing Simulink benchmarks mainly evaluate whether generated models compile, execute, or resemble a reference model. These criteria do not establish whether a model satisfies its engineering requirements. We introduce SimuVerity, a benchmark of 101 text-to-executable Simulink model-generation tasks across ten engineering domains. For each task, executable-system profiles ground the engineering specification and four families of native simulation scenarios. A hierarchical evaluator first checks artifact delivery, native executability, and engineering qualification, then scores qualified models across six dimensions covering accuracy, output quality, mechanistic fidelity, control and causal integrity, operating-domain robustness, and dynamic response. We evaluate six agent systems with SimuVerity. The best system achieves an overall score of only 42.86. The results show that structural similarity is a poor proxy for engineering performance: capability bottlenecks arise both in producing qualified implementations and in satisfying multidimensional requirements after qualification. Meanwhile, some high-scoring models still exhibit severe visual-layout disorder. SimuVerity provides a systematic basis for assessing agents' engineering capabilities and diagnosing failures in executable Simulink model generation.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Zhang, R., Wang, J., Li, M., Luo, H., Wang, C., Mou, G., Lai, K., Lv, H., Wang, J., Zheng, Y., Yang, A., & Wang, X. (2026). SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation. https://omanscience.com/en/articles/simuverity-benchmarking-agents-for-engineering-grade-simulink-model-generation
MLA 9
Zhang, Ruiqi, et al. "SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation." https://omanscience.com/en/articles/simuverity-benchmarking-agents-for-engineering-grade-simulink-model-generation.
Chicago (author–date)
Zhang, Ruiqi, Jiahao Wang, Mingxuan Li, Haichen Luo, Chaoting Wang, Guoyu Mou, Keyu Lai, Hanchao Lv, Jiaxu Wang, Yibo Zheng, Aijun Yang, and Xiaohua Wang. 2026. "SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation." https://omanscience.com/en/articles/simuverity-benchmarking-agents-for-engineering-grade-simulink-model-generation.
Harvard
Zhang, R., Wang, J., Li, M., Luo, H., Wang, C., Mou, G., Lai, K., Lv, H., Wang, J., Zheng, Y., Yang, A. and Wang, X. (2026) 'SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation', Available at: https://omanscience.com/en/articles/simuverity-benchmarking-agents-for-engineering-grade-simulink-model-generation.
Vancouver
Zhang R, Wang J, Li M, Luo H, Wang C, Mou G, et al. SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation. https://omanscience.com/en/articles/simuverity-benchmarking-agents-for-engineering-grade-simulink-model-generation
IEEE
R. Zhang, J. Wang, M. Li, H. Luo, C. Wang, G. Mou, K. Lai, H. Lv, J. Wang, Y. Zheng, A. Yang, and X. Wang, "SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation," https://omanscience.com/en/articles/simuverity-benchmarking-agents-for-engineering-grade-simulink-model-generation.