الملخص
Detecting pretraining data in large language models is challenging because high likelihood can reflect either training exposure or strong generalization. In the joint space of prediction loss and predictive entropy, a likelihood-only detector uses a horizontal boundary and can mistake predictable non-members for members. Motivated by this, we introduce an inclined boundary that evaluates prediction loss relative to predictive entropy. Our analysis shows that entropy correction can preserve the expected membership signal while reducing its variance, thereby improving standardized member--non-member separation. We further extend the mean--variance analysis to the more general setting with a nonzero mean entropy gap. Interestingly, this entropy-adjusted score admits a Helmholtz free-energy interpretation, leading to Energy Transfer Detection (ETD), which views pretraining data detection from a macroscopic residual free-energy transfer perspective. Extensive experiments show that ETD achieves the best average detection performance, improving average AUROC by up to 3.5\% and TPR@5\%FPR by up to 5.1\%, while remaining robust across diverse settings.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Ke, C., Liu, Z., Liu, Q., Zhuang, Y., Zhang, J., Huang, Z., & Wang, S. (2026). Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective. https://omanscience.com/ar/articles/detecting-pretraining-data-in-large-language-models-from-a-free-energy-perspective
MLA 9
Ke, Chenye, et al. "Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective." https://omanscience.com/ar/articles/detecting-pretraining-data-in-large-language-models-from-a-free-energy-perspective.
شيكاغو (المؤلف–التاريخ)
Ke, Chenye, Zirui Liu, Qi Liu, Yan Zhuang, Jintao Zhang, Zhenya Huang, and Shijin Wang. 2026. "Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective." https://omanscience.com/ar/articles/detecting-pretraining-data-in-large-language-models-from-a-free-energy-perspective.
هارفارد
Ke, C., Liu, Z., Liu, Q., Zhuang, Y., Zhang, J., Huang, Z. and Wang, S. (2026) 'Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective', Available at: https://omanscience.com/ar/articles/detecting-pretraining-data-in-large-language-models-from-a-free-energy-perspective.
فانكوفر
Ke C, Liu Z, Liu Q, Zhuang Y, Zhang J, Huang Z, et al. Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective. https://omanscience.com/ar/articles/detecting-pretraining-data-in-large-language-models-from-a-free-energy-perspective
IEEE
C. Ke, Z. Liu, Q. Liu, Y. Zhuang, J. Zhang, Z. Huang, and S. Wang, "Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective," https://omanscience.com/ar/articles/detecting-pretraining-data-in-large-language-models-from-a-free-energy-perspective.