Abstract
Models are usually aligned based on their observed outputs, using demonstrations, preference data, or reward signals. These objectives reward responses that look aligned. More capable models may learn to satisfy them without internalizing the intended behavior, for example by faking compliance during training. Such superficial compliance could be harder when the objective is defined on model internals rather than outputs. Therefore, we study probe-guided fine-tuning, using probes that detect undesired properties in model activations as a direct training signal. We evaluate linear and non-linear probes with different numbers of probes per layer across two alignment objectives: harmlessness and honesty. We find that training against probes that do not update during training is an easily exploitable objective, while continuously updated probes substantially reduce harmfulness and improve honesty while preserving utility. Probe-guided fine-tuning achieves better safety-utility trade-offs than DPO and inference-time steering, while being substantially more robust against jailbreak and abliteration attacks. Moreover, the concepts stay linearly encoded after fine-tuning, meaning oversight is not lost by our method. Training against probes thus offers a way to shape what models represent rather than only what they output, which may become increasingly important as models get better at making their outputs look aligned.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Libon, L., Panfilov, A., Rank, B., Chen, X., Geiping, J., & Andriushchenko, M. (2026). Alignment via Training Against Probes Without Losing Monitorability. https://omanscience.com/en/articles/alignment-via-training-against-probes-without-losing-monitorability
MLA 9
Libon, Lena, et al. "Alignment via Training Against Probes Without Losing Monitorability." https://omanscience.com/en/articles/alignment-via-training-against-probes-without-losing-monitorability.
Chicago (author–date)
Libon, Lena, Alexander Panfilov, Ben Rank, Xin Chen, Jonas Geiping, and Maksym Andriushchenko. 2026. "Alignment via Training Against Probes Without Losing Monitorability." https://omanscience.com/en/articles/alignment-via-training-against-probes-without-losing-monitorability.
Harvard
Libon, L., Panfilov, A., Rank, B., Chen, X., Geiping, J. and Andriushchenko, M. (2026) 'Alignment via Training Against Probes Without Losing Monitorability', Available at: https://omanscience.com/en/articles/alignment-via-training-against-probes-without-losing-monitorability.
Vancouver
Libon L, Panfilov A, Rank B, Chen X, Geiping J, Andriushchenko M. Alignment via Training Against Probes Without Losing Monitorability. https://omanscience.com/en/articles/alignment-via-training-against-probes-without-losing-monitorability
IEEE
L. Libon, A. Panfilov, B. Rank, X. Chen, J. Geiping, and M. Andriushchenko, "Alignment via Training Against Probes Without Losing Monitorability," https://omanscience.com/en/articles/alignment-via-training-against-probes-without-losing-monitorability.