Abstract
Principled exploration in reinforcement learning requires an agent to quantify its epistemic uncertainty and act to resolve it. Uncertainty over the value function provides a natural signal for exploration, yet existing deep approximations remain brittle and perform inconsistently. The central challenge is therefore to scale these ideas robustly. We conduct a systematic empirical study of how epistemic uncertainty is represented, propagated, and optimized in deep epistemic value functions, and uncover distinct failure modes along each of these axes. These findings motivate DEVOTE, a model-free reinforcement learning algorithm that controls how uncertainty generalizes beyond observed data, stabilizes its temporal propagation, and preserves adaptation to the resulting non-stationary exploration objective. Across reward-free exploration and challenging continuous-control tasks, DEVOTE reaches novel states more effectively and achieves higher task return than strong model-free and model-based exploration baselines. These results provide evidence that deep epistemic value functions are a promising path toward scalable, principled exploration.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Diaz-Bone, L., Bagatella, M., Hübotter, J., & Krause, A. (2026). Deep Epistemic Value Functions for Optimistic Exploration. https://omanscience.com/en/articles/deep-epistemic-value-functions-for-optimistic-exploration
MLA 9
Diaz-Bone, Leander, et al. "Deep Epistemic Value Functions for Optimistic Exploration." https://omanscience.com/en/articles/deep-epistemic-value-functions-for-optimistic-exploration.
Chicago (author–date)
Diaz-Bone, Leander, Marco Bagatella, Jonas Hübotter, and Andreas Krause. 2026. "Deep Epistemic Value Functions for Optimistic Exploration." https://omanscience.com/en/articles/deep-epistemic-value-functions-for-optimistic-exploration.
Harvard
Diaz-Bone, L., Bagatella, M., Hübotter, J. and Krause, A. (2026) 'Deep Epistemic Value Functions for Optimistic Exploration', Available at: https://omanscience.com/en/articles/deep-epistemic-value-functions-for-optimistic-exploration.
Vancouver
Diaz-Bone L, Bagatella M, Hübotter J, Krause A. Deep Epistemic Value Functions for Optimistic Exploration. https://omanscience.com/en/articles/deep-epistemic-value-functions-for-optimistic-exploration
IEEE
L. Diaz-Bone, M. Bagatella, J. Hübotter, and A. Krause, "Deep Epistemic Value Functions for Optimistic Exploration," https://omanscience.com/en/articles/deep-epistemic-value-functions-for-optimistic-exploration.