Abstract
Generative actors are transforming offline reinforcement learning (RL) by enabling expressive policy classes that model complex action distributions. However, this expressiveness also exposes a key challenge in heterogeneous datasets: generative policies can reproduce unreliable action modes whose return distributions exhibit high variance, occasionally yielding high returns by chance but lacking consistency. Consequently, maximizing the expected $Q$-value alone is insufficient for identifying reliable actions. We propose VAN-Flow (Variance-Averse $n$-step Flow), a framework that promotes reliable actions in generative offline RL. VAN-Flow combines (i) a categorical distributional critic, (ii) a variance-averse expectation operator that smoothly reweights atom probabilities to favor actions with both high returns and low dispersion, and (iii) a flow-matching generative actor guided via rejection sampling. Unlike CVaR or mean-variance objectives, the operator redistributes probability mass over the categorical return distribution without hard truncation or auxiliary penalty terms. Across more than 40 tasks from D4RL and OGBench, VAN-Flow consistently outperforms strong baselines, with the largest gains in long-horizon and high-variance regimes where reliable action selection becomes critical.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Kang, G., & Kwon, M. (2026). Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments. https://omanscience.com/en/articles/variance-averse-n-step-offline-reinforcement-learning-for-sparse-long-horizon-environments
MLA 9
Kang, Guhyeon, and Minhae Kwon. "Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments." https://omanscience.com/en/articles/variance-averse-n-step-offline-reinforcement-learning-for-sparse-long-horizon-environments.
Chicago (author–date)
Kang, Guhyeon, and Minhae Kwon. 2026. "Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments." https://omanscience.com/en/articles/variance-averse-n-step-offline-reinforcement-learning-for-sparse-long-horizon-environments.
Harvard
Kang, G. and Kwon, M. (2026) 'Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments', Available at: https://omanscience.com/en/articles/variance-averse-n-step-offline-reinforcement-learning-for-sparse-long-horizon-environments.
Vancouver
Kang G, Kwon M. Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments. https://omanscience.com/en/articles/variance-averse-n-step-offline-reinforcement-learning-for-sparse-long-horizon-environments
IEEE
G. Kang, and M. Kwon, "Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments," https://omanscience.com/en/articles/variance-averse-n-step-offline-reinforcement-learning-for-sparse-long-horizon-environments.