Abstract
Despite its widespread use, Proximal Policy Optimization with clipping (PPO-Clip) remains difficult to tune, and the interactions among critic learning, clipping, and rollout reuse remain incompletely understood. We develop a \emph{non-asymptotic} analysis of PPO-Clip as a \emph{closed-loop actor--critic} system. It captures actor--critic coupling, nonsmooth probability-ratio clipping, finite-batch reuse, and predictable early stopping under explicit coverage and critic regularity assumptions, using raw GAE and Monte Carlo critic targets. Our synchronous and asynchronous guarantees jointly characterize policy stationarity and the tracking accuracy of the learned critic, with explicit dependence on algorithmic parameters. A sufficient coupling condition gives optimization, critic tracking, clipping, and finite-batch errors a common amplification bound. The asynchronous result also requires a delay-dependent critic stepsize restriction; violating these conditions does not establish divergence. For finite layered MDPs with tabular critics, a uniform bound on the actual clipped-gradient class replaces complete-trajectory counting. A verified growing-horizon family has polynomial sample complexity, and a two-time-scale schedule gives $O(T^{-2/5})$ stationarity and critic-tracking bounds with explicit fresh-rollout accounting. These results together advance our understanding about PPO and provide theoretical guidance in tuning.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Su, J., Liu, M., Zhang, Y., & Wu, C. (2026). A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping. https://omanscience.com/en/articles/a-closed-loop-non-asymptotic-convergence-analysis-of-ppo-with-learned-critics-and-clipping
MLA 9
Su, Junwei, et al. "A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping." https://omanscience.com/en/articles/a-closed-loop-non-asymptotic-convergence-analysis-of-ppo-with-learned-critics-and-clipping.
Chicago (author–date)
Su, Junwei, Mengfan Liu, Yanyong Zhang, and Chuan Wu. 2026. "A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping." https://omanscience.com/en/articles/a-closed-loop-non-asymptotic-convergence-analysis-of-ppo-with-learned-critics-and-clipping.
Harvard
Su, J., Liu, M., Zhang, Y. and Wu, C. (2026) 'A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping', Available at: https://omanscience.com/en/articles/a-closed-loop-non-asymptotic-convergence-analysis-of-ppo-with-learned-critics-and-clipping.
Vancouver
Su J, Liu M, Zhang Y, Wu C. A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping. https://omanscience.com/en/articles/a-closed-loop-non-asymptotic-convergence-analysis-of-ppo-with-learned-critics-and-clipping
IEEE
J. Su, M. Liu, Y. Zhang, and C. Wu, "A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping," https://omanscience.com/en/articles/a-closed-loop-non-asymptotic-convergence-analysis-of-ppo-with-learned-critics-and-clipping.