References & Citations
Computer Science > Machine Learning
Title: Distillation Policy Optimization
(Submitted on 1 Feb 2023 (v1), last revised 27 Sep 2023 (this version, v5))
Abstract: While on-policy algorithms are known for their stability, they often demand a substantial number of samples. In contrast, off-policy algorithms, which leverage past experiences, are considered sample-efficient but tend to exhibit instability. Can we develop an algorithm that harnesses the benefits of off-policy data while maintaining stable learning? In this paper, we introduce an actor-critic learning framework that harmonizes two data sources for both evaluation and control, facilitating rapid learning and adaptable integration with on-policy algorithms. This framework incorporates variance reduction mechanisms, including a unified advantage estimator (UAE) and a residual baseline, improving the efficacy of both on- and off-policy learning. Our empirical results showcase substantial enhancements in sample efficiency for on-policy algorithms, effectively bridging the gap to the off-policy approaches. It demonstrates the promise of our approach as a novel learning paradigm.
Submission history
From: Jianfei Ma [view email][v1] Wed, 1 Feb 2023 15:59:57 GMT (4960kb,D)
[v2] Thu, 2 Feb 2023 07:30:13 GMT (4959kb,D)
[v3] Wed, 17 May 2023 01:22:36 GMT (0kb,I)
[v4] Fri, 26 May 2023 03:16:32 GMT (5664kb,D)
[v5] Wed, 27 Sep 2023 05:06:28 GMT (6331kb,D)
Link back to: arXiv, form interface, contact.