References & Citations
Computer Science > Artificial Intelligence
Title: An Analytical Update Rule for General Policy Optimization
(Submitted on 3 Dec 2021 (v1), last revised 15 Jul 2022 (this version, v4))
Abstract: We present an analytical policy update rule that is independent of parametric function approximators. The policy update rule is suitable for optimizing general stochastic policies and has a monotonic improvement guarantee. It is derived from a closed-form solution to trust-region optimization using calculus of variation, following a new theoretical result that tightens existing bounds for policy improvement using trust-region methods. The update rule builds a connection between policy search methods and value function methods. Moreover, off-policy reinforcement learning algorithms can be derived from the update rule since it does not need to compute integration over on-policy states. In addition, the update rule extends immediately to cooperative multi-agent systems when policy updates are performed by one agent at a time.
Submission history
From: Hepeng Li [view email][v1] Fri, 3 Dec 2021 17:50:11 GMT (585kb)
[v2] Wed, 19 Jan 2022 13:34:33 GMT (333kb,D)
[v3] Sat, 14 May 2022 20:10:44 GMT (529kb,D)
[v4] Fri, 15 Jul 2022 05:22:39 GMT (17555kb,D)
Link back to: arXiv, form interface, contact.