References & Citations
Computer Science > Machine Learning
Title: CASA: Bridging the Gap between Policy Improvement and Policy Evaluation with Conflict Averse Policy Iteration
(Submitted on 9 May 2021 (v1), last revised 25 Feb 2023 (this version, v5))
Abstract: We study the problem of model-free reinforcement learning, which is often solved following the principle of Generalized Policy Iteration (GPI). While GPI is typically an interplay between policy evaluation and policy improvement, most conventional model-free methods assume the independence of the granularity and other details of the GPI steps, despite of the inherent connections between them. In this paper, we present a method that regularizes the inconsistency between policy evaluation and policy improvement, leading to a conflict averse GPI solution with reduced functional approximation error. To this end, we formulate a novel learning paradigm where taking the policy evaluation step is equivalent to some compensation of performing policy improvement, and thus effectively alleviates the gradient conflict between the two GPI steps. We also show that the form of our proposed solution is equivalent to performing entropy-regularized policy improvement and therefore prevents the policy from being trapped into suboptimal solutions. We conduct extensive experiments to evaluate our method on the Arcade Learning Environment (ALE). Empirical results show that our method outperforms several strong baselines in major evaluation domains.
Submission history
From: Haosen Shi [view email][v1] Sun, 9 May 2021 12:45:13 GMT (5337kb,D)
[v2] Wed, 26 May 2021 12:50:54 GMT (6795kb,D)
[v3] Thu, 27 May 2021 16:28:16 GMT (6816kb,D)
[v4] Sat, 11 Jun 2022 07:14:47 GMT (4669kb,D)
[v5] Sat, 25 Feb 2023 12:36:54 GMT (23237kb,D)
Link back to: arXiv, form interface, contact.