Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Local Advantage Actor-Critic for Robust Multi-Agent Deep Reinforcement Learning
(Submitted on 16 Oct 2021 (v1), last revised 19 Dec 2021 (this version, v3))
Abstract: Policy gradient methods have become popular in multi-agent reinforcement learning, but they suffer from high variance due to the presence of environmental stochasticity and exploring agents (i.e., non-stationarity), which is potentially worsened by the difficulty in credit assignment. As a result, there is a need for a method that is not only capable of efficiently solving the above two problems but also robust enough to solve a variety of tasks. To this end, we propose a new multi-agent policy gradient method, called Robust Local Advantage (ROLA) Actor-Critic. ROLA allows each agent to learn an individual action-value function as a local critic as well as ameliorating environment non-stationarity via a novel centralized training approach based on a centralized critic. By using this local critic, each agent calculates a baseline to reduce variance on its policy gradient estimation, which results in an expected advantage action-value over other agents' choices that implicitly improves credit assignment. We evaluate ROLA across diverse benchmarks and show its robustness and effectiveness over a number of state-of-the-art multi-agent policy gradient algorithms.
Submission history
From: Yuchen Xiao [view email][v1] Sat, 16 Oct 2021 19:03:34 GMT (11514kb,D)
[v2] Tue, 2 Nov 2021 03:25:26 GMT (11516kb,D)
[v3] Sun, 19 Dec 2021 19:39:13 GMT (11517kb,D)
Link back to: arXiv, form interface, contact.