Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Correcting Experience Replay for Multi-Agent Communication
(Submitted on 2 Oct 2020 (v1), last revised 28 Feb 2021 (this version, v2))
Abstract: We consider the problem of learning to communicate using multi-agent reinforcement learning (MARL). A common approach is to learn off-policy, using data sampled from a replay buffer. However, messages received in the past may not accurately reflect the current communication policy of each agent, and this complicates learning. We therefore introduce a 'communication correction' which accounts for the non-stationarity of observed communication induced by multi-agent learning. It works by relabelling the received message to make it likely under the communicator's current policy, and thus be a better reflection of the receiver's current environment. To account for cases in which agents are both senders and receivers, we introduce an ordered relabelling scheme. Our correction is computationally efficient and can be integrated with a range of off-policy algorithms. We find in our experiments that it substantially improves the ability of communicating MARL systems to learn across a variety of cooperative and competitive tasks.
Submission history
From: Sanjeevan Ahilan [view email][v1] Fri, 2 Oct 2020 20:49:24 GMT (2304kb,D)
[v2] Sun, 28 Feb 2021 22:42:12 GMT (5821kb,D)
Link back to: arXiv, form interface, contact.