Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Energy-based Surprise Minimization for Multi-Agent Value Factorization
(Submitted on 16 Sep 2020 (v1), last revised 18 Jan 2021 (this version, v4))
Abstract: Multi-Agent Reinforcement Learning (MARL) has demonstrated significant success in training decentralised policies in a centralised manner by making use of value factorization methods. However, addressing surprise across spurious states and approximation bias remain open problems for multi-agent settings. Towards this goal, we introduce the Energy-based MIXer (EMIX), an algorithm which minimizes surprise utilizing the energy across agents. Our contributions are threefold; (1) EMIX introduces a novel surprise minimization technique across multiple agents in the case of multi-agent partially-observable settings. (2) EMIX highlights a practical use of energy functions in MARL with theoretical guarantees and experiment validations of the energy operator. Lastly, (3) EMIX extends Maxmin Q-learning for addressing overestimation bias across agents in MARL. In a study of challenging StarCraft II micromanagement scenarios, EMIX demonstrates consistent stable performance for multiagent surprise minimization. Moreover, our ablation study highlights the necessity of the energy-based scheme and the need for elimination of overestimation bias in MARL. Our implementation of EMIX can be found at karush17.github.io/emix-web/.
Submission history
From: Karush Suri [view email][v1] Wed, 16 Sep 2020 19:42:42 GMT (27581kb,D)
[v2] Mon, 28 Sep 2020 16:26:43 GMT (27581kb,D)
[v3] Mon, 5 Oct 2020 16:10:26 GMT (27494kb,D)
[v4] Mon, 18 Jan 2021 03:06:32 GMT (27394kb,D)
Link back to: arXiv, form interface, contact.