Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Learning to Play No-Press Diplomacy with Best Response Policy Iteration
(Submitted on 8 Jun 2020 (v1), last revised 4 Jan 2022 (this version, v4))
Abstract: Recent advances in deep reinforcement learning (RL) have led to considerable progress in many 2-player zero-sum games, such as Go, Poker and Starcraft. The purely adversarial nature of such games allows for conceptually simple and principled application of RL methods. However real-world settings are many-agent, and agent interactions are complex mixtures of common-interest and competitive aspects. We consider Diplomacy, a 7-player board game designed to accentuate dilemmas resulting from many-agent interactions. It also features a large combinatorial action space and simultaneous moves, which are challenging for RL algorithms. We propose a simple yet effective approximate best response operator, designed to handle large combinatorial action spaces and simultaneous moves. We also introduce a family of policy iteration methods that approximate fictitious play. With these methods, we successfully apply RL to Diplomacy: we show that our agents convincingly outperform the previous state-of-the-art, and game theoretic equilibrium analysis shows that the new process yields consistent improvements.
Submission history
From: Thomas William Anthony [view email][v1] Mon, 8 Jun 2020 14:33:31 GMT (662kb,D)
[v2] Wed, 17 Jun 2020 21:34:42 GMT (664kb,D)
[v3] Wed, 26 Aug 2020 17:01:22 GMT (664kb,D)
[v4] Tue, 4 Jan 2022 15:11:59 GMT (662kb,D)
Link back to: arXiv, form interface, contact.