References & Citations
Computer Science > Artificial Intelligence
Title: Score vs. Winrate in Score-Based Games: which Reward for Reinforcement Learning?
(Submitted on 31 Jan 2022 (v1), last revised 9 Jan 2023 (this version, v2))
Abstract: In the last years, the DeepMind algorithm AlphaZero has become the state of the art to efficiently tackle perfect information two-player zero-sum games with a win/lose outcome. However, when the win/lose outcome is decided by a final score difference, AlphaZero may play score-suboptimal moves because all winning final positions are equivalent from the win/lose outcome perspective. This can be an issue, for instance when used for teaching, or when trying to understand whether there is a better move. Moreover, there is the theoretical quest for the perfect game. A naive approach would be training an AlphaZero-like agent to predict score differences instead of win/lose outcomes. Since the game of Go is deterministic, this should as well produce an outcome-optimal play. However, it is a folklore belief that "this does not work".
In this paper, we first provide empirical evidence for this belief. We then give a theoretical interpretation of this suboptimality in general perfect information two-player zero-sum game where the complexity of a game like Go is replaced by the randomness of the environment. We show that an outcome-optimal policy has a different preference for uncertainty when it is winning or losing. In particular, when in a losing state, an outcome-optimal agent chooses actions leading to a higher score variance. We then posit that when approximation is involved, a deterministic game behaves like a nondeterministic game, where the score variance is modeled by how uncertain the position is. We validate this hypothesis in AlphaZero-like software with a human expert.
Submission history
From: Maurizio Parton [view email][v1] Mon, 31 Jan 2022 12:38:02 GMT (771kb,D)
[v2] Mon, 9 Jan 2023 15:24:15 GMT (1232kb,D)
Link back to: arXiv, form interface, contact.