Current browse context:
cs.AI
Change to browse by:
References & Citations
Computer Science > Artificial Intelligence
Title: Alternative Function Approximation Parameterizations for Solving Games: An Analysis of $f$-Regression Counterfactual Regret Minimization
(Submitted on 6 Dec 2019 (v1), last revised 1 May 2020 (this version, v5))
Abstract: Function approximation is a powerful approach for structuring large decision problems that has facilitated great achievements in the areas of reinforcement learning and game playing. Regression counterfactual regret minimization (RCFR) is a simple algorithm for approximately solving imperfect information games with normalized rectified linear unit (ReLU) parameterized policies. In contrast, the more conventional softmax parameterization is standard in the field of reinforcement learning and yields a regret bound with a better dependence on the number of actions. We derive approximation error-aware regret bounds for $(\Phi, f)$-regret matching, which applies to a general class of link functions and regret objectives. These bounds recover a tighter bound for RCFR and provide a theoretical justification for RCFR implementations with alternative policy parameterizations ($f$-RCFR), including softmax. We provide exploitability bounds for $f$-RCFR with the polynomial and exponential link functions in zero-sum imperfect information games and examine empirically how the link function interacts with the severity of the approximation. We find that the previously studied ReLU parameterization performs better when the approximation error is small while the softmax parameterization can perform better when the approximation error is large.
Submission history
From: Dustin Morrill [view email][v1] Fri, 6 Dec 2019 03:32:29 GMT (103kb,D)
[v2] Fri, 28 Feb 2020 20:40:46 GMT (122kb,D)
[v3] Sat, 18 Apr 2020 19:30:16 GMT (477kb,D)
[v4] Mon, 27 Apr 2020 20:33:50 GMT (477kb,D)
[v5] Fri, 1 May 2020 15:01:30 GMT (477kb,D)
Link back to: arXiv, form interface, contact.