Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Gradient play in stochastic games: stationary points, convergence, and sample complexity
(Submitted on 1 Jun 2021 (v1), last revised 6 Dec 2023 (this version, v5))
Abstract: We study the performance of the gradient play algorithm for stochastic games (SGs), where each agent tries to maximize its own total discounted reward by making decisions independently based on current state information which is shared between agents. Policies are directly parameterized by the probability of choosing a certain action at a given state. We show that Nash equilibria (NEs) and first-order stationary policies are equivalent in this setting, and give a local convergence rate around strict NEs. Further, for a subclass of SGs called Markov potential games (which includes the setting with identical rewards as an important special case), we design a sample-based reinforcement learning algorithm and give a non-asymptotic global convergence rate analysis for both exact gradient play and our sample-based learning algorithm. Our result shows that the number of iterations to reach an $\epsilon$-NE scales linearly, instead of exponentially, with the number of agents. Local geometry and local stability are also considered, where we prove that strict NEs are local maxima of the total potential function and fully-mixed NEs are saddle points.
Submission history
From: Runyu Zhang Ms. [view email][v1] Tue, 1 Jun 2021 03:03:45 GMT (135kb,D)
[v2] Thu, 17 Jun 2021 19:26:45 GMT (130kb,D)
[v3] Fri, 15 Oct 2021 21:24:46 GMT (820kb,D)
[v4] Mon, 6 Dec 2021 23:53:18 GMT (585kb,D)
[v5] Wed, 6 Dec 2023 21:33:55 GMT (1227kb,D)
Link back to: arXiv, form interface, contact.