We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.MA

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Multiagent Systems

Title: Distributed Policy Gradient with Variance Reduction in Multi-Agent Reinforcement Learning

Abstract: This paper studies a distributed policy gradient in collaborative multi-agent reinforcement learning (MARL), where agents over a communication network aim to find the optimal policy to maximize the average of all agents' local returns. Due to the non-concave performance function of policy gradient, the existing distributed stochastic optimization methods for convex problems cannot be directly used for policy gradient in MARL. This paper proposes a distributed policy gradient with variance reduction and gradient tracking to address the high variances of policy gradient, and utilizes importance weight to solve the non-stationary problem in the sampling process. We then provide an upper bound on the mean-squared stationary gap, which depends on the number of iterations, the mini-batch size, the epoch size, the problem parameters, and the network topology. We further establish the sample and communication complexity to obtain an $\epsilon$-approximate stationary point. Numerical experiments on the control problem in MARL are performed to validate the effectiveness of the proposed algorithm.
Subjects: Multiagent Systems (cs.MA); Machine Learning (cs.LG); Optimization and Control (math.OC)
Cite as: arXiv:2111.12961 [cs.MA]
  (or arXiv:2111.12961v2 [cs.MA] for this version)

Submission history

From: Xiaoxiao Zhao [view email]
[v1] Thu, 25 Nov 2021 08:07:30 GMT (16455kb,D)
[v2] Tue, 30 Nov 2021 06:26:02 GMT (16455kb,D)

Link back to: arXiv, form interface, contact.