We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning

Abstract: Linear value decomposition is a widely-used structure to scale up multi-agent Q-learning algorithms in cooperative settings. To develop a deeper understanding of this popular technique, this paper provides the first theoretical analysis to characterize its internal mechanism. Our main results reveal two novel insights: (1) Linear value decomposition structure implicitly implements a classical credit assignment called difference rewards. (2) This implicit credit assignment requires on-policy data distribution to achieve numerical stability. In the empirical study, our experiments also demonstrate that most deep multi-agent Q-learning algorithms using linear value decomposition structure cannot efficiently utilize off-policy samples.
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA); Machine Learning (stat.ML)
Cite as: arXiv:2006.00587 [cs.LG]
  (or arXiv:2006.00587v1 [cs.LG] for this version)

Submission history

From: Zhizhou Ren [view email]
[v1] Sun, 31 May 2020 19:14:03 GMT (290kb,D)
[v2] Tue, 23 Jun 2020 15:11:41 GMT (386kb,D)
[v3] Mon, 5 Oct 2020 15:24:13 GMT (1160kb,D)
[v4] Wed, 29 Sep 2021 11:07:05 GMT (1090kb,D)
[v5] Sun, 31 Oct 2021 06:21:48 GMT (1128kb,D)

Link back to: arXiv, form interface, contact.