We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.AI

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learning

Abstract: Centralized Training for Decentralized Execution, where training is done in a centralized offline fashion, has become a popular solution paradigm in Multi-Agent Reinforcement Learning. Many such methods take the form of actor-critic with state-based critics, since centralized training allows access to the true system state, which can be useful during training despite not being available at execution time. State-based critics have become a common empirical choice, albeit one which has had limited theoretical justification or analysis. In this paper, we show that state-based critics can introduce bias in the policy gradient estimates, potentially undermining the asymptotic guarantees of the algorithm. We also show that, even if the state-based critics do not introduce any bias, they can still result in a larger gradient variance, contrary to the common intuition. Finally, we show the effects of the theories in practice by comparing different forms of centralized critics on a wide range of common benchmarks, and detail how various environmental properties are related to the effectiveness of different types of critics.
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
Journal reference: Thirty-Sixth AAAI Conference on Artificial Intelligence 2022 (AAAI-22)
Cite as: arXiv:2201.01221 [cs.LG]
  (or arXiv:2201.01221v2 [cs.LG] for this version)

Submission history

From: Xueguang Lyu [view email]
[v1] Mon, 3 Jan 2022 14:51:30 GMT (21991kb,D)
[v2] Wed, 25 May 2022 17:55:25 GMT (21991kb,D)

Link back to: arXiv, form interface, contact.