Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: MARL with General Utilities via Decentralized Shadow Reward Actor-Critic
(Submitted on 29 May 2021 (v1), last revised 24 Jun 2021 (this version, v2))
Abstract: We posit a new mechanism for cooperation in multi-agent reinforcement learning (MARL) based upon any nonlinear function of the team's long-term state-action occupancy measure, i.e., a \emph{general utility}. This subsumes the cumulative return but also allows one to incorporate risk-sensitivity, exploration, and priors. % We derive the {\bf D}ecentralized {\bf S}hadow Reward {\bf A}ctor-{\bf C}ritic (DSAC) in which agents alternate between policy evaluation (critic), weighted averaging with neighbors (information mixing), and local gradient updates for their policy parameters (actor). DSAC augments the classic critic step by requiring agents to (i) estimate their local occupancy measure in order to (ii) estimate the derivative of the local utility with respect to their occupancy measure, i.e., the "shadow reward". DSAC converges to $\epsilon$-stationarity in $\mathcal{O}(1/\epsilon^{2.5})$ (Theorem \ref{theorem:final}) or faster $\mathcal{O}(1/\epsilon^{2})$ (Corollary \ref{corollary:communication}) steps with high probability, depending on the amount of communications. We further establish the non-existence of spurious stationary points for this problem, that is, DSAC finds the globally optimal policy (Corollary \ref{corollary:global}). Experiments demonstrate the merits of goals beyond the cumulative return in cooperative MARL.
Submission history
From: Amrit Singh Bedi [view email][v1] Sat, 29 May 2021 19:05:48 GMT (7457kb,D)
[v2] Thu, 24 Jun 2021 17:20:27 GMT (7469kb,D)
Link back to: arXiv, form interface, contact.