We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms

Abstract: We investigate the discounting mismatch in actor-critic algorithm implementations from a representation learning perspective. Theoretically, actor-critic algorithms usually have discounting for both actor and critic, i.e., there is a $\gamma^t$ term in the actor update for the transition observed at time $t$ in a trajectory and the critic is a discounted value function. Practitioners, however, usually ignore the discounting ($\gamma^t$) for the actor while using a discounted critic. We investigate this mismatch in two scenarios. In the first scenario, we consider optimizing an undiscounted objective $(\gamma = 1)$ where $\gamma^t$ disappears naturally $(1^t = 1)$. We then propose to interpret the discounting in critic in terms of a bias-variance-representation trade-off and provide supporting empirical results. In the second scenario, we consider optimizing a discounted objective ($\gamma < 1$) and propose to interpret the omission of the discounting in the actor update from an auxiliary task perspective and provide supporting empirical results.
Comments: AAMAS 2022
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2010.01069 [cs.LG]
  (or arXiv:2010.01069v4 [cs.LG] for this version)

Submission history

From: Shangtong Zhang [view email]
[v1] Fri, 2 Oct 2020 15:51:48 GMT (1195kb,D)
[v2] Sun, 17 Jan 2021 17:59:12 GMT (2821kb,D)
[v3] Tue, 25 May 2021 20:40:55 GMT (2056kb,D)
[v4] Wed, 26 Jan 2022 18:07:28 GMT (4554kb,D)

Link back to: arXiv, form interface, contact.