We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewards

Abstract: Constrained reinforcement learning involves multiple rewards that must individually accumulate to given thresholds. In this class of problems, we show a simple example in which the desired optimal policy cannot be induced by any linear combination of rewards. Hence, there exist constrained reinforcement learning problems for which neither regularized nor classical primal-dual methods yield optimal policies. This work addresses this shortcoming by augmenting the state with Lagrange multipliers and reinterpreting primal-dual methods as the portion of the dynamics that drives the multipliers evolution. This approach provides a systematic state augmentation procedure that is guaranteed to solve reinforcement learning problems with constraints. Thus, while primal-dual methods can fail at finding optimal policies, running the dual dynamics while executing the augmented policy yields an algorithm that provably samples actions from the optimal policy.
Subjects: Machine Learning (cs.LG); Robotics (cs.RO); Optimization and Control (math.OC)
Cite as: arXiv:2102.11941 [cs.LG]
  (or arXiv:2102.11941v1 [cs.LG] for this version)

Submission history

From: Miguel Calvo-Fullana [view email]
[v1] Tue, 23 Feb 2021 21:07:35 GMT (798kb,D)

Link back to: arXiv, form interface, contact.