We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Real-Time Reinforcement Learning

Abstract: Markov Decision Processes (MDPs), the mathematical framework underlying most algorithms in Reinforcement Learning (RL), are often used in a way that wrongfully assumes that the state of an agent's environment does not change during action selection. As RL systems based on MDPs begin to find application in real-world safety critical situations, this mismatch between the assumptions underlying classical MDPs and the reality of real-time computation may lead to undesirable outcomes. In this paper, we introduce a new framework, in which states and actions evolve simultaneously and show how it is related to the classical MDP formulation. We analyze existing algorithms under the new real-time formulation and show why they are suboptimal when used in real-time. We then use those insights to create a new algorithm Real-Time Actor-Critic (RTAC) that outperforms the existing state-of-the-art continuous control algorithm Soft Actor-Critic both in real-time and non-real-time settings. Code and videos can be found at this https URL
Comments: Neural Information Processing Systems (2019)
Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as: arXiv:1911.04448 [cs.LG]
  (or arXiv:1911.04448v4 [cs.LG] for this version)

Submission history

From: Simon Ramstedt [view email]
[v1] Mon, 11 Nov 2019 18:52:04 GMT (5351kb,D)
[v2] Wed, 13 Nov 2019 18:56:26 GMT (4132kb,D)
[v3] Thu, 14 Nov 2019 18:54:06 GMT (4131kb,D)
[v4] Thu, 12 Dec 2019 08:46:32 GMT (4131kb,D)

Link back to: arXiv, form interface, contact.