Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Real-Time Reinforcement Learning
(Submitted on 11 Nov 2019 (v1), last revised 12 Dec 2019 (this version, v4))
Abstract: Markov Decision Processes (MDPs), the mathematical framework underlying most algorithms in Reinforcement Learning (RL), are often used in a way that wrongfully assumes that the state of an agent's environment does not change during action selection. As RL systems based on MDPs begin to find application in real-world safety critical situations, this mismatch between the assumptions underlying classical MDPs and the reality of real-time computation may lead to undesirable outcomes. In this paper, we introduce a new framework, in which states and actions evolve simultaneously and show how it is related to the classical MDP formulation. We analyze existing algorithms under the new real-time formulation and show why they are suboptimal when used in real-time. We then use those insights to create a new algorithm Real-Time Actor-Critic (RTAC) that outperforms the existing state-of-the-art continuous control algorithm Soft Actor-Critic both in real-time and non-real-time settings. Code and videos can be found at this https URL
Submission history
From: Simon Ramstedt [view email][v1] Mon, 11 Nov 2019 18:52:04 GMT (5351kb,D)
[v2] Wed, 13 Nov 2019 18:56:26 GMT (4132kb,D)
[v3] Thu, 14 Nov 2019 18:54:06 GMT (4131kb,D)
[v4] Thu, 12 Dec 2019 08:46:32 GMT (4131kb,D)
Link back to: arXiv, form interface, contact.