Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methods
(Submitted on 6 Nov 2021 (v1), last revised 27 Jan 2022 (this version, v6))
Abstract: In reinforcement learning, continuous time is often discretized by a time scale $\delta$, to which the resulting performance is known to be highly sensitive. In this work, we seek to find a $\delta$-invariant algorithm for policy gradient (PG) methods, which performs well regardless of the value of $\delta$. We first identify the underlying reasons that cause PG methods to fail as $\delta \to 0$, proving that the variance of the PG estimator can diverge to infinity in stochastic environments under a certain assumption of stochasticity. While durative actions or action repetition can be employed to have $\delta$-invariance, previous action repetition methods cannot immediately react to unexpected situations in stochastic environments. We thus propose a novel $\delta$-invariant method named Safe Action Repetition (SAR) applicable to any existing PG algorithm. SAR can handle the stochasticity of environments by adaptively reacting to changes in states during action repetition. We empirically show that our method is not only $\delta$-invariant but also robust to stochasticity, outperforming previous $\delta$-invariant approaches on eight MuJoCo environments with both deterministic and stochastic settings. Our code is available at this https URL
Submission history
From: Seohong Park [view email][v1] Sat, 6 Nov 2021 19:17:24 GMT (4122kb,D)
[v2] Tue, 23 Nov 2021 07:27:19 GMT (4122kb,D)
[v3] Wed, 1 Dec 2021 01:35:23 GMT (4122kb,D)
[v4] Thu, 23 Dec 2021 10:46:45 GMT (4122kb,D)
[v5] Sat, 22 Jan 2022 12:06:33 GMT (4122kb,D)
[v6] Thu, 27 Jan 2022 16:20:49 GMT (4122kb,D)
Link back to: arXiv, form interface, contact.