Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation
(Submitted on 5 Jan 2022 (v1), last revised 3 May 2022 (this version, v3))
Abstract: In model-free deep reinforcement learning (RL) algorithms, using noisy value estimates to supervise policy evaluation and optimization is detrimental to the sample efficiency. As this noise is heteroscedastic, its effects can be mitigated using uncertainty-based weights in the optimization process. Previous methods rely on sampled ensembles, which do not capture all aspects of uncertainty. We provide a systematic analysis of the sources of uncertainty in the noisy supervision that occurs in RL, and introduce inverse-variance RL, a Bayesian framework which combines probabilistic ensembles and Batch Inverse Variance weighting. We propose a method whereby two complementary uncertainty estimation methods account for both the Q-value and the environment stochasticity to better mitigate the negative impacts of noisy supervision. Our results show significant improvement in terms of sample efficiency on discrete and continuous control tasks.
Submission history
From: Vincent Mai [view email][v1] Wed, 5 Jan 2022 15:46:06 GMT (9636kb,D)
[v2] Wed, 2 Mar 2022 20:54:09 GMT (9637kb,D)
[v3] Tue, 3 May 2022 14:08:13 GMT (9637kb,D)
Link back to: arXiv, form interface, contact.