Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
(Submitted on 9 Jul 2020 (v1), revised 21 Jul 2020 (this version, v2), latest version 11 Jun 2021 (v4))
Abstract: Model-free deep reinforcement learning (RL) has been successful in a range of challenging domains. However, there are some remaining issues, such as stabilizing the optimization of nonlinear function approximators, preventing error propagation due to the Bellman backup in Q-learning, and efficient exploration. To mitigate these issues, we present SUNRISE, a simple unified ensemble method, which is compatible with various off-policy RL algorithms. SUNRISE integrates three key ingredients: (a) bootstrap with random initialization which improves the stability of the learning process by training a diverse ensemble of agents, (b) weighted Bellman backups, which prevent error propagation in Q-learning by reweighing sample transitions based on uncertainty estimates from the ensembles, and (c) an inference method that selects actions using highest upper-confidence bounds for efficient exploration. Our experiments show that SUNRISE significantly improves the performance of existing off-policy RL algorithms, such as Soft Actor-Critic and Rainbow DQN, for both continuous and discrete control tasks on both low-dimensional and high-dimensional environments. Our training code is available at this https URL
Submission history
From: Kimin Lee [view email][v1] Thu, 9 Jul 2020 17:08:44 GMT (1160kb,D)
[v2] Tue, 21 Jul 2020 20:10:34 GMT (1193kb,D)
[v3] Wed, 9 Jun 2021 22:27:09 GMT (2893kb,D)
[v4] Fri, 11 Jun 2021 21:00:13 GMT (2893kb,D)
Link back to: arXiv, form interface, contact.