We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Robotics

Title: Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble

Abstract: Recent advance in deep offline reinforcement learning (RL) has made it possible to train strong robotic agents from offline datasets. However, depending on the quality of the trained agents and the application being considered, it is often desirable to fine-tune such agents via further online interactions. In this paper, we observe that state-action distribution shift may lead to severe bootstrap error during fine-tuning, which destroys the good initial policy obtained via offline RL. To address this issue, we first propose a balanced replay scheme that prioritizes samples encountered online while also encouraging the use of near-on-policy samples from the offline dataset. Furthermore, we leverage multiple Q-functions trained pessimistically offline, thereby preventing overoptimism concerning unfamiliar actions at novel states during the initial training phase. We show that the proposed method improves sample-efficiency and final performance of the fine-tuned robotic agents on various locomotion and manipulation tasks. Our code is available at: this https URL
Comments: CoRL 2021. First two authors contributed equally
Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
Cite as: arXiv:2107.00591 [cs.RO]
  (or arXiv:2107.00591v2 [cs.RO] for this version)

Submission history

From: Younggyo Seo [view email]
[v1] Thu, 1 Jul 2021 16:26:54 GMT (1744kb,D)
[v2] Sat, 30 Oct 2021 04:21:12 GMT (2151kb,D)

Link back to: arXiv, form interface, contact.