Current browse context:
stat
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Learning to Reach Goals via Iterated Supervised Learning
(Submitted on 12 Dec 2019 (v1), last revised 2 Oct 2020 (this version, v4))
Abstract: Current reinforcement learning (RL) algorithms can be brittle and difficult to use, especially when learning goal-reaching behaviors from sparse rewards. Although supervised imitation learning provides a simple and stable alternative, it requires access to demonstrations from a human supervisor. In this paper, we study RL algorithms that use imitation learning to acquire goal reaching policies from scratch, without the need for expert demonstrations or a value function. In lieu of demonstrations, we leverage the property that any trajectory is a successful demonstration for reaching the final state in that same trajectory. We propose a simple algorithm in which an agent continually relabels and imitates the trajectories it generates to progressively learn goal-reaching behaviors from scratch. Each iteration, the agent collects new trajectories using the latest policy, and maximizes the likelihood of the actions along these trajectories under the goal that was actually reached, so as to improve the policy. We formally show that this iterated supervised learning procedure optimizes a bound on the RL objective, derive performance bounds of the learned policy, and empirically demonstrate improved goal-reaching performance and robustness over current RL algorithms in several benchmark tasks.
Submission history
From: Dibya Ghosh [view email][v1] Thu, 12 Dec 2019 17:26:47 GMT (2110kb,D)
[v2] Fri, 13 Dec 2019 01:42:38 GMT (2104kb,D)
[v3] Wed, 10 Jun 2020 17:22:46 GMT (4353kb,D)
[v4] Fri, 2 Oct 2020 19:49:10 GMT (3539kb,D)
Link back to: arXiv, form interface, contact.