Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization
(Submitted on 2 Oct 2020 (v1), last revised 6 May 2021 (this version, v4))
Abstract: We study the offline meta-reinforcement learning (OMRL) problem, a paradigm which enables reinforcement learning (RL) algorithms to quickly adapt to unseen tasks without any interactions with the environments, making RL truly practical in many real-world applications. This problem is still not fully understood, for which two major challenges need to be addressed. First, offline RL usually suffers from bootstrapping errors of out-of-distribution state-actions which leads to divergence of value functions. Second, meta-RL requires efficient and robust task inference learned jointly with control policy. In this work, we enforce behavior regularization on learned policy as a general approach to offline RL, combined with a deterministic context encoder for efficient task inference. We propose a novel negative-power distance metric on bounded context embedding space, whose gradients propagation is detached from the Bellman backup. We provide analysis and insight showing that some simple design choices can yield substantial improvements over recent approaches involving meta-RL and distance metric learning. To the best of our knowledge, our method is the first model-free and end-to-end OMRL algorithm, which is computationally efficient and demonstrated to outperform prior algorithms on several meta-RL benchmarks.
Submission history
From: Lanqing Li [view email][v1] Fri, 2 Oct 2020 17:13:39 GMT (7609kb,D)
[v2] Thu, 26 Nov 2020 03:53:09 GMT (6282kb,D)
[v3] Mon, 29 Mar 2021 09:23:00 GMT (15175kb,D)
[v4] Thu, 6 May 2021 09:06:50 GMT (15229kb,D)
Link back to: arXiv, form interface, contact.