Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Multi-task Batch Reinforcement Learning with Metric Learning
(Submitted on 25 Sep 2019 (v1), last revised 23 Oct 2020 (this version, v6))
Abstract: We tackle the Multi-task Batch Reinforcement Learning problem. Given multiple datasets collected from different tasks, we train a multi-task policy to perform well in unseen tasks sampled from the same distribution. The task identities of the unseen tasks are not provided. To perform well, the policy must infer the task identity from collected transitions by modelling its dependency on states, actions and rewards. Because the different datasets may have state-action distributions with large divergence, the task inference module can learn to ignore the rewards and spuriously correlate $\textit{only}$ state-action pairs to the task identity, leading to poor test time performance. To robustify task inference, we propose a novel application of the triplet loss. To mine hard negative examples, we relabel the transitions from the training tasks by approximating their reward functions. When we allow further training on the unseen tasks, using the trained policy as an initialization leads to significantly faster convergence compared to randomly initialized policies (up to $80\%$ improvement and across 5 different Mujoco task distributions). We name our method $\textbf{MBML}$ ($\textbf{M}\text{ulti-task}$ $\textbf{B}\text{atch}$ RL with $\textbf{M}\text{etric}$ $\textbf{L}\text{earning}$).
Submission history
From: Jiachen Li [view email][v1] Wed, 25 Sep 2019 09:49:01 GMT (3125kb,D)
[v2] Tue, 26 Nov 2019 20:57:44 GMT (3137kb,D)
[v3] Thu, 11 Jun 2020 08:31:42 GMT (1728kb,D)
[v4] Tue, 29 Sep 2020 03:58:33 GMT (2706kb,D)
[v5] Thu, 1 Oct 2020 20:12:10 GMT (2711kb,D)
[v6] Fri, 23 Oct 2020 20:01:46 GMT (2712kb,D)
Link back to: arXiv, form interface, contact.