Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: The Role of Pretrained Representations for the OOD Generalization of Reinforcement Learning Agents
(Submitted on 12 Jul 2021 (v1), last revised 16 Apr 2022 (this version, v4))
Abstract: Building sample-efficient agents that generalize out-of-distribution (OOD) in real-world settings remains a fundamental unsolved problem on the path towards achieving higher-level cognition. One particularly promising approach is to begin with low-dimensional, pretrained representations of our world, which should facilitate efficient downstream learning and generalization. By training 240 representations and over 10,000 reinforcement learning (RL) policies on a simulated robotic setup, we evaluate to what extent different properties of pretrained VAE-based representations affect the OOD generalization of downstream agents. We observe that many agents are surprisingly robust to realistic distribution shifts, including the challenging sim-to-real case. In addition, we find that the generalization performance of a simple downstream proxy task reliably predicts the generalization performance of our RL agents under a wide range of OOD settings. Such proxy tasks can thus be used to select pretrained representations that will lead to agents that generalize.
Submission history
From: Andrea Dittadi [view email][v1] Mon, 12 Jul 2021 18:49:48 GMT (11776kb,D)
[v2] Mon, 18 Oct 2021 08:23:03 GMT (11840kb,D)
[v3] Thu, 3 Mar 2022 09:44:13 GMT (11936kb,D)
[v4] Sat, 16 Apr 2022 14:03:31 GMT (11936kb,D)
Link back to: arXiv, form interface, contact.