Current browse context:
cs.RO
Change to browse by:
References & Citations
Computer Science > Robotics
Title: Grounding Language with Visual Affordances over Unstructured Data
(Submitted on 4 Oct 2022 (v1), last revised 8 Mar 2023 (this version, v3))
Abstract: Recent works have shown that Large Language Models (LLMs) can be applied to ground natural language to a wide variety of robot skills. However, in practice, learning multi-task, language-conditioned robotic skills typically requires large-scale data collection and frequent human intervention to reset the environment or help correcting the current policies. In this work, we propose a novel approach to efficiently learn general-purpose language-conditioned robot skills from unstructured, offline and reset-free data in the real world by exploiting a self-supervised visuo-lingual affordance model, which requires annotating as little as 1% of the total data with language. We evaluate our method in extensive experiments both in simulated and real-world robotic tasks, achieving state-of-the-art performance on the challenging CALVIN benchmark and learning over 25 distinct visuomotor manipulation tasks with a single policy in the real world. We find that when paired with LLMs to break down abstract natural language instructions into subgoals via few-shot prompting, our method is capable of completing long-horizon, multi-tier tasks in the real world, while requiring an order of magnitude less data than previous approaches. Code and videos are available at this http URL
Submission history
From: Oier Mees [view email][v1] Tue, 4 Oct 2022 21:16:48 GMT (10123kb,D)
[v2] Mon, 10 Oct 2022 09:00:57 GMT (10950kb,D)
[v3] Wed, 8 Mar 2023 11:00:55 GMT (10664kb,D)
Link back to: arXiv, form interface, contact.