Current browse context:
cs.RO
Change to browse by:
References & Citations
Computer Science > Robotics
Title: Translating Natural Language Instructions to Computer Programs for Robot Manipulation
(Submitted on 26 Dec 2020 (v1), last revised 20 Mar 2021 (this version, v2))
Abstract: It is highly desirable for robots that work alongside humans to be able to understand instructions in natural language. Existing language conditioned imitation learning models directly predict the actuator commands from the image observation and the instruction text. Rather than directly predicting actuator commands, we propose translating the natural language instruction to a Python function which queries the scene by accessing the output of the object detector and controls the robot to perform the specified task. This enables the use of non-differentiable modules such as a constraint solver when computing commands to the robot. Moreover, the labels in this setup are significantly more informative computer programs that capture the intent of the expert rather than teleoperated demonstrations. We show that the proposed method performs better than training a neural network to directly predict the robot actions.
Submission history
From: Sagar Gubbi Venkatesh [view email][v1] Sat, 26 Dec 2020 07:57:55 GMT (2134kb,D)
[v2] Sat, 20 Mar 2021 07:33:27 GMT (2176kb,D)
Link back to: arXiv, form interface, contact.