Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Transformers are Sample-Efficient World Models
(Submitted on 1 Sep 2022 (v1), last revised 1 Mar 2023 (this version, v2))
Abstract: Deep reinforcement learning agents are notoriously sample inefficient, which considerably limits their application to real-world problems. Recently, many model-based methods have been designed to address this issue, with learning in the imagination of a world model being one of the most prominent approaches. However, while virtually unlimited interaction with a simulated environment sounds appealing, the world model has to be accurate over extended periods of time. Motivated by the success of Transformers in sequence modeling tasks, we introduce IRIS, a data-efficient agent that learns in a world model composed of a discrete autoencoder and an autoregressive Transformer. With the equivalent of only two hours of gameplay in the Atari 100k benchmark, IRIS achieves a mean human normalized score of 1.046, and outperforms humans on 10 out of 26 games, setting a new state of the art for methods without lookahead search. To foster future research on Transformers and world models for sample-efficient reinforcement learning, we release our code and models at this https URL
Submission history
From: Eloi Alonso [view email][v1] Thu, 1 Sep 2022 17:03:07 GMT (290kb,D)
[v2] Wed, 1 Mar 2023 09:21:14 GMT (319kb,D)
Link back to: arXiv, form interface, contact.