References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Translating Images into Maps
(Submitted on 3 Oct 2021 (v1), last revised 30 Mar 2022 (this version, v2))
Abstract: We approach instantaneous mapping, converting images to a top-down view of the world, as a translation problem. We show how a novel form of transformer network can be used to map from images and video directly to an overhead map or bird's-eye-view (BEV) of the world, in a single end-to-end network. We assume a 1-1 correspondence between a vertical scanline in the image, and rays passing through the camera location in an overhead map. This lets us formulate map generation from an image as a set of sequence-to-sequence translations. Posing the problem as translation allows the network to use the context of the image when interpreting the role of each pixel. This constrained formulation, based upon a strong physical grounding of the problem, leads to a restricted transformer network that is convolutional in the horizontal direction only. The structure allows us to make efficient use of data when training, and obtains state-of-the-art results for instantaneous mapping of three large-scale datasets, including a 15% and 30% relative gain against existing best performing methods on the nuScenes and Argoverse datasets, respectively. We make our code available on this https URL
Submission history
From: Avishkar Saha [view email][v1] Sun, 3 Oct 2021 09:52:46 GMT (8227kb,D)
[v2] Wed, 30 Mar 2022 17:27:23 GMT (8586kb,D)
Link back to: arXiv, form interface, contact.