We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.GR

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computer Vision and Pattern Recognition

Title: ASSET: Autoregressive Semantic Scene Editing with Transformers at High Resolutions

Abstract: We present ASSET, a neural architecture for automatically modifying an input high-resolution image according to a user's edits on its semantic segmentation map. Our architecture is based on a transformer with a novel attention mechanism. Our key idea is to sparsify the transformer's attention matrix at high resolutions, guided by dense attention extracted at lower image resolutions. While previous attention mechanisms are computationally too expensive for handling high-resolution images or are overly constrained within specific image regions hampering long-range interactions, our novel attention mechanism is both computationally efficient and effective. Our sparsified attention mechanism is able to capture long-range interactions and context, leading to synthesizing interesting phenomena in scenes, such as reflections of landscapes onto water or flora consistent with the rest of the landscape, that were not possible to generate reliably with previous convnets and transformer approaches. We present qualitative and quantitative results, along with user studies, demonstrating the effectiveness of our method.
Comments: SIGGRAPH 2022 - Journal Track
Subjects: Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR)
Cite as: arXiv:2205.12231 [cs.CV]
  (or arXiv:2205.12231v1 [cs.CV] for this version)

Submission history

From: Difan Liu [view email]
[v1] Tue, 24 May 2022 17:39:53 GMT (47578kb,D)

Link back to: arXiv, form interface, contact.