Current browse context:
cs.CV
Change to browse by:
References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Translation-equivariant Image Quantizer for Bi-directional Image-Text Generation
(Submitted on 1 Dec 2021 (this version), latest version 26 Feb 2023 (v3))
Abstract: Recently, vector-quantized image modeling has demonstrated impressive performance on generation tasks such as text-to-image generation. However, we discover that the current image quantizers do not satisfy translation equivariance in the quantized space due to aliasing, degrading performance in the downstream text-to-image generation and image-to-text generation, even in simple experimental setups. Instead of focusing on anti-aliasing, we take a direct approach to encourage translation equivariance in the quantized space. In particular, we explore a desirable property of image quantizers, called 'Translation Equivariance in the Quantized Space' and propose a simple but effective way to achieve translation equivariance by regularizing orthogonality in the codebook embedding vectors. Using this method, we improve accuracy by +22% in text-to-image generation and +26% in image-to-text generation, outperforming the VQGAN.
Submission history
From: Woncheol Shin [view email][v1] Wed, 1 Dec 2021 10:08:24 GMT (925kb,D)
[v2] Sun, 19 Feb 2023 13:03:16 GMT (154kb,D)
[v3] Sun, 26 Feb 2023 12:28:02 GMT (154kb,D)
Link back to: arXiv, form interface, contact.