References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Vector-quantized Image Modeling with Improved VQGAN
(Submitted on 9 Oct 2021 (v1), last revised 5 Jun 2022 (this version, v3))
Abstract: Pretraining language models with next-token prediction on massive text corpora has delivered phenomenal zero-shot, few-shot, transfer learning and multi-tasking capabilities on both generative and discriminative language tasks. Motivated by this success, we explore a Vector-quantized Image Modeling (VIM) approach that involves pretraining a Transformer to predict rasterized image tokens autoregressively. The discrete image tokens are encoded from a learned Vision-Transformer-based VQGAN (ViT-VQGAN). We first propose multiple improvements over vanilla VQGAN from architecture to codebook learning, yielding better efficiency and reconstruction fidelity. The improved ViT-VQGAN further improves vector-quantized image modeling tasks, including unconditional, class-conditioned image generation and unsupervised representation learning. When trained on ImageNet at \(256\times256\) resolution, we achieve Inception Score (IS) of 175.1 and Fr'echet Inception Distance (FID) of 4.17, a dramatic improvement over the vanilla VQGAN, which obtains 70.6 and 17.04 for IS and FID, respectively. Based on ViT-VQGAN and unsupervised pretraining, we further evaluate the pretrained Transformer by averaging intermediate features, similar to Image GPT (iGPT). This ImageNet-pretrained VIM-L significantly beats iGPT-L on linear-probe accuracy from 60.3% to 73.2% for a similar model size. VIM-L also outperforms iGPT-XL which is trained with extra web image data and larger model size.
Submission history
From: Jiahui Yu [view email][v1] Sat, 9 Oct 2021 18:36:00 GMT (12872kb,D)
[v2] Tue, 1 Mar 2022 20:00:26 GMT (16313kb,D)
[v3] Sun, 5 Jun 2022 01:57:58 GMT (16313kb,D)
Link back to: arXiv, form interface, contact.