We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Variational Diffusion Models

Abstract: Diffusion-based generative models have demonstrated a capacity for perceptually impressive synthesis, but can they also be great likelihood-based models? We answer this in the affirmative, and introduce a family of diffusion-based generative models that obtain state-of-the-art likelihoods on standard image density estimation benchmarks. Unlike other diffusion-based models, our method allows for efficient optimization of the noise schedule jointly with the rest of the model. We show that the variational lower bound (VLB) simplifies to a remarkably short expression in terms of the signal-to-noise ratio of the diffused data, thereby improving our theoretical understanding of this model class. Using this insight, we prove an equivalence between several models proposed in the literature. In addition, we show that the continuous-time VLB is invariant to the noise schedule, except for the signal-to-noise ratio at its endpoints. This enables us to learn a noise schedule that minimizes the variance of the resulting VLB estimator, leading to faster optimization. Combining these advances with architectural improvements, we obtain state-of-the-art likelihoods on image density estimation benchmarks, outperforming autoregressive models that have dominated these benchmarks for many years, with often significantly faster optimization. In addition, we show how to use the model as part of a bits-back compression scheme, and demonstrate lossless compression rates close to the theoretical optimum. Code is available at this https URL .
Comments: Published at NeurIPS'21. Camera-ready version, with code URL
Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as: arXiv:2107.00630 [cs.LG]
  (or arXiv:2107.00630v4 [cs.LG] for this version)

Submission history

From: Diederik P. Kingma Dr. [view email]
[v1] Thu, 1 Jul 2021 17:43:20 GMT (4462kb,D)
[v2] Mon, 12 Jul 2021 22:40:20 GMT (4462kb,D)
[v3] Fri, 18 Feb 2022 00:50:17 GMT (1893kb,D)
[v4] Mon, 13 Jun 2022 22:57:47 GMT (1893kb,D)

Link back to: arXiv, form interface, contact.