We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Maximum Likelihood Training of Score-Based Diffusion Models

Abstract: Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through a connection to continuous normalizing flows, but log-likelihood is not directly optimized by the weighted combination of score matching losses. We show that for a specific weighting scheme, the objective upper bounds the negative log-likelihood, thus enabling approximate maximum likelihood training of score-based diffusion models. We empirically observe that maximum likelihood training consistently improves the likelihood of score-based diffusion models across multiple datasets, stochastic processes, and model architectures. Our best models achieve negative log-likelihoods of 2.83 and 3.76 bits/dim on CIFAR-10 and ImageNet 32x32 without any data augmentation, on a par with state-of-the-art autoregressive models on these tasks.
Comments: NeurIPS 2021 (Spotlight)
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
Cite as: arXiv:2101.09258 [stat.ML]
  (or arXiv:2101.09258v4 [stat.ML] for this version)

Submission history

From: Yang Song [view email]
[v1] Fri, 22 Jan 2021 18:22:29 GMT (166kb)
[v2] Fri, 4 Jun 2021 04:42:19 GMT (1391kb,D)
[v3] Wed, 23 Jun 2021 04:48:04 GMT (1122kb,D)
[v4] Thu, 21 Oct 2021 00:06:11 GMT (1133kb,D)

Link back to: arXiv, form interface, contact.