Maximum Likelihood Training of Score-Based Diffusion Models

Yang Song, Conor Durkan, Iain Murray, Stefano Ermon

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through a connection to continuous normalizing flows, but log-likelihood is not directly optimized by the weighted combination of score matching losses. We show that for a specific weighting scheme, the objective upper bounds the negative log-likelihood, thus enabling approximate maximum likelihood training of score-based diffusion models. We empirically observe that maximum likelihood training consistently improves the likelihood of score-based diffusion models across multiple datasets, stochastic processes, and model architectures. Our best models achieve negative log-likelihoods of 2.83 and 3.76 bits/dim on CIFAR-10 and ImageNet 32 × 32 without any data augmentation, on a par with state-of-the-art autoregressive models on these tasks.
Original languageEnglish
Title of host publicationAdvances in Neural Information Processing Systems 34
PublisherNeural Information Processing Systems Foundation, Inc
Number of pages14
Publication statusAccepted/In press - 28 Sep 2021
EventThirty-fifth Conference on Neural Information Processing Systems - Virtual
Duration: 6 Dec 202114 Dec 2021
https://nips.cc/Conferences/2021

Conference

ConferenceThirty-fifth Conference on Neural Information Processing Systems
Abbreviated titleNeurIPS 2021
Period6/12/2114/12/21
Internet address

Fingerprint

Dive into the research topics of 'Maximum Likelihood Training of Score-Based Diffusion Models'. Together they form a unique fingerprint.

Cite this