Maximum Likelihood Training of Score-Based Diffusion Models

Yang Song; Conor Durkan; Iain Murray; Stefano Ermon

Maximum Likelihood Training of Score-Based Diffusion Models

Yang Song, Conor Durkan, Iain Murray, Stefano Ermon

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through a connection to continuous normalizing flows, but log-likelihood is not directly optimized by the weighted combination of score matching losses. We show that for a specific weighting scheme, the objective upper bounds the negative log-likelihood, thus enabling approximate maximum likelihood training of score-based diffusion models. We empirically observe that maximum likelihood training consistently improves the likelihood of score-based diffusion models across multiple datasets, stochastic processes, and model architectures. Our best models achieve negative log-likelihoods of 2.83 and 3.76 bits/dim on CIFAR-10 and ImageNet 32 × 32 without any data augmentation, on a par with state-of-the-art autoregressive models on these tasks.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 34
Publisher	Neural Information Processing Systems Foundation, Inc
Number of pages	14
Publication status	Published - 6 Dec 2021
Event	Thirty-fifth Conference on Neural Information Processing Systems - Virtual Duration: 6 Dec 2021 → 14 Dec 2021 https://nips.cc/Conferences/2021

Publication series

Name	Advances in Neural Information Processing Systems
ISSN (Print)	1049-5258

Conference

Conference	Thirty-fifth Conference on Neural Information Processing Systems
Abbreviated title	NeurIPS 2021
Period	6/12/21 → 14/12/21
Internet address	https://nips.cc/Conferences/2021

Access to Document

Maximum Likelihood_SONG_DOA28092021_AFVAccepted author manuscript, 1.58 MBLicence: All Rights Reserved

https://proceedings.neurips.cc/paper/2021/hash/0a9fdbb17feb6ccb7ec405cfb85222c4-Abstract.htmlLicence: All Rights Reserved

Cite this

Song, Y., Durkan, C., Murray, I., & Ermon, S. (2021). Maximum Likelihood Training of Score-Based Diffusion Models. In Advances in Neural Information Processing Systems 34 (Advances in Neural Information Processing Systems). Neural Information Processing Systems Foundation, Inc. https://proceedings.neurips.cc/paper/2021/hash/0a9fdbb17feb6ccb7ec405cfb85222c4-Abstract.html

@inproceedings{0a0e6212e7a64dbeb2f95b759fa757cf,

title = "Maximum Likelihood Training of Score-Based Diffusion Models",

abstract = "Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through a connection to continuous normalizing flows, but log-likelihood is not directly optimized by the weighted combination of score matching losses. We show that for a specific weighting scheme, the objective upper bounds the negative log-likelihood, thus enabling approximate maximum likelihood training of score-based diffusion models. We empirically observe that maximum likelihood training consistently improves the likelihood of score-based diffusion models across multiple datasets, stochastic processes, and model architectures. Our best models achieve negative log-likelihoods of 2.83 and 3.76 bits/dim on CIFAR-10 and ImageNet 32 × 32 without any data augmentation, on a par with state-of-the-art autoregressive models on these tasks.",

author = "Yang Song and Conor Durkan and Iain Murray and Stefano Ermon",

year = "2021",

month = dec,

day = "6",

language = "English",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural Information Processing Systems Foundation, Inc",

booktitle = "Advances in Neural Information Processing Systems 34",

note = "Thirty-fifth Conference on Neural Information Processing Systems, NeurIPS 2021 ; Conference date: 06-12-2021 Through 14-12-2021",

url = "https://nips.cc/Conferences/2021",

}

Song, Y, Durkan, C , Murray, I & Ermon, S 2021, Maximum Likelihood Training of Score-Based Diffusion Models. in Advances in Neural Information Processing Systems 34. Advances in Neural Information Processing Systems, Neural Information Processing Systems Foundation, Inc, Thirty-fifth Conference on Neural Information Processing Systems, 6/12/21. <https://proceedings.neurips.cc/paper/2021/hash/0a9fdbb17feb6ccb7ec405cfb85222c4-Abstract.html>

TY - GEN

T1 - Maximum Likelihood Training of Score-Based Diffusion Models

AU - Song, Yang

AU - Durkan, Conor

AU - Murray, Iain

AU - Ermon, Stefano

PY - 2021/12/6

Y1 - 2021/12/6

N2 - Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through a connection to continuous normalizing flows, but log-likelihood is not directly optimized by the weighted combination of score matching losses. We show that for a specific weighting scheme, the objective upper bounds the negative log-likelihood, thus enabling approximate maximum likelihood training of score-based diffusion models. We empirically observe that maximum likelihood training consistently improves the likelihood of score-based diffusion models across multiple datasets, stochastic processes, and model architectures. Our best models achieve negative log-likelihoods of 2.83 and 3.76 bits/dim on CIFAR-10 and ImageNet 32 × 32 without any data augmentation, on a par with state-of-the-art autoregressive models on these tasks.

AB - Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through a connection to continuous normalizing flows, but log-likelihood is not directly optimized by the weighted combination of score matching losses. We show that for a specific weighting scheme, the objective upper bounds the negative log-likelihood, thus enabling approximate maximum likelihood training of score-based diffusion models. We empirically observe that maximum likelihood training consistently improves the likelihood of score-based diffusion models across multiple datasets, stochastic processes, and model architectures. Our best models achieve negative log-likelihoods of 2.83 and 3.76 bits/dim on CIFAR-10 and ImageNet 32 × 32 without any data augmentation, on a par with state-of-the-art autoregressive models on these tasks.

M3 - Conference contribution

T3 - Advances in Neural Information Processing Systems

BT - Advances in Neural Information Processing Systems 34

PB - Neural Information Processing Systems Foundation, Inc

T2 - Thirty-fifth Conference on Neural Information Processing Systems

Y2 - 6 December 2021 through 14 December 2021

ER -

Maximum Likelihood Training of Score-Based Diffusion Models

Abstract

Publication series

Conference

Access to Document

Fingerprint

Cite this