Multiple Importance Sampling ELBO and Deep Ensembles of Variational Approximations

Oskar Kviman, Harald Melin, Victor Elvira, Jens Lagergren

Research output: Chapter in Book/Report/Conference proceedingConference contribution


In variational inference (VI), the marginal log-likelihood is estimated using the standard evidence lower bound (ELBO), or improved versions as the importance weighted ELBO (IWELBO). We propose the multiple importance sampling ELBO (MISELBO), a \textit{versatile} yet \textit{simple} framework. MISELBO is applicable in both amortized and classical VI, and it uses ensembles, e.g., deep ensembles, of independently inferred variational approximations. As far as we are aware, the concept of deep ensembles in amortized VI has not previously been established. We prove that MISELBO provides a tighter bound than the average of standard ELBOs, and demonstrate empirically that it gives tighter bounds than the average of IWELBOs. MISELBO is evaluated in density-estimation experiments that include MNIST and several real-data phylogenetic tree inference problems. First, on the MNIST dataset, MISELBO boosts the density-estimation performances of a state-of-the-art model, nouveau VAE. Second, in the phylogenetic tree inference setting, our framework enhances a state-of-the-art VI algorithm that uses normalizing flows. On top of the technical benefits of MISELBO, it allows to unveil connections between VI and recent advances in the importance sampling literature, paving the way for further methodological advances. We provide our code at \url{this https URL}.
Original languageEnglish
Title of host publicationProceedings of Machine Learning Research
Subtitle of host publicationInternational Conference on Artificial Intelligence and Statistics, 28-30 March 2022, A Virtual Conference
EditorsGustau Camps-Valls, Francisco J.R. Ruiz, Isabel Valera
Number of pages16
Publication statusPublished - 5 Mar 2022
EventInternational Conference on Artificial Intelligence and Statistics (AISTATS) -
Duration: 28 Mar 202230 Mar 2022


ConferenceInternational Conference on Artificial Intelligence and Statistics (AISTATS)


Dive into the research topics of 'Multiple Importance Sampling ELBO and Deep Ensembles of Variational Approximations'. Together they form a unique fingerprint.

Cite this