Variational Bayesian Model Selection for Mixture Distributions

Adrian Corduneanu, Christopher M. Bishop

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Mixture models, in which a probability distribution is represented as a linear superposition of component distributions, are widely used in statistical modelling and pattern recognition. One of the key tasks in the application of mixture models is the determination of a suitable number of components. Conventional approaches based on cross-validation are computationally expensive,are wasteful of data, and give noisy estimates for the optimal number of components.A fully Bayesian treatment, based on Markov chain Monte Carlo methods for instance, will return a posterior distribution over the number of components. However, in practical applications it is generally convenient, or even computationally essential, to select a single, most appropriate model. Recently it has been shown, in the context of linear latent variable models, that the use of hierarchical priors governed by continuous hyper-parameters whose values are set by type-II maximum likelihood, can be used to optimize model complexity. In this paper we extend this framework to mixture distributions by considering the classical task of density estimation using mixtures of Gaussians. We show that, by setting the mixing coefficients to maximize the marginal log-likelihood, unwanted components can be suppressed, and the appropriate number of components for the mixture can be determined in a single training run without recourse to cross validation.Our approach uses a variational treatment based on a factorized approximation to the posterior distribution.
Original languageEnglish
Title of host publicationProceedings Eighth International Conference on Artificial Intelligence and Statistics
PublisherMorgan Kaufmann
Number of pages8
Publication statusPublished - 2001


Dive into the research topics of 'Variational Bayesian Model Selection for Mixture Distributions'. Together they form a unique fingerprint.

Cite this