Learning generative models that span multiple data modalities, such as vision and language, is often motivated by the desire to learn more useful, generalisable representations that faithfully capture common underlying factors between the modalities. In this work, we characterise successful learning of such models as the fulfilment of four criteria: i) implicit latent decomposition into shared and private subspaces, ii) coherent joint generation over all modalities, iii) coherent cross-generation across individual modalities, and iv) improved model learning for individual modalities through multi-modal integration. Here, we propose a mixture-of-experts multi-modal variational autoencoder (MMVAE) for learning of generative models on different sets of modalities, including a challenging image <-> language dataset, and demonstrate its ability to satisfy all four criteria, both qualitatively and quantitatively.
|Title of host publication||Advances in Neural Information Processing Systems 32|
|Editors||H. Wallach, H. Larochelle, A. Beygelzimer, F. dtextquotesingle Alché-Buc, E. Fox, R. Garnett|
|Publisher||Curran Associates Inc|
|Number of pages||12|
|Publication status||Published - 8 Dec 2019|
|Event||33rd Conference on Neural Information Processing Systems - Vancouver Convention Centre, Vancouver, Canada|
Duration: 8 Dec 2019 → 14 Dec 2019
|Conference||33rd Conference on Neural Information Processing Systems|
|Abbreviated title||NeurIPS 2019|
|Period||8/12/19 → 14/12/19|
FingerprintDive into the research topics of 'Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models'. Together they form a unique fingerprint.
- School of Informatics - Reader in Explainable Artificial Intelligence
- Artificial Intelligence and its Applications Institute
- Data Science and Artificial Intelligence
Person: Academic: Research Active