Multimodal MR Synthesis via Modality-Invariant Latent Representation

Agisilaos Chartsias, Thomas Joyce, Mario Valerio Giuffrida, Sotirios Tsaftaris

Research output: Contribution to journalArticlepeer-review


We propose a multi-input multi-output fully convolutional neural network model for MRI synthesis. The model is robust to missing data, as it benefits from, but does not require, additional input modalities. The model is trained end-to-end, and learns to embed all input modalities into a shared modalityinvariant latent space. These latent representations are then combined into a single fused representation, which is transformed into the target output modality with a learnt decoder. We avoid the need for curriculum learning by exploiting the fact that the various input modalities are highly correlated. We also show that by incorporating information from segmentation masks the model can both decrease its error and generate data with synthetic lesions. We evaluate our model on the ISLES and BRATS datasets and demonstrate statistically significant improvements over state-of-the-art methods for single input tasks. This improvement increases further when multiple input modalities are used, demonstrating the benefits of learning a common latent space, again resulting in a statistically significant improvement over the current best method. Lastly, we demonstrate our approach on non skull-stripped brain images, producing a statistically significant improvement over the previous best method. Code is made publicly available at brain synthesis.
Original languageEnglish
Pages (from-to)803-814
JournalIEEE Transactions on Medical Imaging
Issue number3
Early online date18 Oct 2017
Publication statusPublished - Mar 2018


  • neural network
  • multi-modality fusion
  • Magnetic resonance imaging (MRI)
  • machine learning
  • brain


Dive into the research topics of 'Multimodal MR Synthesis via Modality-Invariant Latent Representation'. Together they form a unique fingerprint.

Cite this