Disentangle, align and fuse for multimodal and semi-supervised image segmentation

Agisilaos Chartsias, Giorgos Papanastasiou, Chengjia Wang, Scott Semple, David Newby, Rohan Dharmakumar, Sotirios A. Tsaftaris

Research output: Contribution to journalArticlepeer-review


Magnetic resonance (MR) protocols rely on several sequences to properly assess pathology and organ status. Yet, despite advances in image analysis we tend to treat each sequence, here termed modality, in isolation. Taking advantage of the information shared between modalities (largely an organ's anatomy) is beneficial for multi-modality multi-input processing and learning. However, we must overcome inherent anatomical misregistrations and disparities in signal intensity across the modalities to claim this benefit. We present a method that offers improved segmentation accuracy of the modality of interest (over a single input model), by learning to leverage information present in other modalities, enabling semi-supervised and zero shot learning. Core to our method is learning a disentangled decomposition into anatomical and imaging factors. Shared anatomical factors from the different inputs are jointly processed and fused to extract more accurate segmentation masks. Image misregistrations are corrected with a Spatial Transformer Network, that non-linearly aligns the anatomical factors. The imaging factor captures signal intensity characteristics across different modality data, and is used for image reconstruction, enabling semi-supervised learning. Temporal and slice pairing between inputs are learned dynamically. We demonstrate applications in Late Gadolinium Enhanced (LGE) and Blood Oxygenation Level Dependent (BOLD) cardiac segmentation, as well as in T2 abdominal segmentation.
Original languageEnglish
Pages (from-to)1-10
Number of pages10
JournalIEEE Transactions on Medical Imaging
Early online date6 Nov 2020
Publication statusE-pub ahead of print - 6 Nov 2020


  • Image Segmentation
  • Biomedical imaging
  • Annotations
  • Training
  • Decoding
  • Multimodal segmentation
  • Disentanglement
  • magnetic resonance imaging (MRI)


Dive into the research topics of 'Disentangle, align and fuse for multimodal and semi-supervised image segmentation'. Together they form a unique fingerprint.

Cite this