Back2Seg: Joint Background Estimation and Bacteria Segmentation on Optical Endomicroscopy Images

Pneumonia, a lung infection typically caused by bacteria, requires swift and accurate diagnosis, especially in critical care. Optical endomicroscopy (OEM) facilitates real-time acquisition of in vivo and in situ optical biopsies, aiding in the quick identification of bacteria. However, the challenge of visually analyzing the vast number of images generated by the OEM in real-time can lead to delays in necessary treatments. To address this, we introduce Back2Seg, a novel approach for the segmentation of bacteria in OEM image sequences. Prior research mainly focused on exploiting bacteria motion or relied on less accurate unsupervised background estimation methods. In this regard, to enhance the background estimation and thus bacteria segmentation, Back2Seg employs a two-stage architecture with one sub-network dedicated to estimating the background using a Convolutional Neural Network (CNN)-Transformer architecture and the other is a dual-input network, processing both the original and the estimated background sequences to accurately segment the bacteria. Our experiments demonstrate that Back2Seg effectively integrates the advantages of both supervised and unsupervised learning techniques, showing a 4.62% increase in correlation with annotations over unsupervised models and a 1.05 reduction in root mean squared error (RMSE), outperforming the top supervised approach.