3D reconstruction of deformable colon structures based on preoperative model and deep neural network

In colonoscopy procedures, it is important to rebuild and visualize the colonic surface to minimize the missing regions and reinspect for abnormalities. Due to the fast camera motion and deformation of the colon in standard forward-viewing colonoscopies, traditional simultaneous localization and mapping (SLAM) systems work poorly for 3D reconstruction of colon surfaces and are prone to severe drift. Thus in this paper, a preoperative colon model segmented from CT scans is used together with the colonoscopic images to achieve the 3D colon reconstruction. The proposed framework includes dense depth estimation from monocular colonoscopic images using a deep neural network (DNN), visual odometry (VO) based camera motion estimation and an embedded deformation (ED) graph based non-rigid registration algorithm for deforming 3D scans to the segmented colon model. A realistic simulator is used to generate different simulation datasets with ground truth. Simulation results demonstrate the good performance of the proposed 3D colonic surface reconstruction method in terms of accuracy and robustness. In-vivo experiments are also conducted and the results show the practicality of the proposed framework for providing useful shape and texture information in colonoscopy applications.