I-Corps: Virtual 3D reconstruction of hollow organs from white light endoscopy

白色光内窥镜（WLE）是一种允许医生对器官内部进行实时视频探索以检测疾病或指导手术的医疗器械。不幸的是，WLE视频数据在其原始形式下太麻烦而无法审查治疗后;因此，信息通常被简化为手写笔记或几个静止图像帧以包含在医疗记录中。这种视觉上丰富的信息的丢失限制了WLE告知有关治疗和手术的临床决策的能力。特别是，像膀胱癌这样的疾病，不幸的是，它是男性中第四常见的癌症，并且是所有癌症中每患者一生的最高治疗成本，将受益于审查WLE视频数据的新方法，以促进肿瘤的早期检测，并更好地跟踪可能复发的患者膀胱壁的变化（50%）。该项目的目标是开发新技术，从WLE视频中生成中空器官（如膀胱）内部的3D可视化。这种技术的可用性将为医生提供新的工具，以做出更明智的治疗和手术决策，并为研究人员提供新技术，以实现对疾病进展的新研究，最终导致更好的健康结果和更低的癌症等疾病的治疗成本。所提出的新算法适用于最新的艺术技术在计算机视觉中的中空器官的形状和表面外观的三维重建问题。所提出的方法的一个关键创新是，该算法适合于从用标准临床硬件捕获的内窥镜视频重建器官的完整3D模型，并且仅需要对标准临床工作流程进行微小修改。从标准设备和工作流程创建这些重建的能力源于以下方面的精心设计决策：（1）内窥镜视频采集协议，（2）必要的图像预处理步骤，以及（3）计算机视觉社区开发的最先进技术的特定组合，以形成我们应用程序特有的端到端管道。简而言之，整个算法涉及以下步骤：对原始图像数据进行下采样，处理所选帧，确定相机姿态，将器官表面外推为网格，并将基于图像的纹理应用于最终网格。该团队的最初应用是使用标准刚性膀胱镜检查数据重建膀胱;迄今为止，该团队已经验证了使用30名人类患者的标准临床膀胱镜检查视频进行3D重建的能力。由于所提出的方法可以有力地增强内部器官外观的视觉医疗记录，因此它广泛适用于内窥镜检查，并且代表了随着时间的推移监测患者器官外观的显著进步，这可能非常适合于诸如癌症监测的应用。

White light endoscope (WLE) is a medical device that allows physicians to conduct real-time video explorations of the interior of organs to detect disease or guide surgery. Unfortunately, the WLE video data are too cumbersome in their native form to review post-treatment; hence, the information is often reduced to handwritten notes or a few still-image frames for inclusion in medical records. The loss of this visually rich information limits the ability of WLE to inform clinical decisions about treatment and surgery. In particular, diseases like bladder cancer, which holds the unfortunate distinction as the 4th most common cancer in men and the highest treatment cost per patient-lifetime of all cancers, would benefit from novel ways to review WLE video data to facilitate early detection of tumors and to better track changes in the bladder wall of patients likely to experience recurrence ( 50%). The goal of this project is to develop new technology to produce 3D visualizations of the interior of hollow organs such as the bladder from WLE videos. The availability of such technology will provide physicians with new tools to make better informed decisions about treatment and surgery, as well as provide researchers with new technology to enable novel studies on disease progression, ultimately leading to better health outcomes and lower treatment costs for diseases like cancer.The proposed novel algorithm applies state-of-the-art techniques in computer vision to the problem of 3D reconstruction of the shape and surface appearance of hollow organs. A key innovation in the proposed approach is that the algorithm is suitable to reconstruct a full, 3D model of an organ from endoscopic video captured with standard clinical hardware and requires only minor modifications to the standard clinical workflow. The ability to create these reconstructions from standard equipment and workflows arises from careful design decisions regarding (1) the protocol for endoscopic video collection, (2) necessary image pre-processing steps and (3) the particular combination of state-of-the-art techniques developed in the computer vision community into an end-to-end pipeline unique for our application. In brief, the overall algorithm involves the following steps: down-sample raw image data, process selected frames, determine camera poses, extrapolate the organ surface as a mesh and apply image-based texture to the finalized mesh. The team's initial application is for reconstructing urinary bladder using standard rigid cystoscopy data; to date the team has validated the ability to perform 3D reconstructions using standard clinical cystoscopy videos of 30 human patients. As the proposed method can powerfully augment the visual medical record of internal organ appearance, it is broadly applicable to endoscopy and represents a significant advance in monitoring the appearance of a patient's organ over time, as may be well suited for applications such as cancer surveillance.