I-Corps: Virtual 3D reconstruction of hollow organs from white light endoscopy

白光内窥镜（WLE）是一种允许医生对器官内部进行实时视频探索以发现疾病或指导手术的医疗设备。遗憾的是，WLE视频数据的原始形式过于繁琐，无法进行后处理；因此，这些信息往往被简化为手写笔记或一些静态图像帧，以便纳入医疗记录。这种丰富的视觉信息的丢失限制了WLE为临床治疗和手术决策提供信息的能力。特别是像膀胱癌这样的疾病，它不幸地成为男性第四大最常见的癌症，也是所有癌症中每个患者一生的治疗费用最高的疾病，将受益于审查WLE视频数据的新方法，以促进肿瘤的早期发现，并更好地跟踪可能复发的患者膀胱壁的变化（50%）。该项目的目标是开发新技术，从WLE视频中产生中空器官（如膀胱）内部的3D可视化。这种技术的可用性将为医生提供新的工具，使他们在治疗和手术方面做出更明智的决定，并为研究人员提供新技术，使疾病进展的新研究成为可能，最终为癌症等疾病带来更好的健康结果和更低的治疗成本。提出的新算法将计算机视觉中的最新技术应用于中空器官形状和表面外观的三维重建问题。该方法的一个关键创新之处是，该算法适用于从标准临床硬件捕获的内窥镜视频中重建器官的完整3D模型，并且只需要对标准临床工作流程进行少量修改。从标准设备和工作流程中创建这些重建的能力源于以下方面的精心设计决策：(1)内窥镜视频采集协议，(2)必要的图像预处理步骤，以及(3)计算机视觉社区开发的最先进技术的特定组合，为我们的应用提供独特的端到端管道。简而言之，整个算法包括以下步骤：对原始图像数据进行下采样，处理选定的帧，确定相机姿势，将器官表面外推为网格，并将基于图像的纹理应用于最终网格。该团队最初的应用是使用标准刚性膀胱镜数据重建膀胱；迄今为止，该团队已经验证了使用30名人类患者的标准临床膀胱镜视频进行3D重建的能力。由于所提出的方法可以有力地增强内部器官外观的视觉医疗记录，因此它广泛适用于内窥镜检查，并且代表了随着时间的推移监测患者器官外观的重大进步，可能非常适合癌症监测等应用。

White light endoscope (WLE) is a medical device that allows physicians to conduct real-time video explorations of the interior of organs to detect disease or guide surgery. Unfortunately, the WLE video data are too cumbersome in their native form to review post-treatment; hence, the information is often reduced to handwritten notes or a few still-image frames for inclusion in medical records. The loss of this visually rich information limits the ability of WLE to inform clinical decisions about treatment and surgery. In particular, diseases like bladder cancer, which holds the unfortunate distinction as the 4th most common cancer in men and the highest treatment cost per patient-lifetime of all cancers, would benefit from novel ways to review WLE video data to facilitate early detection of tumors and to better track changes in the bladder wall of patients likely to experience recurrence ( 50%). The goal of this project is to develop new technology to produce 3D visualizations of the interior of hollow organs such as the bladder from WLE videos. The availability of such technology will provide physicians with new tools to make better informed decisions about treatment and surgery, as well as provide researchers with new technology to enable novel studies on disease progression, ultimately leading to better health outcomes and lower treatment costs for diseases like cancer.The proposed novel algorithm applies state-of-the-art techniques in computer vision to the problem of 3D reconstruction of the shape and surface appearance of hollow organs. A key innovation in the proposed approach is that the algorithm is suitable to reconstruct a full, 3D model of an organ from endoscopic video captured with standard clinical hardware and requires only minor modifications to the standard clinical workflow. The ability to create these reconstructions from standard equipment and workflows arises from careful design decisions regarding (1) the protocol for endoscopic video collection, (2) necessary image pre-processing steps and (3) the particular combination of state-of-the-art techniques developed in the computer vision community into an end-to-end pipeline unique for our application. In brief, the overall algorithm involves the following steps: down-sample raw image data, process selected frames, determine camera poses, extrapolate the organ surface as a mesh and apply image-based texture to the finalized mesh. The team's initial application is for reconstructing urinary bladder using standard rigid cystoscopy data; to date the team has validated the ability to perform 3D reconstructions using standard clinical cystoscopy videos of 30 human patients. As the proposed method can powerfully augment the visual medical record of internal organ appearance, it is broadly applicable to endoscopy and represents a significant advance in monitoring the appearance of a patient's organ over time, as may be well suited for applications such as cancer surveillance.