GI Video-to-3D: Gastrointestinal Tract 3D Modeling and Visualization for In-Vivo

描述（由申请人提供）：微型摄像机技术的最新进展导致了称为胶囊摄像机的新一代内窥镜器械[1~5]，其使用可吞咽药丸大小的微型无线视频传感器来采集和传输视频图像序列，同时沿胃肠道（GI）沿着。虽然这项革命性的技术为患者提供了无痛的检查体验，但它只提供了二维（2D）连续视频图像，不包含观察目标的三维（3D）信息。病理结构的识别和评估及其空间维度的估计只能通过经验来实现，使得诊断决策非常主观。对于日常临床使用，前所未有的庞大数据集（每次检查超过57，600张原始2D图像）平均需要45分钟的医生时间进行审查。由于缺乏3D信息，定位特定目标相对于患者身体的空间位置是相当困难的任务。该SBIR计划的主要目标是开发一个3D图像处理计算软件平台，称为“视频到3DTM”，它能够（自动）生产，第一次，一个综合的，病人特异性的，可定量测量的胃肠道（GI）基于胶囊摄像机在检查期间采集的2D视频序列的道3D模型。能够对病理结构进行定量3D测量，校正后的彩色纹理可使内镜诊断更加客观和可重复。此外，患者特异性胃肠道的集成3D模型和3D飞越可视化软件能力将有助于诊断和干预计划，节省医生在视频审查方面的大量时间，并提供相对于患者身体的“感兴趣目标”的准确3D体内定位。由于胶囊摄像机在胃肠道内自由移动，其6自由度运动既不可控，也无法精确测量。因此，我们需要开发复杂的计算效率的算法，以获得一个非常强大的估计相机运动从未校准的图像序列，然后重建三维结构使用已知的相机运动和其他3D图像处理技术。“视频到3DTM”软件平台旨在执行以下任务：（1）3D建模：将任何胶囊摄像机采集的57，600多幅图像相互关联，以重建高分辨率的患者特定胃肠道3D模型;（2）3D可视化：为3D胃肠道模型提供纹理超分辨率和3D飞越功能，帮助医生快速、交互、准确地进行可视化和诊断，效率高，（3）3D测量：对感兴趣的病理结构进行定量3D测量;（4）3D定位：确定患者体内目标的准确3D体内位置。(5)计算机辅助诊断：所提出的Video-to-3D系统的最终目标是能够帮助医生检测、分类和识别某些目标疾病。拟议的“视频到3D TM”增加了一个维度，以现有的2D胶囊相机技术，从字面上和比喻。第一阶段构建视频到3D TM软件的具体目标包括：目标1：为视频到3D计算平台设计软件架构;目标2：开发和优化算法和软件组件;目标3：使用模拟平台和体内胶囊摄像机获取的图像进行广泛测试;目标4：评估第一阶段系统性能并为第二阶段工作计划做准备。公共卫生相关性：我们建议开发一个三维图像处理计算软件平台，被称为“视频到3DTM”，能够产生（自动）集成的，患者特异性的，基于胶囊摄像机在检查过程中采集的2D视频序列，建立可定量测量的胃肠道（GI）3D模型。这是一种全新的活体胶囊相机技术，首次提供前所未有的3D胃肠道建模、3D飞越可视化、病理结构的3D测量以及相对于患者身体的目标区域的3D体内定位功能，用于诊断和干预计划。拟议中的“视频到3D TM”从字面上和比喻上为现有的胶囊相机技术增加了一个维度。鉴于生物医学成像研究的需求不断扩大，所提出的软件平台技术最终是通用的，可扩展的，可互操作的。除了为革命性的胶囊相机技术和许多其他现有的医疗诊断仪器（内窥镜、胃镜、膀胱镜、结肠镜等）带来跨越式的性能进步外，在此开发的3D模型重构技术的潜在应用跨越了许多应用领域的广泛范围，例如管道检查、涡轮机发动机诊断、智能运输系统、安全和监视、车辆导航、移动的机器人、电话会议、虚拟现实产品、实况图像的在线分发、事件和位置观看、城市建模、娱乐、体育网络广播以及许多其它应用。

DESCRIPTION (provided by applicant): Recent advances in miniature camera technology have lead new generation endoscopic devices called capsule cameras [1~5] which use a swallowable pill-size miniature wireless video sensor to acquire and transmit video image sequences while traveling along the gastrointestinal(GI) tract. Although this revolutionary technology offers patients with painless examination experience, it however only provides two dimensional (2D) sequential video images that contains no three dimensional (3D) information of observed targets. Recognition and evaluation of pathological structures and the estimation of their spatial dimension can only be achieved by experience, making a diagnosis decision very subjective. For daily clinical usages, the unprecedented huge data set (more than 57,600 raw 2D images per exam) requires in average 45 minutes of physician's time to review. Locating the spatial position of a specific target with respect to patient's body is fairly difficult task due to lack of 3D information. The primary objective of this SBIR program is to develop a 3D image processing computational software platform, dubbed as the "Video-to-3DTM", which is able to (automatically) produce, for the first time, an integrated, patient-specific, and quantitatively measurable gastrointestinal (GI) tract 3D model based upon the 2D video sequence acquired by a capsule camera during an exam. The ability to perform quantitatively 3D measurement of pathological structures with calibrated color texture would make endoscopic diagnosis more objective and reproducible. Furthermore, an integrated 3D model of patient-specific GI tract and 3D fly-through visualization software capability would assist diagnosis and intervention planning, save physicians tremendous time in video reviewing, and provide accurate 3D intra-body localization of "target of interest" with respect to patient's body. Since a capsule camera moves freely inside the GI tract, its 6-degree-of-freedom motion is neither controllable nor accurately measurable. We therefore need to develop sophisticated computational efficient algorithms to obtain a very robust estimation of the camera motion from the uncalibrated image sequence, and then reconstruct 3D structure using the known camera motion and other 3D image processing techniques. The "Video-to-3DTM" Software Platform is designed to carry out the following tasks: (1) 3D Modeling: Inter-correlate over 57,600 images acquired by any capsule camera to reconstruct a high resolution patient-specific 3D model of GI tract; (2) 3D Visualization: Provide texture super-resolution and 3D fly-through capability for the 3D GI tract model to help physicians to visualize and diagnose quickly, interactively, accurately, and efficiently; (3) 3D Measurement: Perform quantitative 3D measurement of interested pathological structures; (4) 3D Localization: Determine accurate 3D intra-body location of targets within patient's body. (5) Computer-aided diagnosis: The ultimate goal of the proposed Video-to3D system is to be able to assist doctors to detect, classify, and identify certain targeted diseases. The proposed "Video-to-3DTM" adds one more dimension to the existing 2D capsule camera technology, literally and figuratively. Specific aims of Phase I effort to build the proposed Video-to-3DTM software include: Aim 1: Design software architecture for the Video-to-3D computational platform; Aim 2: Develop and optimize algorithms and software components; Aim 3: Perform extensive tests using simulation platform and images acquired by in-vivo capsule cameras; Aim 4: Assess Phase 1 system performance and prepare for Phase II work plan. PUBLIC HEALTH RELEVANCE: We propose to develop a 3D image processing computational software platform, dubbed as the "Video-to-3DTM", capable of producing (automatically) an integrated, patient-specific, and quantitatively measurable gastrointestinal (GI) tract 3D model based upon the 2D video sequence acquired by a capsule camera during an exam. The proposed Video-to-3DTM software platform blazes an entirely new trail in the in- vivo capsule camera technology by providing, for the first time, the unprecedented capability of 3D GI tract modeling, 3D fly-through visualization, 3D measurement of pathological structures and 3D intra-body localization of target area with respect to patient's body for diagnosis and intervention planning. The proposed "Video-to-3DTM" adds one more dimension to the existing capsule camera technology, literally and figuratively. Given the expanding needs in biomedical imaging research, the proposed software platform technology is ultimately generalizable, scalable, extensible, and interoperable. In addition to produce leapfrog performance advances for the revolutionary capsule camera technology and many other existing medical diagnosis instruments (endoscopes, gastroscope, cystoscope, colonscope, etc.), potential applications of 3D model reconstruction technology developed herein span a broad spectrum of many fields of applications, such as pipe inspection, turbine engine diagnosis, smart transportation systems, security and surveillance, vehicle navigation, mobile robotics, teleconferencing, virtual reality products, on-line distribution of live images, event and location viewing, urban modeling, entertainment, sports webcasts, and many others.