Project 3: Multispectral Imaging and Robotic Bronchoscopy Devices for Precision Lung Tumor Detection

项目3：多光谱成像和机器人支气管镜设备用于精确肺部肿瘤 检测 项目摘要 手术中最重要的进步之一是微创设备的发展， 用于内窥镜、腹腔镜和机器人辅助手术和诊断程序的技术。相比 然而，对于传统的或"开放式"手术，微创手术的主要问题是外科医生 失去他或她的感觉/触觉反馈并且不能看到手术视野的全部区域。这个问题出现了 因为切口通常很小，外科医生不能用他们的手/手指直接检查或搜索 用于肿瘤结节和病变。因此，仍然存在重大的技术挑战和未满足的临床需求， 开发用于在微创条件下进行手术导航和引导的小型化装置，特别是 用于机器人体内操作和导航，以及用于实时捕获解剖和分子 信息.为了应对这些挑战，我们组建了一个由资深专家组成的学术和工业团队， 在光电成像芯片设计，小型化仪器， 软件工程和医疗机器人。总体目标是集成多光谱传感器、灵活的机器人 操作和临床适用的靶向示踪剂，用于图像引导活检和未确诊肺的手术 结节和非小细胞肺癌（NSCLC）。具体来说，我们将开发一种多光谱成像传感器， 包括具有像素化多光谱滤波器的垂直堆叠的光电探测器，具有低读出噪声（小于 一个电子）和单独控制的RGB（红绿蓝）和近红外（NIR）曝光 互补金属氧化物半导体（CMOS）架构中的像素。我们还将开发全息图 本发明公开了一种用于提供三维（3D）可视化和增强现实显示的可穿戴护目镜设备， 手术基于全息图的护目镜将在手术设置期间提供增强或混合现实显示 因为外科医生可以用他们的自然视力观察病人，同时观看三维全息图显示， 由生物启发的多光谱成像传感器获取的癌症组织位置。此外，通过 学术-工业伙伴关系，我们将与约翰逊和约翰逊合作，开发和整合近， 机器人柔性支气管镜红外分子成像定位肺部病变恶性沉积物 和淋巴结。总之，总体原理和策略是提高灵敏度和特异性， 通过结合肿瘤靶向近红外在微创手术期间进行术中肿瘤成像 分子示踪剂与生物启发的多光谱成像设备和三维（3-D）全息显示。 这项拟议中的工作预计将产生用于手术导航的尖端分子成像系统， 在微创条件下的指导，以及约翰逊和约翰逊的第一次人体演示 具有分子荧光引导和3D全息增强现实的柔性机器人支气管镜 视觉化

Project 3: Multispectral Imaging and Robotic Bronchoscopy Devices for Precision Lung Tumor Detection Project Summary One of the most significant advances in surgery has been the development of minimally invasive devices and technologies for endoscopic, laparoscopic, and robotic assisted surgical and diagnostic procedures. In comparison with conventional or “open” surgery, a major problem in minimally invasive surgery is however that the surgeon loses his or her sensory/tactile feedback and cannot visualize the full field of surgical view. This problem arises because the incisions are often so small, that surgeons cannot use their hands/fingers to directly inspect or search for tumor nodules and lesions. Thus, there are still major technical challenges and unmet clinical needs in developing miniaturized devices for surgical navigation and guidance under minimally invasive conditions, especially for robotic in-body manipulation and navigation, and for real-time capturing of both anatomical and molecular information. To address these challenges, we have assembled an academic and industrial team of senior investigators with interdisciplinary expertise in optoelectronic imaging chip design, miniaturized instrumentation, software engineering, and medical robotics. The overall goal is to integrate multispectral sensors, flexible robotic manipulation, and clinically applicable targeted tracers for image-guided biopsy and surgery of undiagnosed lung nodules and non-small-cell lung cancer (NSCLC). Specifically, we will develop a multispectral imaging sensor comprised of vertically stacked photodetectors with pixelated multispectral filters, with low readout noise (less than one electron) and individually controlled exposure for both color RGB (red-green-blue) and near-infrared (NIR) pixels in a complementary metal-oxide-semiconductor (CMOS) architecture. We will also develop hologram wearable goggle devices for providing three-dimensional (3D) visualization and augmented reality display during surgery. The hologram-based goggles will provide an augmented or mixed-reality display during surgical settings because the surgeons can observe the patient with their natural eyesight while viewing a 3-D hologram display of cancer tissue locations acquired by the bio-inspired multispectral imaging sensor. Furthermore, through an academic-industrial partnership, we will collaborate with Johnson and Johnson to develop and incorporate near- infrared molecular imaging into a robot-enabled flexible bronchoscope to locate malignant deposits in lung lesions and lymph nodes. In summary, the overall rationale and strategies are to improve the sensitivity and specificity of intraoperative tumor imaging during minimally invasive procedures by combining tumor targeted near-infrared molecular tracers with bio-inspired multispectral imaging devices and three-dimensional (3-D) holographic displays. The proposed work is expected to yield cutting-edge molecular imaging systems for surgical navigation and guidance under minimally invasive conditions, as well as the first in-human demonstration of Johnson & Johnson's flexible robotic bronchoscope with molecular fluorescence guidance and 3D hologram augmented reality visualization.