Investigating keypoint descriptors for camera relocalization in endoscopy surgery.
Investigating keypoint descriptors for camera relocalization in endoscopy surgery.
复制标题
研究内窥镜手术中相机重新定位的关键点描述符。
DOI：
10.1007/s11548-023-02918-x
复制
发表时间：
2023
影响因子：
3
通讯作者：
Unberath,Mathias
中科院分区：
文献类型：
--
作者：
Hernández,Isabela;Soberanis-Mukul,Roger;Mangulabnan,JanEmily;Sahu,Manish;Winter,Jonas;Vedula,Swaroop;Ishii,Masaru;Hager,Gregory;Taylor,RussellH;Unberath,Mathias
目的计算机视觉和机器学习的最新进展已经产生了基于内窥镜视频的解决方案，用于解剖结构的密集重建。为了在手术导航中有效地使用这些系统，需要可靠的基于图像的技术来持续跟踪内窥镜摄像机在解剖结构内的位置，尽管频繁地移除和重新插入。在这项工作中，我们研究了使用最近的学习为基础的关键点描述符为6度的自由度的摄像机姿态估计在术中内窥镜序列和解剖结构的变化，由于surgical surgery.MethodsOur方法采用了密集的结构从运动（SfM）重建的术前解剖结构，获得了一个国家的最先进的患者特定的学习为基础的描述符。在重建步骤期间，每个估计的3D点与描述符相关联。在术中序列中采用该信息来建立用于透视n点（PSPOT）相机姿态估计的2D-3D对应关系。我们评估这种方法在6个术中序列，包括从两个尸体subject.ResultsShow，这种方法导致的平移和旋转误差为3.9毫米和0.2弧度，分别与21.86%的本地化相机平均超过6个序列的解剖修改。与附加的基于学习的描述符（HardNet++）相比，所选描述符可以实现具有类似姿态估计性能的本地化相机的更好百分比。我们进一步讨论了潜在的错误原因和局限性的建议approach.ConclusionPatient-specific学习为基础的描述符可以重新定位图像，以及分布在整个检查的解剖结构，即使在解剖结构被修改。然而，内窥镜序列中的摄像机重新定位仍然是一个具有挑战性的问题，未来的研究是必要的，以提高这种技术的鲁棒性和准确性。
PurposeRecent advances in computer vision and machine learning have resulted in endoscopic video-based solutions for dense reconstruction of the anatomy. To effectively use these systems in surgical navigation, a reliable image-based technique is required to constantly track the endoscopic camera’s position within the anatomy, despite frequent removal and re-insertion. In this work, we investigate the use of recent learning-based keypoint descriptors for six degree-of-freedom camera pose estimation in intraoperative endoscopic sequences and under changes in anatomy due to surgical resection.MethodsOur method employs a dense structure from motion (SfM) reconstruction of the preoperative anatomy, obtained with a state-of-the-art patient-specific learning-based descriptor. During the reconstruction step, each estimated 3D point is associated with a descriptor. This information is employed in the intraoperative sequences to establish 2D–3D correspondences for Perspective-n-Point (PnP) camera pose estimation. We evaluate this method in six intraoperative sequences that include anatomical modifications obtained from two cadaveric subjects.ResultsShow that this approach led to translation and rotation errors of 3.9 mm and 0.2 radians, respectively, with 21.86% of localized cameras averaged over the six sequences. In comparison to an additional learning-based descriptor (HardNet++), the selected descriptor can achieve a better percentage of localized cameras with similar pose estimation performance. We further discussed potential error causes and limitations of the proposed approach.ConclusionPatient-specific learning-based descriptors can relocalize images that are well distributed across the inspected anatomy, even where the anatomy is modified. However, camera relocalization in endoscopic sequences remains a persistently challenging problem, and future research is necessary to increase the robustness and accuracy of this technique.