Towards Surgical Context Inference and Translation to Gestures
Towards Surgical Context Inference and Translation to Gestures
复制标题
DOI：
10.1109/icra48891.2023.10160383
复制
发表时间：
2023-02
期刊：
影响因子：
--
通讯作者：
Kay Hutchinson;Zongyu Li;Ian Reyes;H. Alemzadeh
中科院分区：
文献类型：
--
作者：
Kay Hutchinson;Zongyu Li;Ian Reyes;H. Alemzadeh
在机器人辅助手术中手动标记手势是劳动密集型的，容易出错，并且需要专业知识或培训。我们提出了一种自动化和可解释的生成手势成绩单，利用丰富的数据进行图像分割的方法。通过检查工具和对象之间的距离和交点，使用分割掩模检测手术背景。接下来，使用基于知识的有限状态机（FSM）和数据驱动的长短期记忆（LSTM）模型将上下文标签转换为手势转录。我们通过将结果与JIGSAWS数据集中的地面真值分割掩码、共识上下文标签和手势标签进行比较，来评估我们方法每个阶段的性能。我们的研究结果表明，我们的分割模型在识别缝合中的针和线方面达到了最先进的性能，并且我们可以自动检测重要的手术状态，与众包标签（例如，缝合中抓钳和物体之间的接触）。我们还发现，FSM模型比LSTM更强大的分割和标记性能差。我们提出的方法可以显着缩短手势标记过程（~2.8倍）。
Manual labeling of gestures in robot-assisted surgery is labor intensive, prone to errors, and requires expertise or training. We propose a method for automated and explainable generation of gesture transcripts that leverages the abundance of data for image segmentation. Surgical context is detected using segmentation masks by examining the distances and intersections between the tools and objects. Next, context labels are translated into gesture transcripts using knowledge-based Finite State Machine (FSM) and data-driven Long Short Term Memory (LSTM) models. We evaluate the performance of each stage of our method by comparing the results with the ground truth segmentation masks, the consensus context labels, and the gesture labels in the JIGSAWS dataset. Our results show that our segmentation models achieve state-of-the-art performance in recognizing needle and thread in Suturing and we can automatically detect important surgical states with high agreement with crowd-sourced labels (e.g., contact between graspers and objects in Suturing). We also find that the FSM models are more robust to poor segmentation and labeling performance than LSTMs. Our proposed method can significantly shorten the gesture labeling process (~2.8 times).