Integrating Haptic and Visual Data for Low Latency transmission

集成触觉和视觉数据以实现低延迟传输

• 批准号: 2446549
• 金额: --
• 依托单位: King's College London
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2020
• 资助国家: 英国
• 起止时间: 2020 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2446549
• 关键词: Integrating; Haptic; Visual; Data; Low

Precisely determining the contact force during safe interaction in Minimally Invasive Surgery is still an open research challenge. Inspired by post-operative qualitative analysis from surgical videos, the use of cross-modality data driven deep neural network models has been one of the newest approach to predict sensorless force trends. The limited availability to vision-state-force datasets inspired the creation of a teleoperated laparoscopic forceps via robotic arm to recreate the forces of such tool when interacting with soft environments. To solve this task, we use a laparoscopic forceps mounted on a robotic arm, that we teleoperate using a haptic device. We collect a dataset containing visual, robot state and force readings for different types of soft silicone environments. We make a deep analysis on how different artificial intelligence models predict forces, first on a randomized training scheme. As we have a big variability on our dataset, we test the adaptability of these models to different geometries, colors and stiffnesses, and how good the visual and state vectors help on this part. Addtionally, we determined the effectiveness of such models to adapt and learn from multiple dataset.In the next part of my project, we decided to use generative deep learning techniques in order to model soft tissue dynamics from purely visual information. The approximation of such motion is done using modal coordinate analysis, really common in Finite Element Method simulations. We use the mathematical representation of the sinusoidal solutions on the frequency domain, using the Fast Fourier Transform, so we can further discretize the components and consider only a reasonable set of solutions.

精确确定微创手术中安全相互作用过程中的接触力仍然是一个开放的研究挑战。受手术视频的术后定性分析的启发，使用跨模态数据驱动的深度神经网络模型已成为预测无传感器力趋势的最新方法之一。视觉状态力数据集的有限可用性激发了通过机械臂创建远程操作腹腔镜钳的灵感，以在与软环境相互作用时重新创建这种工具的力。为了完成这项任务，我们使用安装在机械臂上的腹腔镜钳，我们使用触觉设备远程操作。我们收集了一个数据集，其中包含不同类型软硅环境的视觉，机器人状态和力读数。我们深入分析了不同的人工智能模型如何预测力，首先在随机训练方案上。由于我们的数据集有很大的可变性，我们测试了这些模型对不同几何形状、颜色和刚度的适应性，以及视觉和状态向量在这部分的帮助有多好。此外，我们还确定了这些模型在适应和学习多个数据集方面的有效性。在我项目的下一部分，我们决定使用生成式深度学习技术，以便从纯粹的视觉信息中建模软组织动力学。这种运动的近似是用模态坐标分析来完成的，这在有限元模拟中是很常见的。我们在频域上使用正弦解的数学表示，使用快速傅里叶变换，因此我们可以进一步离散分量并只考虑一组合理的解。