Integrating Haptic and Visual Data for Low Latency transmission

精确确定微创手术中安全交互过程中的接触力仍然是一个开放的研究挑战。受手术视频的术后定性分析的启发，使用跨模态数据驱动的深度神经网络模型已经成为预测无传感器力趋势的最新方法之一。视觉-状态-力数据集的有限可用性激发了通过机器人臂创建远程操作的腹腔镜镊子，以在与软环境交互时重新创建这种工具的力。为了解决这个任务，我们使用安装在机器人手臂上的腹腔镜镊子，我们使用触觉设备进行远程操作。我们收集了一个数据集，其中包含不同类型的软硅胶环境的视觉，机器人状态和力读数。我们深入分析了不同的人工智能模型如何预测力量，首先是随机训练方案。由于我们的数据集具有很大的可变性，我们测试了这些模型对不同几何形状，颜色和刚度的适应性，以及视觉和状态向量在这方面的帮助。此外，我们还确定了这些模型在适应和学习多个数据集方面的有效性。在我项目的下一部分，我们决定使用生成式深度学习技术，从纯粹的视觉信息中建模软组织动态。这种运动的近似是使用模态坐标分析来完成的，这在有限元方法模拟中非常常见。我们使用频域上的正弦解的数学表示，使用快速傅立叶变换，因此我们可以进一步离散分量并仅考虑合理的解集。

Precisely determining the contact force during safe interaction in Minimally Invasive Surgery is still an open research challenge. Inspired by post-operative qualitative analysis from surgical videos, the use of cross-modality data driven deep neural network models has been one of the newest approach to predict sensorless force trends. The limited availability to vision-state-force datasets inspired the creation of a teleoperated laparoscopic forceps via robotic arm to recreate the forces of such tool when interacting with soft environments. To solve this task, we use a laparoscopic forceps mounted on a robotic arm, that we teleoperate using a haptic device. We collect a dataset containing visual, robot state and force readings for different types of soft silicone environments. We make a deep analysis on how different artificial intelligence models predict forces, first on a randomized training scheme. As we have a big variability on our dataset, we test the adaptability of these models to different geometries, colors and stiffnesses, and how good the visual and state vectors help on this part. Addtionally, we determined the effectiveness of such models to adapt and learn from multiple dataset.In the next part of my project, we decided to use generative deep learning techniques in order to model soft tissue dynamics from purely visual information. The approximation of such motion is done using modal coordinate analysis, really common in Finite Element Method simulations. We use the mathematical representation of the sinusoidal solutions on the frequency domain, using the Fast Fourier Transform, so we can further discretize the components and consider only a reasonable set of solutions.