CRII: RI: RUI: Generating Haptics in Telerobotics through Perception Complementarities during Physical Distancing

CRII：RI：RUI：通过物理距离期间的感知互补性在远程机器人中生成触觉

• 批准号: 2101107
• 负责人: Yun-Hsuan Su
• 金额: $ 17.44万
• 依托单位: Mount Holyoke College
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-06-01 至 2024-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2101107&HistoricalAwards=false
• 关键词: CRII; RI; RUI; Generating; Haptics

Haptic feedback, the sense of touch and awareness of movement, is a fundamental sensory pathway for human everyday life and plays a particularly essential role in object manipulation. In a physically distanced or miniaturized world, telepresence using robot proxies can assist humans with remote tasks, yet an ongoing issue is the limited haptic capabilities due to the lack of high-fidelity, low-cost haptic interfaces or sensors. As a plethora of information exists in the digital age, mostly through images or video streams, the ability to interpret the sense of touch from vision enables new opportunities. This project explores software solutions that can estimate contact force/torque from visual data and train robots to replicate force sensitive soft body manipulation tasks. This software agent will be trained to transmit real-time force/torque estimation without the need for haptic sensors. Promising application domains include providing surgeons with vision-based haptic feedback during robot-assisted minimally invasive surgery, short term telepresence demand for emergencies or disaster response, and teleoperating a companion robot to perform a haptically enabled virtual hug or a remote handshake. The goal is to create a novel software solution that helps humans stay connected, complete remote tasks through intelligent touch estimation, and lower the barrier to entry for haptic teleoperation by reducing accessibility and hardware requirements. This project leverages preliminary endeavors in vision-based force estimation in robot assisted minimally invasive surgery (RMIS). Meanwhile, the technology will evaluate the accuracy of the artificial force, analyze added benefits or limitations of the artificial haptic information, explore the sim-to-real transfer learning capabilities of the proposed framework through Variational Autoencoder-Generative Adversarial Networks (VAE-GANs) and prioritize cross-robot support by ensuring software compatibility with Robot Operating System (ROS), Collaborative Robotics Toolkit (CRTK) and Asynchronous Multi-Body Framework (AMBF) for dynamic simulation and visualization. The wide spectrum of applications ensure significant potential of this forward looking research to impact telerobotics in soft object manipulation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

触觉反馈，触觉和运动意识，是人类日常生活的基本感觉通路，在物体操作中起着特别重要的作用。在一个物理距离或小型化的世界中，使用机器人代理的远程呈现可以帮助人类完成远程任务，但由于缺乏高保真，低成本的触觉接口或传感器，目前的问题是触觉能力有限。由于数字时代存在大量信息，主要是通过图像或视频流，因此从视觉中解读触觉的能力带来了新的机会。该项目探索软件解决方案，可以从视觉数据中估计接触力/扭矩，并训练机器人复制力敏软体操作任务。该软件代理将接受培训，以传输实时力/扭矩估计，而无需触觉传感器。有前途的应用领域包括在机器人辅助微创手术期间为外科医生提供基于视觉的触觉反馈，紧急情况或灾难响应的短期远程呈现需求，以及远程操作伴侣机器人以执行触觉启用的虚拟拥抱或远程握手。我们的目标是创建一个新的软件解决方案，帮助人类保持连接，通过智能触摸估计完成远程任务，并通过降低可访问性和硬件要求来降低触觉遥操作的进入门槛。该项目利用机器人辅助微创手术（RMIS）中基于视觉的力估计的初步努力。同时，该技术将评估人工力的准确性，分析人工触觉信息的附加优点或局限性，通过变分自动编码器生成对抗网络探索所提出的框架的模拟到真实的迁移学习能力（VAE-GAN），并通过确保软件与机器人操作系统（ROS）的兼容性来优先考虑跨机器人支持，协作机器人工具包（CRTK）和异步多体框架（AMBF）用于动态仿真和可视化。广泛的应用范围确保了这一前瞻性研究在影响软物体操纵中的遥控机器人方面的巨大潜力。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Enhanced U-Net Tool Segmentation using Hybrid Coordinate Representations of Endoscopic Images

使用内窥镜图像的混合坐标表示增强 U-Net 工具分割

• DOI: 10.1109/ismr48346.2021.9661519
• 发表时间: 2021
• 期刊: 2021 International Symposium on Medical Robotics (ISMR
• 作者: Huang, Kevin;Chitrakar, Digesh;Jiang, Wenfan;Su, Yun-Hsuan
• 通讯作者: Su, Yun-Hsuan

Securing Robot-assisted Minimally Invasive Surgery through Perception Complementarities

通过感知互补确保机器人辅助微创手术

• DOI: 10.1109/irc.2020.00013
• 发表时间: 2020
• 期刊: Fourth IEEE International Conference on Robotic Computing (IRC
• 作者: Su, Yun-Hsuan;Sosnovskaya, Yana;Hannaford, Blake;Huang, Kevin
• 通讯作者: Huang, Kevin

Characterizing Limits of Vision-Based Force Feedback in Simulated Surgical Tool-Tissue Interaction

表征模拟手术工具-组织相互作用中基于视觉的力反馈的局限性

• DOI: 10.1109/embc44109.2020.9176658
• 发表时间: 2020
• 期刊: 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC
• 作者: Huang, Kevin;Chitrakar, Digesh;Mitra, Rahul;Subedi, Divas;Su, Yun-Hsuan
• 通讯作者: Su, Yun-Hsuan

A Deep-Learning Approach to Marble-Burying Quantification: Image Segmentation of Marbles and Bedding

大理石埋藏量化的深度学习方法：大理石和层理的图像分割

• DOI: 10.1109/sii55687.2023.10039320
• 发表时间: 2023
• 期刊: 2023 IEEE/SICE International Symposium on System Integration (SII
• 作者: Zhu, Yicheng;Hudson, Brandon;Chakraborttii, Chandranil;Su, Yun-Hsuan;Huang, Kevin
• 通讯作者: Huang, Kevin

Contact Localization via Active Oscillatory Actuation

通过主动振荡驱动进行接触定位

• DOI: 10.1109/sii52469.2022.9708753
• 发表时间: 2022
• 期刊: 2022 IEEE/SICE International Symposium on System Integration (SII
• 作者: Subedi, Divas;Schoemer, Elizabeth;Chitrakar, Digesh;Su, Yun-Hsuan;Huang, Kevin
• 通讯作者: Huang, Kevin