CRII: RI: RUI: Generating Haptics in Telerobotics through Perception Complementarities during Physical Distancing

触觉反馈，触觉和运动意识，是人类日常生活的基本感觉通路，在物体操作中起着特别重要的作用。在一个物理距离或小型化的世界中，使用机器人代理的远程呈现可以帮助人类完成远程任务，但由于缺乏高保真，低成本的触觉接口或传感器，目前的问题是触觉能力有限。由于数字时代存在大量信息，主要是通过图像或视频流，因此从视觉中解读触觉的能力带来了新的机会。该项目探索软件解决方案，可以从视觉数据中估计接触力/扭矩，并训练机器人复制力敏软体操作任务。该软件代理将接受培训，以传输实时力/扭矩估计，而无需触觉传感器。有前途的应用领域包括在机器人辅助微创手术期间为外科医生提供基于视觉的触觉反馈，紧急情况或灾难响应的短期远程呈现需求，以及远程操作伴侣机器人以执行触觉启用的虚拟拥抱或远程握手。我们的目标是创建一个新的软件解决方案，帮助人类保持连接，通过智能触摸估计完成远程任务，并通过降低可访问性和硬件要求来降低触觉遥操作的进入门槛。该项目利用机器人辅助微创手术（RMIS）中基于视觉的力估计的初步努力。同时，该技术将评估人工力的准确性，分析人工触觉信息的附加优点或局限性，通过变分自动编码器生成对抗网络探索所提出的框架的模拟到真实的迁移学习能力（VAE-GAN），并通过确保软件与机器人操作系统（ROS）的兼容性来优先考虑跨机器人支持，协作机器人工具包（CRTK）和异步多体框架（AMBF）用于动态仿真和可视化。广泛的应用范围确保了这一前瞻性研究在影响软物体操纵中的遥控机器人方面的巨大潜力。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Haptic feedback, the sense of touch and awareness of movement, is a fundamental sensory pathway for human everyday life and plays a particularly essential role in object manipulation. In a physically distanced or miniaturized world, telepresence using robot proxies can assist humans with remote tasks, yet an ongoing issue is the limited haptic capabilities due to the lack of high-fidelity, low-cost haptic interfaces or sensors. As a plethora of information exists in the digital age, mostly through images or video streams, the ability to interpret the sense of touch from vision enables new opportunities. This project explores software solutions that can estimate contact force/torque from visual data and train robots to replicate force sensitive soft body manipulation tasks. This software agent will be trained to transmit real-time force/torque estimation without the need for haptic sensors. Promising application domains include providing surgeons with vision-based haptic feedback during robot-assisted minimally invasive surgery, short term telepresence demand for emergencies or disaster response, and teleoperating a companion robot to perform a haptically enabled virtual hug or a remote handshake. The goal is to create a novel software solution that helps humans stay connected, complete remote tasks through intelligent touch estimation, and lower the barrier to entry for haptic teleoperation by reducing accessibility and hardware requirements. This project leverages preliminary endeavors in vision-based force estimation in robot assisted minimally invasive surgery (RMIS). Meanwhile, the technology will evaluate the accuracy of the artificial force, analyze added benefits or limitations of the artificial haptic information, explore the sim-to-real transfer learning capabilities of the proposed framework through Variational Autoencoder-Generative Adversarial Networks (VAE-GANs) and prioritize cross-robot support by ensuring software compatibility with Robot Operating System (ROS), Collaborative Robotics Toolkit (CRTK) and Asynchronous Multi-Body Framework (AMBF) for dynamic simulation and visualization. The wide spectrum of applications ensure significant potential of this forward looking research to impact telerobotics in soft object manipulation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.