Collaborative Research: CPS: Medium: Closing the Teleoperation Gap: Integrating Scene and Network Understanding for Dexterous Control of Remote Robots

这项提议的目的是使人们能够使用虚拟现实远程控制机器人。使用安装在机器人上的摄像头和虚拟现实耳机，人们可以看到机器人周围的环境。然而，使用现有技术控制机器人是困难的：存在时间延迟，因为它在互联网上发送高质量视频的速度很慢。此外，图像的逼真度比人眼看起来更差，视角固定而狭窄。这项提议将通过创建一种新的系统来解决这些限制，该系统可以理解机器人环境的几何和外观。这个新系统不会通过互联网发送高质量的视频，而是只会发送较少的关于环境的几何和外观随时间变化的信息。此外，了解几何图形和外观将使我们能够扩展对该人可见的视图。总体而言，这些将通过提高保真度和响应速度来提高人类远程控制机器人的能力。我们将在家居任务、装配任务和操纵小对象上演示这项技术。该建议的目的是验证这样一个假设，即将场景和网络理解相结合可以通过虚拟现实界面实现高效的传输和渲染，从而灵活地控制远程机器人。该系统将产生灵巧的遥操作，使远程人类操作员能够使用远程机器人操作器执行复杂的任务，如打扫房间或修理机器。这样的任务以前没有被证明是远程操作的，原因有两个：1)缺乏对远程机器人周围场景的直观感知和理解，以及2)缺乏有效的低延迟接口来控制机器人。我们将通过创建新的场景和网络感知算法来解决这些问题，这些算法将传感、显示、交互和传输紧密结合，使操作员能够快速和直观地了解机器人周围的环境。该项目将研究新的界面，允许操作员使用他们的手直接指定机器人六个自由度的末端执行器姿势，并结合基于空间和语义对象的模型，允许安全的高级命令。该项目将通过评估远程操作员完成复杂任务(包括装配任务)的速度和准确性来评估建议的系统；目标将是完成以前从未被证明是远程操作的非结构化装配任务。该项目是对NSF网络物理系统20-563请愿的响应。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The aim of this proposal is to enable people to control robots remotely using virtual reality. Using cameras mounted on the robot and a virtual reality headset, a person can see the environment around the robot. However, controlling the robot using existing technologies is hard: there is a time delay because it’s slow to send high quality video over the Internet. In addition, the fidelity of the image is worse than looking through human eyes, with a fixed and narrow view. This proposal will address these limitations by creating a new system which understands the geometry and appearance of the robot’s environment. Instead of sending high-quality video over the Internet, this new system will only send a smaller amount of information about how the environment’s geometry and appearance has changed over time. Further, understanding the geometry and appearance will let us expand the view visible to the person. Overall, these will improve a human’s ability to remotely control the robot by increasing fidelity and responsiveness. We will demonstrate this technology on household tasks, on assembly tasks, and by manipulating small objects.The aim of this proposal is to test the hypothesis that integrating scene and networking understanding can enable efficient transmission and rendering for dexterous control of remote robots through virtual reality interfaces. This system will result in dexterous teleoperation that enables remote human operators to perform complex tasks with remote robot manipulators, such as cleaning a room or repairing a machine. Such tasks have not previously been demonstrated to be teleoperated for two reasons: 1) lack of an intuitive awareness and understanding of the scene around the remote robot, and 2) lack of an effective low-latency interface to control the robot. We will address these problems by creating new scene- and network-aware algorithms which tightly couple sensing, display, interaction and transmission, enabling the operator to quickly and intuitively understand the environment around the robot. This project will research new interfaces which allow the operator to use their hand to directly specify the robot’s end effector pose in six degrees of freedom, combined with spatial- and semantic-object-based models that allow safe high-level commands. This project will evaluate the proposed system by assessing the speed and accuracy of the remote operator’s ability to complete complex tasks, including assembly tasks; the aim will be to complete unstructured assembly tasks that have never been demonstrated to be remotely teleoperated before.This project is in response to the NSF Cyber-Physical Systems 20-563 solicitation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.