Collaborative Research: CPS: Medium: Closing the Teleoperation Gap: Integrating Scene and Network Understanding for Dexterous Control of Remote Robots

协作研究：CPS：中：缩小远程操作差距：集成场景和网络理解以实现远程机器人的灵巧控制

• 批准号: 2038897
• 负责人: Stefanie Tellex
• 金额: $ 80万
• 依托单位: Brown University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-02-15 至 2025-01-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2038897&HistoricalAwards=false
• 关键词: Collaborative; Research; CPS; Medium; Closing

The aim of this proposal is to enable people to control robots remotely using virtual reality. Using cameras mounted on the robot and a virtual reality headset, a person can see the environment around the robot. However, controlling the robot using existing technologies is hard: there is a time delay because it’s slow to send high quality video over the Internet. In addition, the fidelity of the image is worse than looking through human eyes, with a fixed and narrow view. This proposal will address these limitations by creating a new system which understands the geometry and appearance of the robot’s environment. Instead of sending high-quality video over the Internet, this new system will only send a smaller amount of information about how the environment’s geometry and appearance has changed over time. Further, understanding the geometry and appearance will let us expand the view visible to the person. Overall, these will improve a human’s ability to remotely control the robot by increasing fidelity and responsiveness. We will demonstrate this technology on household tasks, on assembly tasks, and by manipulating small objects.The aim of this proposal is to test the hypothesis that integrating scene and networking understanding can enable efficient transmission and rendering for dexterous control of remote robots through virtual reality interfaces. This system will result in dexterous teleoperation that enables remote human operators to perform complex tasks with remote robot manipulators, such as cleaning a room or repairing a machine. Such tasks have not previously been demonstrated to be teleoperated for two reasons: 1) lack of an intuitive awareness and understanding of the scene around the remote robot, and 2) lack of an effective low-latency interface to control the robot. We will address these problems by creating new scene- and network-aware algorithms which tightly couple sensing, display, interaction and transmission, enabling the operator to quickly and intuitively understand the environment around the robot. This project will research new interfaces which allow the operator to use their hand to directly specify the robot’s end effector pose in six degrees of freedom, combined with spatial- and semantic-object-based models that allow safe high-level commands. This project will evaluate the proposed system by assessing the speed and accuracy of the remote operator’s ability to complete complex tasks, including assembly tasks; the aim will be to complete unstructured assembly tasks that have never been demonstrated to be remotely teleoperated before.This project is in response to the NSF Cyber-Physical Systems 20-563 solicitation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该提案的目的是使人们能够使用虚拟现实远程控制机器人。使用安装在机器人上的摄像头和虚拟现实耳机，人们可以看到机器人周围的环境。然而，使用现有技术控制机器人是困难的：有一个时间延迟，因为它的速度很慢，通过互联网发送高质量的视频。此外，图像的保真度比通过人眼观看更差，具有固定和狭窄的视野。该提案将通过创建一个新的系统来解决这些限制，该系统可以理解机器人环境的几何形状和外观。这个新系统不会通过互联网发送高质量的视频，而是只发送少量关于环境几何形状和外观如何随时间变化的信息。此外，理解几何形状和外观将使我们扩大对人可见的视野。总的来说，这些将通过提高保真度和响应能力来提高人类远程控制机器人的能力。我们将展示这项技术在家庭任务，装配任务，并通过操纵小物体。本提案的目的是测试的假设，集成场景和网络理解可以使有效的传输和渲染，通过虚拟现实接口的远程机器人的灵巧控制。该系统将实现灵巧的远程操作，使远程操作员能够使用远程机器人操作器执行复杂的任务，例如清洁房间或修理机器。由于两个原因，这些任务以前没有被证明是远程操作的：1）缺乏对远程机器人周围场景的直观感知和理解，以及2）缺乏有效的低延迟接口来控制机器人。我们将通过创建新的场景和网络感知算法来解决这些问题，这些算法将传感，显示，交互和传输紧密耦合，使操作员能够快速直观地了解机器人周围的环境。该项目将研究新的界面，允许操作员使用他们的手直接指定机器人的末端执行器的姿态在六个自由度，结合空间和语义对象为基础的模型，允许安全的高级命令。该项目将通过评估远程操作员完成复杂任务，包括组装任务的能力的速度和准确性来评价拟议的系统;其目标是完成以前从未被证明可以远程操作的非结构化装配任务。该项目是对美国国家科学基金会信息物理系统20-该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

TöRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis

• 发表时间: 2021-09
• 期刊: ArXiv
• 作者: Benjamin Attal;Eliot Laidlaw;Aaron Gokaslan;Changil Kim;Christian Richardt;J. Tompkin;Matthew O'Toole

HuMoR: 3D Human Motion Model for Robust Pose EstimationHuMoR: 3D Human Motion Model for Robust Pose Estimation

HuMoR：用于鲁棒姿势估计的 3D 人体运动模型HuMoR：用于鲁棒姿势估计的 3D 人体运动模型

• 发表时间: 2021
• 期刊: International Conference on Computer Vision (ICCV
• 作者: Rempe, Davis and

ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model

• DOI: 10.48550/arxiv.2207.09446
• 发表时间: 2022-07
• 期刊: ArXiv
• 作者: Rao Fu;Xiaoyu Zhan;Yiwen Chen;Daniel Ritchie;Srinath Sridhar