Collaborative Research: CPS: Medium: Robotic Perception and Manipulation via Full-Spectral Wireless Sensing

机器人操作和自动化系统在过去几年中受到了广泛的关注，并在智能制造、远程手术和家庭自动化等各种应用中表现出了良好的性能。这些进步部分是由于先进的感知能力（使用视觉和触觉）以及用于操作和控制的新学习模型和算法。然而，最先进的网络物理系统在传感和感知方面仍然局限于直接视线和与需要感知的物体的直接接触。该项目的目标是设计、构建和评估一个网络物理系统，该系统可以感知、感知、学习和操作，远远超出使用现有系统的可行性。为此，该研究将探索太赫兹波段，该波段通过无线太赫兹信号在没有直接接触的情况下推断物体的固有材料特性，从而提供了一个新的传感维度。该项目还将探索可以穿越遮挡的射频信号。在这些新兴传感模式的基础上，该项目的核心重点是开发全频谱感知、控制、学习和操作任务。该项目的成功将导致CPS系统架构具有前所未有的功能，从根本上提供新的机会，使机器人操作更有效，并允许机器人执行以前不可能完成的新的复杂任务。该项目将通过全频谱无线传感实现机器人感知，以解锁前所未有的机器人操作能力。这项研究包括学习感知和控制之间的协同作用，即感知用于控制，反之亦然，以优化端到端网络物理任务。特别是，这项研究包括三个相互关联的重点：(i)它将启用一种新的传感模式，利用高分辨率太赫兹频率进行机器人成像和推理；（ii）旨在为全频谱（毫米波、太赫兹和视觉）感知建立一个新的学习平台，以便在光学系统性能不足的非视距和杂乱环境中实现超视觉感知和推理；（iii）提供了一个学习传感和控制之间协同作用的平台，以进一步共同优化端到端机器人操作任务。这些能力可以为工业机器人、辅助机器人、仓储机器人和智能家居机器人开辟全新的可能性领域。这项研究将通过广泛的实验、原型设计和系统实施来评估。研究结果将通过与业界和顶级研究机构的出版物密切合作进行传播。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Robotic manipulation and automation systems have received a lot of attention in the past few years and have demonstrated promising performance in various applications spanning smart manufacturing, remote surgery, and home automation. These advances have been partly due to advanced perception capabilities (using vision and haptics) and new learning models and algorithms for manipulation and control. However, state-of-the-art cyber-physical systems remain limited in their sensing and perception to a direct line of sight and direct contact with the objects they need to perceive. The goal of this project is to design, build, and evaluate a cyber-physical system that can sense, perceive, learn, and manipulate far beyond what is feasible using existing systems. To do so, the research will explore the terahertz band, which offers a new sensing dimension by inferring the inherent material properties of objects via wireless terahertz signals and without direct contact. This project will also explore radio-frequency signals that can traverse occlusions. Building on these emerging sensing modalities, the core of the project focuses on developing full-spectrum perception, control, learning, and manipulation tasks. The success of this project will result in CPS system architectures with unprecedented capabilities, enabling fundamentally new opportunities to make robotic manipulation more efficient and allowing robots to perform new complex tasks that have not been possible before. The project will enable robotic perception via full-spectral wireless sensing in order to unlock unprecedented robotic manipulation capabilities. This research involves learning synergies between sensing and control- whereby sensing is used for control and vice-versa - to optimize the end-to-end cyber-physical tasks. In particular, this research includes three inter-connected thrusts: (i) It will enable a new sensing modality that exploits high-resolution terahertz frequencies for robotic imaging and inference; (ii) It aims to build a new learning platform for full spectrum (mmWave, THz, and vision) perception to enable beyond-vision perception and reasoning in non-line-of-sight and cluttered environments, where optical systems lack in performance; and (iii) It presents a platform to learn the synergies between sensing and control to further co-optimize the end-to-end robotic manipulations tasks. These capabilities can open up entirely new realms of possibility to industrial robotics as well as assistive, warehousing, and smart home robotic. The research will be evaluated through extensive experimentation, prototype design, and system implementation. The results will be disseminated through close collaboration with industry and publications in top research venues.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.