Collaborative Research: CPS: Medium: Robotic Perception and Manipulation via Full-Spectral Wireless Sensing

机器人操作和自动化系统在过去几年中受到了广泛的关注，并在智能制造、远程手术和家庭自动化等各种应用中表现出了良好的性能。这些进步部分归功于先进的感知能力（使用视觉和触觉）以及用于操纵和控制的新学习模型和算法。然而，最先进的网络物理系统在其感知和感知方面仍然局限于直接视线和与它们需要感知的对象的直接接触。该项目的目标是设计，构建和评估一个网络物理系统，该系统可以感知，感知，学习和操纵远远超出使用现有系统的可行性。为此，该研究将探索太赫兹波段，该波段通过无线太赫兹信号推断物体的固有材料特性，提供了一个新的传感维度，而无需直接接触。 该项目还将探索可以穿越闭塞的射频信号。基于这些新兴的传感模式，该项目的核心重点是开发全频谱感知，控制，学习和操纵任务。该项目的成功将使CPS系统架构具有前所未有的功能，从而为机器人操作提供全新的机会，使机器人操作更高效，并允许机器人执行以前不可能完成的新的复杂任务。该项目将通过全光谱无线传感实现机器人感知，以解锁前所未有的机器人操作能力。这项研究涉及学习感知和控制之间的协同作用-感知用于控制，反之亦然-以优化端到端的网络物理任务。特别是，这项研究包括三个相互关联的推动力：（i）它将实现一种新的传感模式，利用高分辨率太赫兹频率进行机器人成像和推理;（ii）它旨在建立一个新的全光谱学习平台。（毫米波、太赫兹和视觉）感知，以在光学系统缺乏性能的非视线和杂乱环境中实现超视觉感知和推理;以及（iii）它提供了一个学习传感和控制之间协同作用的平台，以进一步共同优化端到端机器人操作任务。这些功能可以为工业机器人以及辅助，仓储和智能家居机器人开辟全新的可能性领域。该研究将通过广泛的实验，原型设计和系统实现进行评估。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Robotic manipulation and automation systems have received a lot of attention in the past few years and have demonstrated promising performance in various applications spanning smart manufacturing, remote surgery, and home automation. These advances have been partly due to advanced perception capabilities (using vision and haptics) and new learning models and algorithms for manipulation and control. However, state-of-the-art cyber-physical systems remain limited in their sensing and perception to a direct line of sight and direct contact with the objects they need to perceive. The goal of this project is to design, build, and evaluate a cyber-physical system that can sense, perceive, learn, and manipulate far beyond what is feasible using existing systems. To do so, the research will explore the terahertz band, which offers a new sensing dimension by inferring the inherent material properties of objects via wireless terahertz signals and without direct contact. This project will also explore radio-frequency signals that can traverse occlusions. Building on these emerging sensing modalities, the core of the project focuses on developing full-spectrum perception, control, learning, and manipulation tasks. The success of this project will result in CPS system architectures with unprecedented capabilities, enabling fundamentally new opportunities to make robotic manipulation more efficient and allowing robots to perform new complex tasks that have not been possible before. The project will enable robotic perception via full-spectral wireless sensing in order to unlock unprecedented robotic manipulation capabilities. This research involves learning synergies between sensing and control- whereby sensing is used for control and vice-versa - to optimize the end-to-end cyber-physical tasks. In particular, this research includes three inter-connected thrusts: (i) It will enable a new sensing modality that exploits high-resolution terahertz frequencies for robotic imaging and inference; (ii) It aims to build a new learning platform for full spectrum (mmWave, THz, and vision) perception to enable beyond-vision perception and reasoning in non-line-of-sight and cluttered environments, where optical systems lack in performance; and (iii) It presents a platform to learn the synergies between sensing and control to further co-optimize the end-to-end robotic manipulations tasks. These capabilities can open up entirely new realms of possibility to industrial robotics as well as assistive, warehousing, and smart home robotic. The research will be evaluated through extensive experimentation, prototype design, and system implementation. The results will be disseminated through close collaboration with industry and publications in top research venues.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.