CAREER: Robot Perception of Human Physical Skills

该奖项的部分资金来自《2021年美国救援计划法案》(公法117-2)。日常人类活动是令人印象深刻的身体智能壮举--从走路时小心放置双脚避开障碍物，到手指精确而高度协调地移动来输入句子。机器人的身体智能即使只有人类的一小部分，也可以通过自动化重复的任务来彻底改变人们的生活。然而，尽管取得了进展，拥有这种身体能力的机器人仍然难以捉摸。该项目通过构建3D计算机视觉和机器学习算法，从互联网上随时可以获得或在野外捕获的大规模图像和视频集合中自动分析人类技能，从而朝着更有能力的机器人迈出了一步。它将产生一个大型的高级物理技能储存库，然后可以转移到机器人身上。该项目的教育和推广活动将传授机器人感知的理论知识，并为毕业生、本科生和高中生提供实践经验。此外，该项目将在基于计算机视觉的人类身体技能理解方面取得进展，在野外捕获大量的技能数据，并帮助解决CS以外的问题，例如在猴子手操纵的神经科学研究中。为了实现研究目标，该项目将促进基于计算机视觉的建模和从大规模视觉数据估计人类身体技能的最新水平。现有方法仅限于在结构化环境中操作，不能捕获在房屋等杂乱环境中获取的不受限制的可视数据中的交互。为了解决这一限制，该项目将建立(1)神经网络，以根据不受限制的数据对人体的物理属性(如形状和关节)进行建模和估计，(2)对视频中的人体运动和交互进行建模和估计，以及(3)收集和分析大量(10,000人/小时)人类活动的不受限制的视频，以建立身体技能存储库。这个储存库将通知从人类到机器人的技能转移。这项研究的长期目标是证明，从图像和视频中学习是机器人获得类似人类的身体能力的可行途径。为了实现教育和推广目标，该项目将通过购买几个摄像头和机器人手臂来教授一门高级课程、一个为期一学期的本科生研究体验计划和一个虚拟研讨会计划，从而将理论和实践结合起来。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Everyday human activities are impressive feats of physical intelligence - from careful placement of feet to avoid obstacles when walking, to the precise and highly coordinated movement of fingers to type a sentence. Robots with even a fraction of human physical intelligence could revolutionize lives by automating repetitive tasks. Despite advances however, robots with such physical abilities remain elusive. This project takes a step towards more capable robots by building 3D computer vision and machine learning algorithms for automatically analyzing human skills from large-scale image and video collections readily available on the internet or captured in the wild. It will produce a large repository of high-level physical skills that can then be transferred to robots. The education and outreach activities of the project will impart theoretical knowledge in robot perception and provide practical experience to graduates, undergraduates, and high school students. Furthermore, the project will lead to advances in computer vision-based understanding of human physical skills, in-the-wild capture of a significant amount of skills data and help to solve problems outside of CS such as in the study of the neuroscience of hand manipulation in monkeys.To meet the research goals, the project will advance the state of the art in computer vision-based modeling and estimation of human physical skills from large-scale visual data. Existing methods are limited to operating in structured environments and cannot capture interactions in unconstrained visual data taken in cluttered environments like homes. To address this limitation, the project will build (1) neural networks to model and estimate human physical properties such as shape and articulation from unconstrained data, (2) neural networks that model and estimate human motion and interaction from videos, and (3) methods for gathering and analyzing large amounts (10,000 person-hours) of unconstrained videos of human activities to build a repository of physical skills. This repository will inform the transfer of skills from humans to robots. The long-term aim of this research is to demonstrate that learning from images and videos is a viable path for robots to gain human-like physical abilities. To meet the education and outreach goals, the project will integrate theory and practice by acquiring several cameras and robot arms to teach an advanced course, a semester-long undergraduate research experience program, and a virtual workshop program.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.