CRII: RI: Towards Learning Skills from First Person Demonstrations

人类从专家的演示中学习一项技能，比如打网球，这需要理解连续动作的细微细节，例如，挥杆动作中的眼手协调。该项目开发了通过观察第一人称视频的演示来学习此类技能的技术。由于剧烈的头部运动，第一人称视频具有高度的动态性、局部性和人的偏见，这会产生更大的视觉数据差异。分析头盔摄像头系统产生的视频是具有挑战性的，因为建立在第三人称视频基础上的最先进的计算机视觉系统不能直接应用。研究团队通过开发硬件和计算模型来应对这些挑战。该项目的首席研究员将把研究成果整合到明尼苏达大学新设计的一系列计算机视觉课程中。研究团队将公开分享数据集、表示法和经过训练的模型，并为更广泛的受众组织计算机视觉和机器人方面的研讨会和教程。这项研究调查了从第一人称演示中学习技能的问题。本项目设计了一个由一个第一人称摄像头和多个代理摄像头组成的能够完全覆盖互动空间的头盔摄像头系统。该项目还开发了一种特定于头盔摄像系统的新表示法，称为Proxemic Afforance map，以有效地在3D中表示视觉场景和动作。代理启示图以3D深度图、视觉注意力和身体姿势的形式对3D视觉语义进行编码，从而能够测量动作与其周围环境之间的相关性。这使得学习代用负担地图的动态来模拟不同的身体活动，例如，一个行动将如何改变其周围环境的状态。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Humans learn a skill from an expert's demonstrations such as playing tennis, which requires understanding subtle details of sequential actions, e.g., eye-hand coordination across swing motion. This project develops technologies to learn such skills by observing demonstrations from first-person videos. The first-person videos are highly dynamic, local, and person-biased due to severe head movements, which generates a larger variation of visual data. Analyzing the videos produced by the head-mounted camera system is challenging because state-of-the-art computer vision systems built upon third-person videos cannot be directly applied. The research team addresses these challenges by developing both hardware and computational models. The principal investigator of the project will integrate the research results into a sequence of newly designed computer vision courses in the University of Minnesota. The research team will publicly share the dataset, representation, and trained models, and organize workshops and tutorials to broader audiences in computer vision and robotics.This research investigates problems in learning skills from first person demonstrations. This project designs a head-mounted camera system composed of a first-person camera and multiple proxemic cameras that can fully cover the space of interactions. The project also develops a new representation specific to the head-mounted camera system, called proxemic affordance map, to efficiently represent visual scene and action in 3D. The proxemic affordance map encodes 3D visual semantics in a form of 3D depth map, visual attention, and body pose, which enables measuring the correlation between action and its surroundings. This allows learning the dynamics of proxemic affordance map to model diverse physical activities, e.g., how an action will change the state of its surrounding contexts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.