CRII: RI: Towards Learning Skills from First Person Demonstrations

人类从专家的示范中学习一项技能，比如打网球，这需要理解连续动作的微妙细节，例如，挥拍运动中的手眼协调。该项目开发了通过观看第一人称视频演示来学习这些技能的技术。由于严重的头部运动，第一人称视频是高度动态的、局部的和个人偏见的，这会产生更大的视觉数据变化。分析头戴式摄像机系统产生的视频具有挑战性，因为建立在第三人称视频基础上的最先进的计算机视觉系统不能直接应用。研究小组通过开发硬件和计算模型来解决这些挑战。该项目的首席研究员将把研究成果整合到明尼苏达大学新设计的一系列计算机视觉课程中。研究团队将公开分享数据集、表示和训练过的模型，并为计算机视觉和机器人技术的更广泛受众组织研讨会和教程。本研究调查了从第一人称演示中学习技能的问题。本项目设计了一个由第一人称摄像机和多个近身摄像机组成的头戴式摄像机系统，可以完全覆盖交互空间。该项目还开发了一种特定于头戴式摄像机系统的新表示方式，称为邻近可视性地图，以有效地表示3D视觉场景和动作。邻近可视性地图以3D深度图、视觉注意力和身体姿势的形式编码3D视觉语义，从而可以测量动作与其周围环境之间的相关性。这允许学习动态的邻近可视性映射来模拟不同的身体活动，例如，一个动作将如何改变其周围环境的状态。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Humans learn a skill from an expert's demonstrations such as playing tennis, which requires understanding subtle details of sequential actions, e.g., eye-hand coordination across swing motion. This project develops technologies to learn such skills by observing demonstrations from first-person videos. The first-person videos are highly dynamic, local, and person-biased due to severe head movements, which generates a larger variation of visual data. Analyzing the videos produced by the head-mounted camera system is challenging because state-of-the-art computer vision systems built upon third-person videos cannot be directly applied. The research team addresses these challenges by developing both hardware and computational models. The principal investigator of the project will integrate the research results into a sequence of newly designed computer vision courses in the University of Minnesota. The research team will publicly share the dataset, representation, and trained models, and organize workshops and tutorials to broader audiences in computer vision and robotics.This research investigates problems in learning skills from first person demonstrations. This project designs a head-mounted camera system composed of a first-person camera and multiple proxemic cameras that can fully cover the space of interactions. The project also develops a new representation specific to the head-mounted camera system, called proxemic affordance map, to efficiently represent visual scene and action in 3D. The proxemic affordance map encodes 3D visual semantics in a form of 3D depth map, visual attention, and body pose, which enables measuring the correlation between action and its surroundings. This allows learning the dynamics of proxemic affordance map to model diverse physical activities, e.g., how an action will change the state of its surrounding contexts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.