CRII: RI: Towards Learning Skills from First Person Demonstrations

CRII：RI：从第一人称演示中学习技能

• 批准号: 1755895
• 负责人: Hyun Soo Park
• 金额: $ 17.5万
• 依托单位: University of Minnesota-Twin Cities
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-03-15 至 2021-01-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1755895&HistoricalAwards=false
• 关键词: CRII; RI; Towards; Learning; Skills

Humans learn a skill from an expert's demonstrations such as playing tennis, which requires understanding subtle details of sequential actions, e.g., eye-hand coordination across swing motion. This project develops technologies to learn such skills by observing demonstrations from first-person videos. The first-person videos are highly dynamic, local, and person-biased due to severe head movements, which generates a larger variation of visual data. Analyzing the videos produced by the head-mounted camera system is challenging because state-of-the-art computer vision systems built upon third-person videos cannot be directly applied. The research team addresses these challenges by developing both hardware and computational models. The principal investigator of the project will integrate the research results into a sequence of newly designed computer vision courses in the University of Minnesota. The research team will publicly share the dataset, representation, and trained models, and organize workshops and tutorials to broader audiences in computer vision and robotics.This research investigates problems in learning skills from first person demonstrations. This project designs a head-mounted camera system composed of a first-person camera and multiple proxemic cameras that can fully cover the space of interactions. The project also develops a new representation specific to the head-mounted camera system, called proxemic affordance map, to efficiently represent visual scene and action in 3D. The proxemic affordance map encodes 3D visual semantics in a form of 3D depth map, visual attention, and body pose, which enables measuring the correlation between action and its surroundings. This allows learning the dynamics of proxemic affordance map to model diverse physical activities, e.g., how an action will change the state of its surrounding contexts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人类从专家的示范中学习一项技能，比如打网球，这需要理解顺序动作的微妙细节，例如，挥杆动作中的手眼协调。该项目开发了通过观察第一人称视频演示来学习这些技能的技术。第一人称视频是高度动态的，局部的，并且由于严重的头部运动而具有人的偏见，这会产生更大的视觉数据变化。分析头戴式摄像头系统产生的视频具有挑战性，因为基于第三人称视频构建的最先进的计算机视觉系统无法直接应用。研究团队通过开发硬件和计算模型来解决这些挑战。该项目的主要研究者将把研究成果整合到明尼苏达大学新设计的计算机视觉课程中。研究团队将公开分享数据集、表示和训练模型，并为计算机视觉和机器人领域的更广泛受众组织研讨会和教程。这项研究调查了从第一人称演示中学习技能的问题。本项目设计了一个由第一人称摄像头和多个虚拟摄像头组成的头戴式摄像系统，可以完全覆盖交互空间。该项目还开发了一种特定于头戴式摄像机系统的新表示，称为视觉示能图，以有效地表示3D中的视觉场景和动作。语义示能图以3D深度图、视觉注意力和身体姿势的形式编码3D视觉语义，这使得能够测量动作与其周围环境之间的相关性。这允许学习情感示能图的动态以模拟各种身体活动，例如，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。