EAGER: Construction of Social Interactions in 3D Space from First-Person Videos

EAGER：从第一人称视频构建 3D 空间中的社交互动

• 批准号: 1651389
• 负责人: Jianbo Shi
• 金额: $ 20万
• 依托单位: University of Pennsylvania
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-09-01 至 2018-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1651389&HistoricalAwards=false
• 关键词: EAGER; Construction; Social; Interactions; 3D

Precision modeling tools for realistic and complex human social interaction are not available today. First-person videos provide a unique opportunity to capture social interaction at unprecedented precision. In contrast, current third person surveillance video only records the few distance views of the interaction passively at a much reduced spatial resolution. This exploratory research project proposes to harness multiple first-person cameras as one collective instrument to capture, model, and predict social behaviors. The proposed research transforms the way we construct realistic social interaction models, while also advancing first-person video recognition. If successful, the envisioned computational model can act as a coach who learns what constitutes successful interactions and failures, thus being able to find solutions to mediate and prevent potential conflicts. The proposed research will model dynamic social interactions in 3D space from multiple personal perspectives. Recognition and prediction of complex social group interactions are challenging because people in the group can carry out unexpected actions intentionally or by mistake. In addition, due to variances in individuals' preferences and abilities, the same activities could be carried out in different ways. First-person videos can be highly jittery, resulting in fast and unpredictable object motions in the field of view. Building on PI's recent work establishing computational foundations for modeling social (people-people) and personal (people-scene) interactions using first-person cameras, this research will explore the novel concept the duality between social attention and roles: social attention provides a cue for recognizing social roles, and social roles facilitate the predictions of dynamic social formation change and its associated social attention. The formal foundation of the 3D model is based on constructing a visual memory that stores first-person social experiences in three forms: (a) geometric social formation, (b) visual image of first-person view, and (c) first-person seen by nearby third person views. As a proof-of-concept, the 3D space model capturing social interactions will be tested on collaborative social tasks such as assembling (Ikea) furniture, or building a block house with a group of friends. This research will construct a labeled dataset capturing the interactions, and perform analysis on both accuracy in recognizing social roles and precision in predicting spatial movements of the members in that social interaction. The results of this project, including papers and dataset, will be disseminated to the public through our project website (http://www.cis.upenn.edu/~jshi/NSF_SocialMemory/nsf_social_visual_memory.html). The software created under this project will be made available to the public through GitHub, a web-based Git repository hosting service

用于现实和复杂的人类社会互动的精确建模工具目前还不存在。第一人称视频提供了以前所未有的精确度捕捉社交互动的独特机会。相比之下，当前的第三人称监控视频仅以较低的空间分辨率被动地记录了交互的少数几个距离视图。这一探索性研究项目建议利用多个第一人称摄像头作为一个集体工具来捕获、建模和预测社会行为。提出的研究改变了我们构建现实社交模型的方式，同时也促进了第一人称视频识别。如果成功，设想的计算模型可以充当教练，学习什么构成成功的交互和失败，从而能够找到解决方案来调解和防止潜在的冲突。这项拟议的研究将从多个个人角度对3D空间中的动态社交互动进行建模。识别和预测复杂的社会群体互动是具有挑战性的，因为群体中的人可能会故意或错误地做出意想不到的行为。此外，由于个人喜好和能力的不同，相同的活动可能会以不同的方式进行。第一人称视频可能会高度抖动，导致视野中快速且不可预测的对象运动。在Pi最近的工作基础上，建立了使用第一人称摄像机模拟社会(人-人)和个人(人-场景)交互的计算基础，本研究将探索社会注意和角色之间的二元性这一新概念：社会注意为识别社会角色提供线索，而社会角色有助于预测社会形态的动态变化及其相关的社会关注。3D模型的形式基础是构建以三种形式存储第一人称社交经验的视觉记忆：(A)几何社会形态，(B)第一人称视角的视觉图像，以及(C)附近第三人称视角所看到的第一人称。作为概念验证，捕捉社交互动的3D空间模型将在协作社交任务中进行测试，如组装(宜家)家具或与一群朋友一起建造积木屋。本研究将构建一个捕捉互动的标签数据集，并对识别社会角色的准确性和预测该社会互动中成员的空间运动的准确性进行分析。该项目的结果，包括论文和数据集，将通过我们的项目网站(http://www.cis.upenn.edu/~jshi/NSF_SocialMemory/nsf_social_visual_memory.html).向公众发布在这个项目下创建的软件将通过GitHub向公众提供，GitHub是一个基于Web的Git存储库托管服务