EAGER: Construction of Social Interactions in 3D Space from First-Person Videos

用于现实和复杂的人类社会互动的精确建模工具目前还不可用。第一人称视频提供了一个独特的机会，以前所未有的精度捕捉社交互动。相比之下，当前的第三人称监控视频仅以低得多的空间分辨率被动地记录交互的少数距离视图。这个探索性的研究项目提出利用多个第一人称相机作为一个集体工具来捕捉，建模和预测社会行为。这项研究改变了我们构建现实社交互动模型的方式，同时也推进了第一人称视频识别。如果成功，设想的计算模型可以作为教练，学习成功的互动和失败的构成，从而能够找到调解和预防潜在冲突的解决方案。拟议的研究将从多个人的角度在3D空间中模拟动态的社会互动。识别和预测复杂的社会群体互动是具有挑战性的，因为群体中的人可能会故意或错误地执行意想不到的行为。此外，由于个人的喜好和能力不同，同样的活动可以以不同的方式进行。第一人称视频可能非常抖动，导致视野中的物体运动快速且不可预测。基于PI最近的工作，建立了使用第一人称相机建模社会（人与人）和个人（人与场景）互动的计算基础，本研究将探索社会注意力和角色之间的双重性这一新概念：社会注意力为识别社会角色提供了线索，社会角色促进了动态社会形态变化及其相关社会注意力的预测。3D模型的形式基础是基于构建以三种形式存储第一人称社会经验的视觉记忆：（a）几何社会形成，（B）第一人称视图的视觉图像，以及（c）附近第三人称视图看到的第一人称。作为概念验证，捕捉社交互动的3D空间模型将在协作社交任务中进行测试，例如组装（宜家）家具，或与一群朋友一起建造积木屋。本研究将建立一个标记的数据集，捕捉的互动，并进行分析，在识别社会角色的准确性和预测的空间移动的成员在该社会互动的精度。该项目的成果，包括论文和数据集，将通过我们的项目网站（http：//www.cis.upenn.edu/numjshi/NSF_SocialMemory/nsf_social_visual_memory.html）向公众传播。在这个项目下创建的软件将通过GitHub向公众提供，GitHub是一个基于Web的Git存储库托管服务

Precision modeling tools for realistic and complex human social interaction are not available today. First-person videos provide a unique opportunity to capture social interaction at unprecedented precision. In contrast, current third person surveillance video only records the few distance views of the interaction passively at a much reduced spatial resolution. This exploratory research project proposes to harness multiple first-person cameras as one collective instrument to capture, model, and predict social behaviors. The proposed research transforms the way we construct realistic social interaction models, while also advancing first-person video recognition. If successful, the envisioned computational model can act as a coach who learns what constitutes successful interactions and failures, thus being able to find solutions to mediate and prevent potential conflicts. The proposed research will model dynamic social interactions in 3D space from multiple personal perspectives. Recognition and prediction of complex social group interactions are challenging because people in the group can carry out unexpected actions intentionally or by mistake. In addition, due to variances in individuals' preferences and abilities, the same activities could be carried out in different ways. First-person videos can be highly jittery, resulting in fast and unpredictable object motions in the field of view. Building on PI's recent work establishing computational foundations for modeling social (people-people) and personal (people-scene) interactions using first-person cameras, this research will explore the novel concept the duality between social attention and roles: social attention provides a cue for recognizing social roles, and social roles facilitate the predictions of dynamic social formation change and its associated social attention. The formal foundation of the 3D model is based on constructing a visual memory that stores first-person social experiences in three forms: (a) geometric social formation, (b) visual image of first-person view, and (c) first-person seen by nearby third person views. As a proof-of-concept, the 3D space model capturing social interactions will be tested on collaborative social tasks such as assembling (Ikea) furniture, or building a block house with a group of friends. This research will construct a labeled dataset capturing the interactions, and perform analysis on both accuracy in recognizing social roles and precision in predicting spatial movements of the members in that social interaction. The results of this project, including papers and dataset, will be disseminated to the public through our project website (http://www.cis.upenn.edu/~jshi/NSF_SocialMemory/nsf_social_visual_memory.html). The software created under this project will be made available to the public through GitHub, a web-based Git repository hosting service