CHS: Small: A data-driven computational model of dyadic rapport: Learning and transforming nonverbal behavior in shared virtual environments.

社交互动是一项复杂且要求很高的任务——它可以以和谐而轻松的方式展开，但有时也会以灾难性的方式失败。成功的一个关键决定因素是合作伙伴是否能够建立融洽的关系。融洽关系影响着所有的沟通环境，从最初的联系形成到私人冲突的解决和商业谈判。虽然人类对融洽关系中的缺陷很敏感，但他们往往无法找出原因或采取应对措施。这个项目的动机是，新媒体技术，如社交虚拟现实（VR），可以增强人们在这方面的社会认知能力，提高我们在日常生活互动中的沟通技巧。在相关方面，机器的能力可以优于人类的社会感知，并且比人类的社会感知更少偏见。一方面，在虚拟现实中发生的互动允许系统记录影响融洽关系的行为细节（如运动，眼睛注视和面部表情）。此外，移动传感器技术可以无缝集成到VR设备中，如耳机或控制器，以测量情绪、动机和注意力调节的神经生理学相关性。另一方面，计算能力现在允许我们在标准的个人计算机上运行高度复杂的机器学习算法。该项目将利用虚拟现实捕捉技术、移动神经生理传感和深度学习方法来开发一种关系的生物行为模型。在此基础上，该项目将开发和评估工具，以监测正在进行的互动中的融洽关系，并管理反馈，以便采取纠正措施，改善融洽关系。调查人员将实现这两个目标。首先，他们将积累一个带注释的交互数据库，该数据库由150对执行三种不同交互任务的二人组（对话中的受试者对）组成，总持续时间为30分钟。该数据库将包括语言、运动、凝视、并发脑活动的脑电图测量和心血管测量。互动协议将由观察员小组注释，以便融洽相处，受试者自己将评估互动的质量和结果。其次，他们将开发和验证机器学习算法，以识别注释多通道数据库中的生物行为关系特征，并预测非语言行为的感知关系和生理反应。这一发展将导致一种关系的生物行为模型，它为社交AI组件提供了基础，能够在持续的虚拟角色互动中监控和促进关系。长期目标是将这些工具整合到社交VR或现实生活互动之外的通信媒体中，这取决于更先进、便携和不显眼的传感设备。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Social interaction is a complex and highly demanding task -- it can unfold in a harmonious and effortless way, yet sometimes also fail catastrophically. A critical determinant of success is whether or not the partners are able to establish rapport. Rapport impacts all communication contexts, from initial contact formation to private conflict resolution and business negotiations. While humans are sensitive to flaws in rapport, they frequently fail to identify the reasons or take counter measures. This project is motivated by the idea that new media technologies, such as social virtual reality (VR), can augment people's social-cognitive capacities in this regard and improve our communication skills in daily life interactions. In relevant respects, machine capabilities can be superior to, and less biased than, human social perception. On the one hand, interactions taking place in VR allow the system to register behavioral details affecting rapport (such as movement, eye gaze, and facial expressions). Moreover, mobile sensor technologies can be seamlessly integrated into VR devices, such as headsets or controllers to measure the neurophysiological correlates of emotional, motivational, and attentional attunement. On the other hand computational power now allows us to run highly complex machine learning algorithms on standard personal computers. This project will leverage VR capture technologies, mobile neurophysiological sensing and deep learning methods to develop a bio-behavioral model of rapport. Based on this the project will develop and evaluate tools to monitor rapport in ongoing interactions and administer feedback to enable corrective actions that improve rapport. The investigators will meet the two objectives. First, they will accumulate an annotated interaction database consisting of 150 dyads (pairs of subjects in conversation) performing three different interaction tasks, with an overall duration of 30 minutes. The database will include speech, movement, gaze, EEG measures of concurrent brain activity, and cardiovascular measures. The interaction protocols will be annotated for rapport by groups of observers, and the subjects themselves will evaluate interaction quality and outcomes. Second, they will develop and validate machine learning algorithms that identify bio-behavioral rapport signatures in the annotated multichannel database, and predict perceived rapport and physiological responses from nonverbal behavior. This development will lead to a bio-behavioral model of rapport, which provides the basis for social AI components, capable of monitoring and facilitating rapport in ongoing avatar interactions. Long term goals are to integrate these tools in communication media beyond social VR or in real life interactions, depending on more advanced, portable and unobtrusive sensing devices.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.