CHS: Small: A data-driven computational model of dyadic rapport: Learning and transforming nonverbal behavior in shared virtual environments.

CHS：小型：数据驱动的二元关系计算模型：在共享虚拟环境中学习和转变非语言行为。

• 批准号: 1907807
• 负责人: Gary Bente
• 金额: $ 30万
• 依托单位: Michigan State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1907807&HistoricalAwards=false
• 关键词: CHS; Small; data; driven; computational

Social interaction is a complex and highly demanding task -- it can unfold in a harmonious and effortless way, yet sometimes also fail catastrophically. A critical determinant of success is whether or not the partners are able to establish rapport. Rapport impacts all communication contexts, from initial contact formation to private conflict resolution and business negotiations. While humans are sensitive to flaws in rapport, they frequently fail to identify the reasons or take counter measures. This project is motivated by the idea that new media technologies, such as social virtual reality (VR), can augment people's social-cognitive capacities in this regard and improve our communication skills in daily life interactions. In relevant respects, machine capabilities can be superior to, and less biased than, human social perception. On the one hand, interactions taking place in VR allow the system to register behavioral details affecting rapport (such as movement, eye gaze, and facial expressions). Moreover, mobile sensor technologies can be seamlessly integrated into VR devices, such as headsets or controllers to measure the neurophysiological correlates of emotional, motivational, and attentional attunement. On the other hand computational power now allows us to run highly complex machine learning algorithms on standard personal computers. This project will leverage VR capture technologies, mobile neurophysiological sensing and deep learning methods to develop a bio-behavioral model of rapport. Based on this the project will develop and evaluate tools to monitor rapport in ongoing interactions and administer feedback to enable corrective actions that improve rapport. The investigators will meet the two objectives. First, they will accumulate an annotated interaction database consisting of 150 dyads (pairs of subjects in conversation) performing three different interaction tasks, with an overall duration of 30 minutes. The database will include speech, movement, gaze, EEG measures of concurrent brain activity, and cardiovascular measures. The interaction protocols will be annotated for rapport by groups of observers, and the subjects themselves will evaluate interaction quality and outcomes. Second, they will develop and validate machine learning algorithms that identify bio-behavioral rapport signatures in the annotated multichannel database, and predict perceived rapport and physiological responses from nonverbal behavior. This development will lead to a bio-behavioral model of rapport, which provides the basis for social AI components, capable of monitoring and facilitating rapport in ongoing avatar interactions. Long term goals are to integrate these tools in communication media beyond social VR or in real life interactions, depending on more advanced, portable and unobtrusive sensing devices.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

社会互动是一项复杂而高要求的任务--它可以以一种和谐而轻松的方式展开，但有时也会以灾难性的方式失败。成功的一个关键决定因素是合作伙伴是否能够建立融洽的关系。融洽关系影响着所有的沟通环境，从最初的接触到私人冲突的解决和商业谈判。虽然人类对融洽关系中的缺陷很敏感，但他们经常无法找出原因或采取对策。该项目的动机是，新媒体技术，如社交虚拟现实（VR），可以增强人们在这方面的社会认知能力，提高我们在日常生活中的沟通能力。 在相关方面，机器的能力可能比人类的社会感知更上级，也更少偏见。一方面，在VR中发生的交互允许系统记录影响融洽关系的行为细节（例如运动，眼睛凝视和面部表情）。此外，移动的传感器技术可以无缝集成到VR设备中，例如耳机或控制器，以测量情绪，动机和注意力协调的神经生理学相关性。另一方面，计算能力现在允许我们在标准个人计算机上运行高度复杂的机器学习算法。该项目将利用VR捕捉技术、移动的神经生理传感和深度学习方法来开发一个融洽关系的生物行为模型。在此基础上，该项目将开发和评估各种工具，以监测正在进行的互动中的融洽关系，并管理反馈，以便采取改善融洽关系的纠正行动。调查人员将实现这两个目标。首先，他们将积累一个注释的交互数据库，由150对（对话中的主体对）组成，执行三个不同的交互任务，总持续时间为30分钟。该数据库将包括语音、运动、凝视、并发大脑活动的EEG测量和心血管测量。互动协议将由观察员小组进行注释，并由受试者自己评估互动质量和结果。其次，他们将开发和验证机器学习算法，以识别注释多通道数据库中的生物行为融洽签名，并预测非语言行为的感知融洽和生理反应。这一发展将导致一个融洽关系的生物行为模型，它为社交AI组件提供了基础，能够监控和促进正在进行的化身交互中的融洽关系。长期目标是将这些工具集成到社交VR之外的通信媒体中，或真实的生活互动中，这取决于更先进、便携和不显眼的传感设备。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Measuring the effects of co-location on emotion perception in shared virtual environments: An ecological perspective

测量共享虚拟环境中共处一地对情绪感知的影响：生态视角

• DOI: 10.3389/frvir.2023.1032510
• 发表时间: 2023
• 期刊: Frontiers in Virtual Reality
• 作者: Bente, Gary;Schmälzle, Ralf;Jahn, Nolan T.;Schaaf, Andrea
• 通讯作者: Schaaf, Andrea