CHS: Small: Digitally Mediated Multi-party Communication: Acquisition, Modeling, and Evaluation

在线持久和共享的多用户虚拟环境（MUVEs）拥有数千甚至数百万用户，构成了一个新兴和快速增长的领域，可能在不久的将来对高等教育产生巨大影响。玩家与玩家之间的直接互动，以及玩家在虚拟世界中建立的网络，是这些muve独特体验和成功的核心。然而，尽管它们的视觉真实感增强了，但当前muve的沉浸式“社交功能”充其量也只是初级的，因为现实世界的对话和社交互动还没有被模仿和建模。这是因为，由于非语言行为和交互模式的显著差异，将现有的一对一对话建模方法扩展到数字媒介的多方对话和虚拟世界中的交互在技术上具有挑战性。数字媒介的多方通信和交互的自动生成因此成为限制各种在线虚拟世界和虚拟现实应用的深度和有用性的主要技术障碍。在这项研究中，PI将通过设计新的算法和系统来解决这个问题，这些算法和系统由不同位置的用户的实时语音驱动，可以自动在化身上生成同步的多模态会话手势，包括头/眼运动、嘴唇运动、手势和身体姿势。项目成果将促进在以计算机为媒介的通信发挥作用的应用中广泛采用有用的虚拟形象和远程沉浸技术，包括教育、商业、保健和工程。PI将使获得的高保真多模态多方对话行为数据集可供科学界广泛使用，因此它们可以用于未来的研究。这个雄心勃勃的项目将集中在三个相互关联的研究重点上，这些研究重点与PI在计算机动画、虚拟人类和人机交互方面的研究专长相一致。基于实时语音输入自动生成逼真的说话化身；PI将设计高效和自动化的方案，通过融合已建立的社会交换规则和数据驱动的统计模型，基于实时语音输入生成实时说话的虚拟人物。自动生成具有沉浸式社交交流的可信倾听化身；PI将基于对现实生活中多方对话数据的深入统计分析，设计数据驱动的方案，以产生密切协调的目光、头部运动和身体姿势变化，以及倾听同伴之间的社交目光交换。在一个内部构建的研究测试平台上对拟建的虚拟角色介导的多方对话和交互方法进行比较评价所提议的框架的健壮性和有效性将通过将其集成到内部构建的研究测试平台（即简化的MUVE原型）来评估。

Online persistent and shared multi-user virtual environments (MUVEs), with thousands or even millions of users, constitute an emerging and rapidly growing field that is likely to dramatically impact higher education in the near future. Direct player-to-player interaction, and the networks that players develop in the virtual world, are central to the unique experience and success of these MUVEs. However, despite their increased visual realism, the immersive "social functionality" in current MUVEs is still rudimentary at best, since real-world conversations and social interactions have not been mimicked and modeled. This is because it is technically challenging to extend existing one-to-one conversation modeling approaches to digitally mediated multi-party conversations and interactions in virtual worlds, due to the significant differences in nonverbal behavior and interaction patterns. The automated generation of digitally mediated multi-party communication and interaction has thus become a major technical barrier that restricts the depth and usefulness of various online virtual worlds and virtual reality applications. In this research, the PI will tackle this issue by designing new algorithms and systems driven by live speech from users in different locations, which can automatically generate synchronized multi-modal conversational gestures on embodied avatars, including head/eye movement, lip movement, hand gesture, and body posture. Project outcomes will facilitate the widespread adoption of useful avatar and tele-immersion technology in applications where computer-mediated communication plays a role, including education, commerce, health and engineering. The PI will make the acquired high-fidelity multi-modal multi-party conversational behavior datasets available to the scientific community at large, so they can be used in future research. This ambitious project will focus on three inter-related research thrusts that are aligned with the PI's research expertise in computer animation, virtual humans, and human computer interaction. Automated generation of realistic talking avatars based on live speech input alone; the PI will design efficient and automated schemes to generate on-the-fly talking avatars based on live speech input, by fusing established social exchange rules with data-driven statistical modeling. Automated generation of believable listening avatars with immersive social exchanges; based on in-depth statistical analysis of real life multiparty conversation data, the PI will design data-driven schemes for generating tightly coordinated gazes, head movements, and body posture shifts on listening avatars, as well as social gaze exchanges between listening peers. Comparative evaluation of the proposed avatar-mediated multi-party conversation and interaction approach in an in-house built research testbed; the robustness and effectiveness of the proposed framework will be evaluated by integrating it into an in-house built research testbed (i.e., a simplified MUVE prototype).