CHS: Small: An Analysis-and-Synthesis Framework for Small Group Conversations

作为人与人之间最常见的交流形式之一，小组对话对我们的日常生活至关重要。人类本质上可以通过凝视、头部运动和手势的共同努力来管理和调节群体对话。然而，研究界对其模式和机制的认识和认识有限。对群体对话的结构、机制和交互模式的理解和建模不仅有助于揭示群体对话的科学，而且有助于在科学和工程中找到潜在的数值应用。计算建模和组会话的生成可以提供一个潜在的变革范例，以增强机器和工程系统（包括虚拟代理、人形机器人和物联网）的组会话能力，以便它们在与人类交互时遵循现实世界的社会规范。此外，它还可以极大地促进在教育、模拟和虚拟世界应用中创造更具社交性和生活化的数字人。此外，该项目将支持在休斯敦大学培养多样化的研究生和本科生。该项目解决了一个研究问题：我们是否以及如何通过计算分析小组对话，并进一步全面地综合各种潜在应用的生活化小组对话运动？大多数现有的研究工作都是利用各种平均量或统计度量计算的综合分析方法来解决这个问题。本项目采用了一种不同的方法，通过对群体对话行为进行计算建模来解决这个问题，并利用这个模型来全面综合各种潜在应用的生活化群体对话运动。研究团队将首先在内部获得一个高质量的小群体会话运动数据集，包括声学语音、面部表情、凝视、头部运动、手势和身体运动。基于此数据集，该项目将通过关注三个相互关联的研究重点来实现目标。第一个研究重点是开发一个计算分析模型，以提取和比较小组对话中轮流和轮流的有意义的注视模式。有了从第一个重点中获得的洞察力，第二个研究重点是设计一个整体的，基于深度学习的框架，根据各种用户输入有效地生成或合成逼真的小组会话动画。第三个研究重点是通过两个选定的应用程序全面评估所提出框架的有效性和有用性：虚拟世界中的小组远程会议和群体模拟小组对话。合成框架可以潜在地改变处理虚拟人群的应用程序的开发，例如购物广场、市中心街道和机场候机区的人群。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

As one of the most common forms of human-human communication, group conversation is essential to our daily life. Humans can intrinsically manage and regulate a group conversation via the concerted effort of gaze, head movement, and hand gesture. However, research communities only gain limited knowledge and insight on its pattern and mechanism. Understanding and modeling its structure, mechanism, and interaction patterns can not only help to reveal the science of group conversation but also find numerical potential applications in science and engineering. Computationally modeling and generation of group conversations can provide a potential transformative paradigm to empower the group conversational capability on machines and engineering systems, including virtual agents, humanoid robots, and Internet of Things, so that they can follow real-world social norms while interacting with humans. Also, it can significantly facilitate the creation of more socially-engaging and life-like digital humans in education, simulation, and virtual world applications. Furthermore, this project will support the training of a diverse cohort of graduate and undergraduate students at the University of Houston.The project addresses the research question: whether and how we can computationally analyze small group conversations and further holistically synthesize life-like group conversational motion for a variety of potential applications? Most existing research works have approached this problem using aggregated methods of analysis on the computation of various average quantities or statistical measures. This project takes a different approach and addresses this problem by computationally modeling the group conversation behaviors and leverages this model to holistically synthesize life-like group conversational motion for a variety of potential applications. The research team will first acquire, in-house, a high quality, small group conversational motion dataset that includes acoustic speech, facial expression, gaze, head movement, hand gesture, and body movement. Based on this dataset, the project will work towards the objectives by focusing on three inter-related research thrusts. The first research thrust is to develop a computational analysis model to extract and compare meaningful gaze patterns for turn-taking and turn-keeping in small group conversations. With the insight obtained from the first thrust, the second research thrust is to design a holistic, deep learning based framework to efficiently generate or synthesize life-like small group conversational animations based on various user inputs. The third research thrust is to comprehensively evaluate the effectiveness and usefulness of the proposed framework via two selected applications: small group teleconferencing within virtual worlds, and crowd simulations with group conversations. The synthesis framework can potentially transform development of applications that deal with virtual human crowds such as the ones in shopping plazas, downtown streets, and airport waiting areas.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.