CHS: Small: An Analysis-and-Synthesis Framework for Small Group Conversations

作为人与人之间最常见的交流形式之一，群组对话在我们的日常生活中是必不可少的。人类可以通过凝视、头部移动和手势的协同努力，从本质上管理和调节群体对话。然而，研究界对其模式和机制的了解和洞察有限。理解和建模群体对话的结构、机制和相互作用模式，不仅有助于揭示群体对话的科学性，而且可以在科学和工程中发现潜在的数值应用。通过计算建模和生成群组对话可以提供一种潜在的变革性范式，以增强机器和工程系统(包括虚拟代理、类人机器人和物联网)上的群组对话能力，使它们在与人类交互时能够遵循现实世界的社会规范。此外，它还可以显著促进在教育、模拟和虚拟世界应用中创建更具社会参与性和更逼真的数字人类。此外，这个项目将支持对休斯顿大学不同的研究生和本科生的培训。该项目解决了这样的研究问题：我们是否以及如何能够对小组对话进行计算分析，并进一步综合出逼真的小组对话动作，以满足各种潜在应用？大多数现有的研究工作都是通过计算各种平均值或统计措施的综合分析方法来处理这一问题的。这个项目采取了一种不同的方法，通过对群体对话行为进行计算建模来解决这个问题，并利用这个模型来整体合成逼真的群体对话运动，以满足各种潜在的应用。研究团队将首先在内部获得一个高质量的小组对话运动数据集，其中包括声学语音、面部表情、凝视、头部运动、手势和身体运动。在此数据集的基础上，该项目将通过专注于三个相互关联的研究推动力来实现这些目标。第一个研究重点是开发一个计算分析模型来提取和比较小群体对话中话轮转换和话轮保持的有意义的凝视模式。从第一个推力中获得的洞察力，第二个研究推力是设计一个基于深度学习的整体框架，基于各种用户输入高效地生成或合成逼真的小组对话动画。第三个研究重点是通过两个选定的应用程序来全面评估所提出的框架的有效性和有用性：虚拟世界中的小组电话会议和使用小组对话的人群模拟。该综合框架可能会改变处理虚拟人群的应用程序的开发，如购物广场、市中心街道和机场候机区的应用程序。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

As one of the most common forms of human-human communication, group conversation is essential to our daily life. Humans can intrinsically manage and regulate a group conversation via the concerted effort of gaze, head movement, and hand gesture. However, research communities only gain limited knowledge and insight on its pattern and mechanism. Understanding and modeling its structure, mechanism, and interaction patterns can not only help to reveal the science of group conversation but also find numerical potential applications in science and engineering. Computationally modeling and generation of group conversations can provide a potential transformative paradigm to empower the group conversational capability on machines and engineering systems, including virtual agents, humanoid robots, and Internet of Things, so that they can follow real-world social norms while interacting with humans. Also, it can significantly facilitate the creation of more socially-engaging and life-like digital humans in education, simulation, and virtual world applications. Furthermore, this project will support the training of a diverse cohort of graduate and undergraduate students at the University of Houston.The project addresses the research question: whether and how we can computationally analyze small group conversations and further holistically synthesize life-like group conversational motion for a variety of potential applications? Most existing research works have approached this problem using aggregated methods of analysis on the computation of various average quantities or statistical measures. This project takes a different approach and addresses this problem by computationally modeling the group conversation behaviors and leverages this model to holistically synthesize life-like group conversational motion for a variety of potential applications. The research team will first acquire, in-house, a high quality, small group conversational motion dataset that includes acoustic speech, facial expression, gaze, head movement, hand gesture, and body movement. Based on this dataset, the project will work towards the objectives by focusing on three inter-related research thrusts. The first research thrust is to develop a computational analysis model to extract and compare meaningful gaze patterns for turn-taking and turn-keeping in small group conversations. With the insight obtained from the first thrust, the second research thrust is to design a holistic, deep learning based framework to efficiently generate or synthesize life-like small group conversational animations based on various user inputs. The third research thrust is to comprehensively evaluate the effectiveness and usefulness of the proposed framework via two selected applications: small group teleconferencing within virtual worlds, and crowd simulations with group conversations. The synthesis framework can potentially transform development of applications that deal with virtual human crowds such as the ones in shopping plazas, downtown streets, and airport waiting areas.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.