CRII: HCC: Modeling computer-mediated task-oriented dialogues with multi-modality information theoretic approaches

CRII：HCC：用多模态信息理论方法对计算机介导的面向任务的对话进行建模

• 批准号: 2105192
• 负责人: Yang Xu
• 金额: $ 16.69万
• 依托单位: San Diego State University Foundation
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-06-01 至 2024-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2105192&HistoricalAwards=false
• 关键词: CRII; HCC; Modeling; computer; mediated

Effective real-time remote communication is challenging because unlike face-to-face conversations, information is exchanged indirectly through computer screens and speakers. In these situations, rich non-verbal information such as body poses, hand gestures, and facial expressions is often blocked or neglected. In order to improve the quality of existing remote communication systems, this project will develop an in-depth quantitative understanding of the kinds of information non-verbal aspects of communication might add to the conversation. The research team will address this issue by developing novel measures of non-verbal communication content based on state-of-the-art machine learning and computer vision techniques. The team will also develop software toolkits that can support both behavioral scientists who study conversation and designers of future computer-mediated communication systems. The research team proposes a series of studies that use information theoretic models to distill information from the non-verbal communication channels, including facial expressions, upper-body gestures, and whole-body poses, and synthesize them into the ultimate goal of predicting the success of computer-mediated collaborative tasks. As part of the work the team will develop several novel analysis methods. These include discretizing and symbolizing non-verbal information through joint representation learning and aggregation, using neural sequential models to estimate the amount of information in parallel discretized series, and using the temporal-spectral patterns in all communication channels to predict the task success. A multimodal information incorporated corpus of task-oriented dialogues will also be collected as part of the work.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

有效的实时远程通信具有挑战性，因为与面对面的对话不同，信息是通过计算机屏幕和扬声器间接交换的。在这些情况下，丰富的非语言信息，如身体姿势、手势和面部表情，往往被阻止或忽视。为了提高现有远程通信系统的质量，该项目将对通信的非语言方面可能添加到对话中的信息类型进行深入的量化理解。研究小组将通过基于最先进的机器学习和计算机视觉技术开发非语言交流内容的新措施来解决这一问题。该团队还将开发软件工具包，既可以支持研究对话的行为科学家，也可以支持未来计算机中介交流系统的设计者。该研究小组提出了一系列研究，利用信息论模型从非语言交流渠道中提取信息，包括面部表情、上半身手势和全身姿势，并将其综合为预测计算机中介协作任务成功的最终目标。作为工作的一部分，该团队将开发几种新的分析方法。这包括通过联合表征学习和聚合对非语言信息进行离散化和符号化，使用神经序列模型估计并行离散化序列中的信息量，以及使用所有通信渠道的时间-谱模式来预测任务成功。作为工作的一部分，还将收集包含以任务为导向的对话语料库的多模式信息。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Spontaneous gestures encoded by hand positions improve language models: An Information-Theoretic motivated study

由手部位置编码的自发手势改善了语言模型：一项信息理论驱动的研究

• 发表时间: 2023
• 期刊: Proceedings of the conference Association for Computational Linguistics Meeting
• 作者: Xu, Yang