CRII: HCC: Modeling computer-mediated task-oriented dialogues with multi-modality information theoretic approaches

有效的实时远程通信具有挑战性，因为与面对面的对话不同，信息是通过计算机屏幕和扬声器间接交换的。 在这些情况下，丰富的非语言信息，如身体姿势，手势和面部表情往往被封锁或忽视。为了提高现有远程通信系统的质量，本项目将对通信的非语言方面可能添加到对话中的信息种类进行深入的定量理解。研究团队将通过基于最先进的机器学习和计算机视觉技术开发非语言交流内容的新措施来解决这个问题。该团队还将开发软件工具包，以支持研究对话的行为科学家和未来计算机中介通信系统的设计者。该研究小组提出了一系列研究，使用信息理论模型从非语言沟通渠道中提取信息，包括面部表情，上身姿势和全身姿势，并将其合成为预测计算机介导的协作任务成功的最终目标。 作为工作的一部分，该团队将开发几种新的分析方法。 这些方法包括通过联合表征学习和聚合将非语言信息离散化和符号化，使用神经序列模型来估计并行离散序列中的信息量，以及使用所有通信信道中的时间谱模式来预测任务成功。作为工作的一部分，还将收集多模态信息，包括面向任务的对话语料库。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

Effective real-time remote communication is challenging because unlike face-to-face conversations, information is exchanged indirectly through computer screens and speakers. In these situations, rich non-verbal information such as body poses, hand gestures, and facial expressions is often blocked or neglected. In order to improve the quality of existing remote communication systems, this project will develop an in-depth quantitative understanding of the kinds of information non-verbal aspects of communication might add to the conversation. The research team will address this issue by developing novel measures of non-verbal communication content based on state-of-the-art machine learning and computer vision techniques. The team will also develop software toolkits that can support both behavioral scientists who study conversation and designers of future computer-mediated communication systems. The research team proposes a series of studies that use information theoretic models to distill information from the non-verbal communication channels, including facial expressions, upper-body gestures, and whole-body poses, and synthesize them into the ultimate goal of predicting the success of computer-mediated collaborative tasks. As part of the work the team will develop several novel analysis methods. These include discretizing and symbolizing non-verbal information through joint representation learning and aggregation, using neural sequential models to estimate the amount of information in parallel discretized series, and using the temporal-spectral patterns in all communication channels to predict the task success. A multimodal information incorporated corpus of task-oriented dialogues will also be collected as part of the work.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.