CAREER: Learning and Optimization for Robust Multimodal Interpretation in Conversation Systems

多模式系统允许用户通过语音、手势和凝视等多种模式与计算机交互。多通道系统的一个挑战是多通道解释，这是理解用户打算交流什么的过程。尽管最近在多模式解释方面取得了进展，但大多数系统在处理意外或不可靠的用户输入时仍然存在问题。本项目试图通过两个目标来提高多模式解释的稳健性：1)通过自动知识获取来适应系统的解释能力；2)通过概率推理来优化解释。具体地说，这项研究将开发监督和非监督学习方法来自动获取知识，包括离线从注释数据获取知识，以及在线从实时交互获取知识。此外，该项目将制定机制，以解决口译过程不同阶段出现的不确定性，并通过对背景和用户意图的概率推理得出最佳口译。为了支持这些方法，将从人机对话中收集大量多模式数据，并根据用户意图和交互上下文进行标注。本研究将为算法改进和评估提供基准。多通道解释的增强的稳健性和可靠性将使多通道系统在实际应用中更有效。通过多模式对话系统，帮助学生探索在不同机构和地点进行本科生和研究生学习的选择，这项研究的结果将直接应用于包括外展活动、课程开发和学生指导在内的教育计划。这种研究和教育的紧密结合将提供一个独特的多学科机会，以协同大学内的其他课程，如语音处理、心理语言学和认知科学。

Multimodal systems allow users to interact with computers through multiple modalities such as speech, gesture, and gaze. One challenge for multimodal systems is multimodal interpretation, which is the process of understanding what a user intends to communicate. Despite recent progress in multimodal interpretation, most systems still have problems handling unexpected or unreliable user inputs.This project seeks to improve the robustness of multimodal interpretation through two objectives: 1) to adapt system interpretation capability over time through automated knowledge acquisition, 2) to optimize interpretation through probabilistic reasoning. Specifically, this research will develop supervised and unsupervised learning approaches to automatically acquire knowledge, both offline from annotated data and online from real time interactions. Furthermore, this project will develop mechanisms to account for uncertainties that occur at different stages in the interpretation process and to derive an optimal interpretation through probabilistic reasoning about context and user intent. To support these approaches, large corpora of multimodal data will be collected from human machine conversation and annotated in terms of user intent and interaction context.This research will provide a benchmark for algorithmic advancement and evaluation. The enhanced robustness and reliability in multimodal interpretation will make multimodal systems more effective for real world applications. Through a multimodal conversation system that helps students explore options for undergraduate and graduate study at various institutions and locations, the results of this research will be directly applied to an education plan that includes outreach activities, curriculum development, and student mentoring. This tight integration of research and education will offer a unique multidisciplinary opportunity to synergize other programs within the university such as speech processing, psycholinguistics, and cognitive science.