CAREER: Learning and Optimization for Robust Multimodal Interpretation in Conversation Systems

多模态系统允许用户通过多种模态与计算机交互，例如语音、手势和凝视。多模态系统面临的一个挑战是多模态解释，这是理解用户想要交流什么的过程。尽管最近在多模态解释方面取得了进展，但大多数系统在处理意外或不可靠的用户输入时仍然存在问题。本项目旨在通过两个目标来提高多模态解释的鲁棒性：1)通过自动化知识获取来适应系统解释能力，2)通过概率推理来优化解释。具体而言，本研究将开发有监督和无监督学习方法，以自动获取知识，包括离线从注释数据和在线从实时交互。此外，该项目将开发机制来解释解释过程中不同阶段发生的不确定性，并通过关于上下文和用户意图的概率推理得出最佳解释。为了支持这些方法，将从人机对话中收集多模态数据的大型语料库，并根据用户意图和交互上下文进行注释。本研究将为算法的发展和评估提供一个基准。在多模态解释中增强的鲁棒性和可靠性将使多模态系统在实际应用中更加有效。通过一个多模式对话系统，帮助学生在不同的机构和地点探索本科和研究生学习的选择，这项研究的结果将直接应用于包括外展活动、课程开发和学生指导在内的教育计划。这种研究和教育的紧密结合将提供一个独特的多学科机会，以协同大学内的其他项目，如语音处理、心理语言学和认知科学。

Multimodal systems allow users to interact with computers through multiple modalities such as speech, gesture, and gaze. One challenge for multimodal systems is multimodal interpretation, which is the process of understanding what a user intends to communicate. Despite recent progress in multimodal interpretation, most systems still have problems handling unexpected or unreliable user inputs.This project seeks to improve the robustness of multimodal interpretation through two objectives: 1) to adapt system interpretation capability over time through automated knowledge acquisition, 2) to optimize interpretation through probabilistic reasoning. Specifically, this research will develop supervised and unsupervised learning approaches to automatically acquire knowledge, both offline from annotated data and online from real time interactions. Furthermore, this project will develop mechanisms to account for uncertainties that occur at different stages in the interpretation process and to derive an optimal interpretation through probabilistic reasoning about context and user intent. To support these approaches, large corpora of multimodal data will be collected from human machine conversation and annotated in terms of user intent and interaction context.This research will provide a benchmark for algorithmic advancement and evaluation. The enhanced robustness and reliability in multimodal interpretation will make multimodal systems more effective for real world applications. Through a multimodal conversation system that helps students explore options for undergraduate and graduate study at various institutions and locations, the results of this research will be directly applied to an education plan that includes outreach activities, curriculum development, and student mentoring. This tight integration of research and education will offer a unique multidisciplinary opportunity to synergize other programs within the university such as speech processing, psycholinguistics, and cognitive science.