CAREER: Learning and Optimization for Robust Multimodal Interpretation in Conversation Systems

职业：对话系统中稳健的多模态解释的学习和优化

• 批准号: 0347548
• 负责人: Joyce Chai
• 金额: $ 50万
• 依托单位: Michigan State University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2004
• 资助国家: 美国
• 起止时间: 2004-02-01 至 2010-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0347548&HistoricalAwards=false
• 关键词: CAREER; Learning; Optimization; Robust; Multimodal

Multimodal systems allow users to interact with computers through multiple modalities such as speech, gesture, and gaze. One challenge for multimodal systems is multimodal interpretation, which is the process of understanding what a user intends to communicate. Despite recent progress in multimodal interpretation, most systems still have problems handling unexpected or unreliable user inputs.This project seeks to improve the robustness of multimodal interpretation through two objectives: 1) to adapt system interpretation capability over time through automated knowledge acquisition, 2) to optimize interpretation through probabilistic reasoning. Specifically, this research will develop supervised and unsupervised learning approaches to automatically acquire knowledge, both offline from annotated data and online from real time interactions. Furthermore, this project will develop mechanisms to account for uncertainties that occur at different stages in the interpretation process and to derive an optimal interpretation through probabilistic reasoning about context and user intent. To support these approaches, large corpora of multimodal data will be collected from human machine conversation and annotated in terms of user intent and interaction context.This research will provide a benchmark for algorithmic advancement and evaluation. The enhanced robustness and reliability in multimodal interpretation will make multimodal systems more effective for real world applications. Through a multimodal conversation system that helps students explore options for undergraduate and graduate study at various institutions and locations, the results of this research will be directly applied to an education plan that includes outreach activities, curriculum development, and student mentoring. This tight integration of research and education will offer a unique multidisciplinary opportunity to synergize other programs within the university such as speech processing, psycholinguistics, and cognitive science.

多模态系统允许用户通过多种模态（如语音、手势和凝视）与计算机交互。多模态系统的一个挑战是多模态解释，这是理解用户意图传达的过程。尽管近年来多模态口译取得了一定的进展，但大多数系统在处理意外或不可靠的用户输入时仍然存在问题，本项目试图通过两个目标来提高多模态口译的鲁棒性：1）通过自动知识获取来适应系统随时间的口译能力，2）通过概率推理来优化口译。具体来说，这项研究将开发监督和无监督学习方法来自动获取知识，离线从注释数据和在线从真实的时间互动。此外，该项目将开发机制，以解释在解释过程的不同阶段出现的不确定性，并通过对上下文和用户意图的概率推理得出最佳解释。 为了支持这些方法，将从人机对话中收集大量多模态数据，并根据用户意图和交互上下文进行注释。这项研究将为算法的改进和评估提供基准。多模态解释中增强的鲁棒性和可靠性将使多模态系统对于真实的世界应用更有效。通过多模式对话系统，帮助学生探索在不同机构和地点的本科和研究生学习的选择，这项研究的结果将直接应用于教育计划，包括推广活动，课程开发和学生辅导。这种研究和教育的紧密结合将提供一个独特的多学科机会，协同大学内的其他项目，如语音处理，心理语言学和认知科学。