CAREER: Modeling and Optimizing User-Centric Mixed-Initiative Spoken Dialog Systems

对话界面有望提供对信息的自然、简单和通用的访问。本项目在统一的随机建模框架下，针对会话工程中的两个具体问题：(1)对话交互建模(2)口语对话中的用户建模。这些问题是相互关联的：用户模型是设计最佳对话策略的基础，而话语状态信息是用户行为建模的基础。会话参与者——人类或机器——被建模为随机动态系统，在嘈杂的通信信道上相互作用。信道中的噪声可能是由于机器语音识别、语言理解或其他机器/用户通信错误造成的。提出了基于数学博弈论和鲁棒设计的最优混合主动对话控制策略推导算法。建模用户行为的方法是数据驱动的，强调错误条件下的行为，并包含自动情绪跟踪。同样，一个统一的统计框架提供了一种基于信息论原则整合多种信息源（例如，声学、词汇、非语言和话语）的方法。该研究开发将基于来自各种实际应用领域的真实人机对话的注释口语语料库，包括航空旅行和呼叫中心应用，以及南加州大学（USC）正在开发的用于军事/安全应用的沉浸式培训系统。教育部分在口语处理方面创建了一个新的课程：为在校园里和南加州大学“远程教育网”同时提供的两个学期的研究生课程，将通过语音识别、语音合成和口语对话方面的最新进展，针对基本的语音处理。除了核心的电子工程外，这些课程还涉及广泛的专业，包括计算语言学、计算机科学中的智能系统、综合媒体和创意技术、语音学和心理学。此外，将开发和实施一门独特的跨学科本科课程，主题是人类与机器处理口语。研究和教育部分紧密结合，课程通过演示和实践项目展示最新的研究成果，强调“边做边学”。同样，研究也是培养本科生和博士生的有效手段。这个口语工程项目在其范围和社会相关性方面是独一无二的，涵盖了口语研究的基础和应用方面，包括行业视角，并与大学内多媒体，自然语言处理和代理技术的其他活动产生协同作用。它将作为促进跨学科研究生和本科生研究和教育的工具。

Conversational interfaces hold the promise of providing natural, easy, and universal access to information. This project targets two specific problems in conversational engineering under a unifying stochastic modeling framework: (1) dialog interaction modeling (2) user modeling in spoken dialog. These problems are related: user models are essential in designing optimal dialog strategies, while discourse state information is essential for user behavior modeling. Conversational participants -- humans or machines -- are modeled as stochastic dynamical systems, interacting with one another over a noisy communication channel. Noise in the channel may be due to errors in machine speech recognition, language understanding, or other machine/user communication errors. Algorithms based on mathematical game theory and robust design are formulated for derivation of optimal mixed-initiative dialog control strategies. The approach to modeling user behavior is data-driven with an emphasis on behavior under error conditions and on the inclusion of automatic emotion tracking. Again, a unified statistical framework provides a way for integrating multiple sources of information (e.g., acoustic, lexical, nonverbal and discourse) based on information-theoretic principles. The research development will be based on annotated spoken language corpora from real human-machine dialogs from a variety of real application domains including air travel and call center applications and an immersive training system for military/security applications being developed at the University of Southern California (USC).The educational component creates a new curriculum in spoken language processing: a two-semester graduate sequence, developed for simultaneous offering both on campus and on USC ``Distance Education Network" will target basic speech processing through the latest advances in speech recognition, speech synthesis, and spoken dialog. The courses are relevant to a wide range of programs, in addition to core electrical engineering, including computational linguistics, intelligent systems in computer science, integrated media and creative technologies, phonetics, and psychology. In addition, a unique interdisciplinary undergraduate course with a theme on human versus machine processing of spoken language will be developed and implemented. The research and education components are tightly integrated, with courses featuring the latest research results through demonstrations and hands-on projects, with an emphasis on `learning by doing'. Likewise, research serves as an effective vehicle for education in terms of enabling the training of both undergraduate and doctoral students. This project in spoken language engineering is unique in terms of its scope and societal relevance, spanning both fundamental and applied aspects of spoken language research, including an industry perspective, and creating synergies with other activities within the university in multimedia, natural language processing, and agent technology. It will serve as a vehicle to promote cross-disciplinary graduate and undergraduate research and education.