CAREER: Modeling and Optimizing User-Centric Mixed-Initiative Spoken Dialog Systems

对话式界面有望提供自然、简单和通用的信息访问。 本研究在一个统一的随机建模框架下，针对会话工程中的两个具体问题：（1）对话交互建模（2）口语对话中的用户建模。这些问题是相互关联的：用户模型是设计最佳对话策略的基础，而会话状态信息是用户行为建模的基础。会话参与者-人类或机器-被建模为随机动态系统，通过嘈杂的通信信道相互交互。信道中的噪声可能是由于机器语音识别、语言理解中的错误或其他机器/用户通信错误。基于数学博弈论和鲁棒设计的算法推导出最优的混合主动对话控制策略。建模用户行为的方法是数据驱动的，重点是错误条件下的行为和自动情感跟踪。同样，统一的统计框架提供了一种整合多种信息来源的方法（例如，声学、词汇、非语言和话语）。研究开发将基于来自各种真实的应用领域的真实的人机对话的注释口语语料库，包括航空旅行和呼叫中心应用以及南加州大学（USC）正在开发的军事/安全应用的沉浸式培训系统。一个两学期的研究生课程，同时在校园和南加州大学"远程教育网络”上提供，将通过语音识别、语音合成和口语对话的最新进展，以基本语音处理为目标。这些课程与广泛的课程有关，除了核心电气工程，包括计算语言学，计算机科学智能系统，综合媒体和创意技术，语音学和心理学。 此外，一个独特的跨学科的本科课程，以人类与机器处理口语为主题将开发和实施。研究和教育部分紧密结合，课程通过演示和实践项目介绍最新研究成果，强调“边做边学”。 同样，研究是教育的有效工具，可以培养本科生和博士生。这个口语工程项目在其范围和社会相关性方面是独一无二的，涵盖口语研究的基础和应用方面，包括行业视角，并与大学内的多媒体，自然语言处理和代理技术的其他活动产生协同作用。它将作为促进跨学科研究生和本科生研究和教育的工具。

Conversational interfaces hold the promise of providing natural, easy, and universal access to information. This project targets two specific problems in conversational engineering under a unifying stochastic modeling framework: (1) dialog interaction modeling (2) user modeling in spoken dialog. These problems are related: user models are essential in designing optimal dialog strategies, while discourse state information is essential for user behavior modeling. Conversational participants -- humans or machines -- are modeled as stochastic dynamical systems, interacting with one another over a noisy communication channel. Noise in the channel may be due to errors in machine speech recognition, language understanding, or other machine/user communication errors. Algorithms based on mathematical game theory and robust design are formulated for derivation of optimal mixed-initiative dialog control strategies. The approach to modeling user behavior is data-driven with an emphasis on behavior under error conditions and on the inclusion of automatic emotion tracking. Again, a unified statistical framework provides a way for integrating multiple sources of information (e.g., acoustic, lexical, nonverbal and discourse) based on information-theoretic principles. The research development will be based on annotated spoken language corpora from real human-machine dialogs from a variety of real application domains including air travel and call center applications and an immersive training system for military/security applications being developed at the University of Southern California (USC).The educational component creates a new curriculum in spoken language processing: a two-semester graduate sequence, developed for simultaneous offering both on campus and on USC ``Distance Education Network" will target basic speech processing through the latest advances in speech recognition, speech synthesis, and spoken dialog. The courses are relevant to a wide range of programs, in addition to core electrical engineering, including computational linguistics, intelligent systems in computer science, integrated media and creative technologies, phonetics, and psychology. In addition, a unique interdisciplinary undergraduate course with a theme on human versus machine processing of spoken language will be developed and implemented. The research and education components are tightly integrated, with courses featuring the latest research results through demonstrations and hands-on projects, with an emphasis on `learning by doing'. Likewise, research serves as an effective vehicle for education in terms of enabling the training of both undergraduate and doctoral students. This project in spoken language engineering is unique in terms of its scope and societal relevance, spanning both fundamental and applied aspects of spoken language research, including an industry perspective, and creating synergies with other activities within the university in multimedia, natural language processing, and agent technology. It will serve as a vehicle to promote cross-disciplinary graduate and undergraduate research and education.