Eye Gaze in Salience Modeling for Robust Spoken Language Understanding

在口语对话系统中，由于有限的语音识别和语言理解性能，解释用户语音输入仍然是一个重大挑战。如果用户有口音或在嘈杂的环境中讲话，这个问题会进一步放大。然而，先前的研究表明，在多模态系统中，融合两个或多个信息源可以是减少识别不确定性的有效手段，例如通过相互消歧。受早期多模态系统工作的启发，在这个项目中，PI将研究眼睛注视在人机对话中的作用，特别是在强大的口语理解的显著性建模方面。认知研究表明，人眼注视是一个人“在想什么”的可靠指标之一。具体来说，眼睛注视与人类语言处理密切相关。先前的心理语言学研究表明，在听到一个单词后，眼睛几乎立即转移到相应的现实参照物上。在说一个词之前，眼睛也会移到提到的物体上。目光凝视不仅是高度可靠的，它也是一种隐含的、潜意识的语言反射。用户不需要做出有意识的决定；眼睛会自动移动到相关的物体上，用户甚至都没有意识到。在这些心理语言学发现的激励下，PI的假设是，在人机对话中，用户的眼睛注视信息与对话上下文相结合，可以向物理世界的一部分（与领域和图形界面相关）发出信号，这些信息在每个通信点上都是最突出的，因此它可以潜在地用于定制语音输入的解释。基于这一假设，PI将寻求通过一个新的基于显著性的框架来提高会话界面中的口语理解，其目标有两个：(1)更好地理解眼睛注视在人类语言产生中的作用及其对自动输入解释的显著性建模的影响；(2)开发算法和系统，将基于计算凝视的显著性建模应用于稳健的口语理解。这些目标将在以下四个方向上实现：(a)通过心理语言学研究调查人眼注视的效用及其对人机对话期间显著性建模的影响；(b)开发计算显著性模型，将眼睛注视与谈话内容结合起来，以便在每个通信点自动识别物理世界的显著部分；(c)发展应用新显著性模型的方法，以限制健全口语理解的假设空间；(d)评估两种不同应用程序中新方法的通用性：基于3D渲染界面的室内设计/培训应用程序，以及使用基于2D地图的界面的信息搜索应用程序。更广泛的影响：在这个跨学科项目中开发的技术可以应用于许多应用，例如虚拟培训系统，用户可以同时看到界面并与计算机系统交谈。这些技术将使各种不同的用户受益，特别是那些无法用手与图形界面交互的个人（例如，运动障碍用户）。由于这项工作的一个主要应用领域是电子培训和电子学习，因此拟议研究的教育和推广影响可能是深远的；今后，将致力于把研究成果转化为课堂教学。”该项目还将为计算机科学、心理学和认知科学的学生提供一个独特的合作机会，从而将密歇根州立大学的多学科研究活动协同起来。

In spoken dialog systems, interpreting user speech input is still a significant challenge due to limited speech recognition and language understanding performance. This problem is further amplified if a user has an accent or is speaking in a noisy environment. However, previous research has shown that, in multimodal systems, fusing two or more information sources can be an effective means of reducing recognition uncertainties, for example through mutual disambiguation. Inspired by earlier work on multimodal systems, in this project the PI will investigate the role of eye gaze in human machine conversation, in particular in salience modeling for robust spoken language understanding. Cognitive studies have shown that human eye gaze is one of the reliable indicators of what a person is "thinking about." Specifically, eye gaze is tightly linked to human language processing. Previous psycholinguistic work has shown that almost immediately after hearing a word, the eyes move to the corresponding real-world referent. And right before speaking a word, the eyes also move to the mentioned object. Not only is eye gaze highly reliable, it is also an implicit, subconscious reflex of speech. The user does not need to make a conscious decision; the eye automatically moves towards the relevant object, without the user even being aware. Motivated by these psycholinguistic findings, the PI's hypothesis is that during human machine conversation user eye gaze information coupled with conversation context can signal a part of the physical world (related to the domain and the graphical interface) that is most salient at each point of communication, thus it can potentially be used to tailor the interpretation of speech input. Based on this hypothesis, the PI will seek to improve spoken language understanding in conversational interfaces through a new salience-based framework with two objectives: (1) To better understand the role of eye gaze in human language production and its implications in salience modeling for automated input interpretation; and (2) To develop algorithms and systems that apply computational gaze based salience modeling to robust spoken language understanding. These objectives will be pursued in the following four directions: (a) Investigation of the utility of human eye gaze and its implications for salience modeling during human machine conversation through psycholinguistic studies; (b) Development of computational salience models that integrate eye gaze with conversation context to automatically identify a salient part of the physical world at each point of communication; (c) Development of approaches that apply the new salience models to constrain the hypothesis space for robust spoken language understanding; and (d) Evaluation of the generality of the new approaches in two different applications: an interior design/training application based on a 3D rendered interface, and an information seeking application using a 2D map-based interface.Broader Impacts: The technologies to be developed in this interdisciplinary project can be applied to many applications such as virtual training systems where users can see the interface and talk to the computer system at the same time. The technologies will benefit a variety of diverse users, and particularly individuals who are unable to interact with graphical interfaces with their hands (e.g., motion disabled users). Since one major application area of the work is e-training and e-learning, the education and outreach impact of the proposed research is potentially profound; the PI will make specific efforts to transfer the research results into classrooms. The project will also provide a unique opportunity for students in Computer Science, Psychology, and Cognitive Science to work together, and thus will synergize multidisciplinary research activities at Michigan State University.