CAREER: Multimodal Language Processing for Natural Interfaces

通过提供在用户和计算机之间传递信息的多个通道，多模式界面有望显著增加人与机器之间界面的带宽和流动性。自然语言处理技术通常集中于处理单个维度内的单词串。这一研究项目旨在开发一种多模式语言处理的方法，该方法能够对分布在语音的两个或三个空间维度、时间维度和声学维度上的自然人类输入进行解析和解释。提出了一种基于声明性统一的文法表示法来描述多通道集成策略。该表示由多维解析器解释，该多维解析器产生一种体系结构，该体系结构为通过语音、笔、手势和其他模式支持人机交互的下一代自然接口提供基础。多模式界面方面的根本性进展对于成功地将人机交互从台式计算机转移到小型移动计算设备、公共信息亭和其他复杂的图形用户界面不可行的平台至关重要。多模式接口在实现普遍获取信息的愿景和为残疾人开发辅助技术方面都将发挥关键作用。

By providing a number of channels through which information may pass between user and computer, multimodal interfaces promise to significantly increase the bandwidth and fluidity of the interface between humans and machines. Natural language processing techniques typically focus on processing strings of words within a single dimension. This program of research seeks to develop an approach to multimodal language processing which enables parsing and interpretation of natural human input distributed across two or three spatial dimensions, time, and the acoustic dimension of speech. A declarative unification-based grammar representation is proposed for the description of multimodal integration strategies. This representation is interpreted by a multidimensional parser yielding an architecture which provides the foundation for next generation natural interfaces supporting human-computer interaction through speech, pen, gesture, and other modes. Fundamental advances in multimodal interfaces are essential to the successful migration of human-computer interaction from the desktop to small mobile computing devices, public information kiosks, and other platforms for which complex graphical user interfaces are not feasible. Multimodal interfaces stand to play a critical role both in realizing the vision of universal access to information and in the development of assistive technologies for people with disabilities.