CAREER: Co-analysis of Signal and Sense for Understanding Non-verbal Communications and their Applications

本研究的目的是促进我们对韵律关系及其在言语交际和非言语交际模式之间的同步性的理解。韵律和肢体动作，如手势、点头、面部表情和姿势，通过增加表现力和组织信息在日常交流中发挥着至关重要的作用。尽管这些模式之间有不同类型和不同程度的同步，但它们的准确映射仍然不清楚。到目前为止，言语情态和非言语情态之间的跨情态同步主要是在语义和语篇层面上进行的，而在这部著作中，PI将关注它们之间在不同粒度水平上的相互作用。特别是，使用语言和话语模型对语音和运动进行联合分析将被用来发现韵律对应，这反过来将被用来开发用于模拟对话行为、情感和其他运动的新算法。该项目的两个主要目标是开发计算方法和软件工具，以迭代地发现非语言行为和语言行为之间的韵律关系，并使用导出的韵律关系及其同步来开发新的计算方法和软件工具，用于健壮地识别手势、面部表情、情绪、头部和对话行为。PI希望借此为多通道发音的共同分析框架奠定基础，从而更深入地了解(A)话语的核心如何与视觉韵律相互作用以呈现话语的意图，以及(B)与其他通道的同步如何影响多通道协同发音的产生。为了进一步提高分类器的鲁棒性和识别精度，考虑到分类器之间的多样性，设计了一组分类器并进行了融合。将制定系统的方法，使用各种业绩衡量标准(即准确度、查全率、F-MEASURES)、图形分析和衡量函数对分类器进行评价。这项研究的结果与PI之前的工作一起，最终将使AutoTutor(一个人工智能的基于Web的教学系统)的感知界面的开发成为可能，为教学提供一种与多媒体内容交互的自然手段。广泛的影响：本研究的结果将对理解和跟踪人类和代理之间的多模式交流产生深远的影响。它们同步的互补形式和韵律表现之间的相互作用也将扩大对认知科学、话语处理、语言学和人机交互中的多渠道交流的理解，这将使创新应用的开发成为可能，如代理人和人类的协作环境，以及老年人和残疾人的辅助技术。这项拟议研究的长期愿景是为AutoTutor等基于网络的教学系统开发一个感知界面。使用基于网络的增强型人工智能家教为孟菲斯城市学校和其他地区或国家客户的即将入学的工程和理科本科生提供了改善数学和科学准备的重要机会。PI还将使用Web 2.0等较新的框架创建在线协作学习环境，以组织大量数字内容，使学习者社区可以有效地共享和共同管理信息。作为该项目一部分开发的软件和数据库将通过该项目网站向其他研究人员提供。

The objective of this research is to advance our understanding of prosodic relationships and their synchronizations between verbal and nonverbal communication modes. Prosody and kinesics such as hand gestures, head nods, facial expressions, and posture, play a crucial role in everyday communication by adding expressiveness, as well as by structuring information. Although there are different types and various levels of synchronization across these modalities, their exact mapping remains unclear. Whereas cross-modal synchronization between verbal and nonverbal modalities has been explored mostly at the semantic and discourse levels to date, in this work the PI will focus on the interplay between them at various levels of granularity. In particular, co-analyses of speech, using language and discourse models, with kinesics will be used to uncover prosodic correspondences which, in turn, will be used to develop novel algorithms for modeling dialog acts, emotions and other kinesics. The two primary goals of the project are to develop computational methods and software tools to iteratively uncover prosodic relationships between nonverbal and verbal behaviors, and to use derived prosodic relationships and their synchronizations to develop novel computational methods and software tools for the robust recognition of gestures, facial expressions, emotions, head nods, and dialog acts. The PI hopes to thereby lay the foundation for a framework for co-analysis of multimodal articulations to obtain a deeper understanding of (a) how the nucleus of an utterance and visual prosody interact to render the intent of the utterance, and (b) how synchronization with other modalities affects the production of multimodal co-articulation. To further improve the robustness and recognition accuracies, a set of classifiers will be designed and fused by taking into account the diversity among them. Systematic methods will be developed to evaluate the classifiers using various performance metrics (i.e., precision, recall, F-measures), graphical analyses and measure functions. The outcomes of the research, together with the PI's prior work, will ultimately enable the development of a perceptual interface for AutoTutor (an artificially intelligent web-based tutoring system), providing a natural means to interact with multimedia contents for instruction.Broader Impact: The results of this research will have profound impact on the understanding and tracking of multimodal communications in humans and agents. The interplay between the complementary modalities and prosodic manifestations of their synchronization will also broaden the understanding of multi-channel communications in cognitive science, discourse processing, linguistics, and human-machine interaction, which will enable the development of innovative applications such as collaborative environments for agents and humans, and assistive technologies for the elderly and disabled. The long-term vision of the proposed research is to develop a perceptual interface for web-based tutoring systems such as AutoTutor. Use of an enhanced artificially intelligent web-based tutor offers significant opportunities for improving the math and science preparation of incoming engineering and science undergraduates of the Memphis City Schools and other regional or national clients. The PI will also create an online collaborative learning environment, using newer frameworks such as Web 2.0, to organize a massive amount of digital contents in such a way that communities of learners can effectively share and co-manage the information. The software and databases developed as part of this project will be made available to other researchers through the project website.