CAREER: Co-analysis of Signal and Sense for Understanding Non-verbal Communications and their Applications

职业：信号和感觉的共同分析以理解非语言交流及其应用

• 批准号: 0746790
• 负责人: Mohammed Yeasin
• 金额: $ 49.49万
• 依托单位: University of Memphis
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2008
• 资助国家: 美国
• 起止时间: 2008-05-01 至 2014-04-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0746790&HistoricalAwards=false
• 关键词: CAREER; Co; analysis; Signal; Sense

The objective of this research is to advance our understanding of prosodic relationships and their synchronizations between verbal and nonverbal communication modes. Prosody and kinesics such as hand gestures, head nods, facial expressions, and posture, play a crucial role in everyday communication by adding expressiveness, as well as by structuring information. Although there are different types and various levels of synchronization across these modalities, their exact mapping remains unclear. Whereas cross-modal synchronization between verbal and nonverbal modalities has been explored mostly at the semantic and discourse levels to date, in this work the PI will focus on the interplay between them at various levels of granularity. In particular, co-analyses of speech, using language and discourse models, with kinesics will be used to uncover prosodic correspondences which, in turn, will be used to develop novel algorithms for modeling dialog acts, emotions and other kinesics. The two primary goals of the project are to develop computational methods and software tools to iteratively uncover prosodic relationships between nonverbal and verbal behaviors, and to use derived prosodic relationships and their synchronizations to develop novel computational methods and software tools for the robust recognition of gestures, facial expressions, emotions, head nods, and dialog acts. The PI hopes to thereby lay the foundation for a framework for co-analysis of multimodal articulations to obtain a deeper understanding of (a) how the nucleus of an utterance and visual prosody interact to render the intent of the utterance, and (b) how synchronization with other modalities affects the production of multimodal co-articulation. To further improve the robustness and recognition accuracies, a set of classifiers will be designed and fused by taking into account the diversity among them. Systematic methods will be developed to evaluate the classifiers using various performance metrics (i.e., precision, recall, F-measures), graphical analyses and measure functions. The outcomes of the research, together with the PI's prior work, will ultimately enable the development of a perceptual interface for AutoTutor (an artificially intelligent web-based tutoring system), providing a natural means to interact with multimedia contents for instruction.Broader Impact: The results of this research will have profound impact on the understanding and tracking of multimodal communications in humans and agents. The interplay between the complementary modalities and prosodic manifestations of their synchronization will also broaden the understanding of multi-channel communications in cognitive science, discourse processing, linguistics, and human-machine interaction, which will enable the development of innovative applications such as collaborative environments for agents and humans, and assistive technologies for the elderly and disabled. The long-term vision of the proposed research is to develop a perceptual interface for web-based tutoring systems such as AutoTutor. Use of an enhanced artificially intelligent web-based tutor offers significant opportunities for improving the math and science preparation of incoming engineering and science undergraduates of the Memphis City Schools and other regional or national clients. The PI will also create an online collaborative learning environment, using newer frameworks such as Web 2.0, to organize a massive amount of digital contents in such a way that communities of learners can effectively share and co-manage the information. The software and databases developed as part of this project will be made available to other researchers through the project website.

本研究的目的是进一步了解韵律关系及其在言语和非言语交际模式之间的同步。 韵律和身势语，如手势、头部动作、面部表情和姿势，通过增加表达力和结构化信息在日常交流中发挥着至关重要的作用。 虽然这些模式之间有不同类型和不同程度的同步，但它们的确切映射仍然不清楚。 而跨模态之间的语言和非语言模态的同步主要是在语义和话语层面上进行了探索，在这项工作中，PI将集中在它们之间的相互作用，在不同的粒度水平。 特别是，共同分析的语音，使用语言和话语模型，与身势将被用来揭示韵律的对应关系，反过来，将被用来开发新的算法，用于建模对话行为，情感和其他身势。 该项目的两个主要目标是开发计算方法和软件工具，以迭代地揭示非语言和语言行为之间的韵律关系，并使用派生的韵律关系及其同步开发新的计算方法和软件工具，用于手势，面部表情，情绪，头部表情和对话行为的鲁棒识别。 PI希望借此奠定基础的多模态发音的共同分析框架，以获得更深入的理解（a）如何话语的核心和视觉韵律相互作用，以呈现话语的意图，（B）如何同步与其他模态影响生产的多模态共同发音。 为了进一步提高分类器的鲁棒性和识别精度，设计了一组分类器，并考虑到它们之间的差异进行融合。将开发系统方法，使用各种性能指标（即，精确度，召回率，F-测量），图形分析和测量功能。 该研究的成果，加上PI的前期工作，最终将使AutoTutor（一个人工智能的基于Web的辅导系统）的感知界面的开发，提供一个自然的手段与多媒体内容进行交互的指令。更广泛的影响：这项研究的结果将对人类和代理人的多模态通信的理解和跟踪产生深远的影响。 它们的同步的互补模态和韵律表现之间的相互作用也将扩大认知科学，话语处理，语言学和人机交互，这将使创新的应用程序，如代理和人类的协作环境，以及老年人和残疾人的辅助技术的发展多渠道通信的理解。 所提出的研究的长期愿景是开发一个感性的基于Web的辅导系统，如AutoTutor的接口。 使用一个增强的人工智能的基于网络的导师提供了显着的机会，以改善即将到来的工程和科学本科生的孟菲斯城市学校和其他区域或国家客户的数学和科学准备。 PI还将创建一个在线协作学习环境，使用Web 2.0等较新的框架来组织大量的数字内容，使学习者社区能够有效地共享和共同管理信息。 作为该项目一部分开发的软件和数据库将通过项目网站提供给其他研究人员。