Collaborative Research: CCRI: New: An Open Data Infrastructure for Bodily Expressed Emotion Understanding

合作研究：CCRI：新：用于理解身体表达情绪的开放数据基础设施

• 批准号: 2234195
• 负责人: James Wang
• 金额: $ 183.23万
• 依托单位: Pennsylvania State Univ University Park
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-04-15 至 2026-03-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2234195&HistoricalAwards=false
• 关键词: Collaborative; Research; CCRI; New; Open

The project goal is to unlock the wealth of information about human expression that is already found in videos on the internet. The multidisciplinary project team will collect videos of human movement available online and use experts in movement analysis and non-experts to pinpoint at characteristics of the human movement that can be used to drive algorithms that will attempt to classify the emotion expressed by the human mover. These characteristics will form labels on the data that include context, demographics, technical concepts from movement analysis, and emotion. This work will take an unprecedented, multidisciplinary approach in creating a data infrastructure for computational modeling of bodily expression of emotion. To ensure the infrastructure's compatibility with human-robot interaction research, the team will conduct a public-facing feasibility study. The team will also employ advisory boards and continue to engage with active researchers in multiple sub-disciplines of the computer and information science and engineering research community in the designing, creation, testing, and dissemination of the data infrastructure, and organizing annual user community workshops and benchmarking challenges. The data infrastructure is expected to promote technological innovations and breakthroughs in data-driven modeling of human bodily expression of emotion and affect, a highly complex problem with applications in healthcare, e.g., caregiving robots and diagnostic tools for mental health, manufacturing, e.g., socially-aware autonomous forklifts and safety monitoring systems, security, e.g., monitoring, and consumer electronics, e.g., improved interactions with a home robot.Bodily movement expresses important information, including conveying emotion, which is crucial for future human-machine interactions. As in other areas of artificial intelligence (AI), such as image recognition, a large-scale data-driven approach holds promise for revealing new insights into the complex, subtle, and contextual nature of human bodily expression. However, research on computational recognition of bodily expression, an area of affective computing, AI, and human-robot interaction, is struggling to mature as researchers must replicate many of the same work-intensive steps, creating divergent efforts and expense. This NSF project aims to create a large-scale, high-quality, multifaceted, annotated, open, and extensible data infrastructure for computational understanding of human bodily expressions in a variety of settings. It will leverage the team's expertise in AI, computer vision, affective computing, expressive robotics, emotion recognition, psychology, movement analysis, statistics and data mining, data ethics, and the arts to create (1) a data-sharing infrastructure tailored to the needs of research into subjective experience, emotion, and bodily movement, (2) a crowdsourced annotated video dataset, and (3) a collection of tools and software for rigorous reliability validation, reproducibility and transparency assessment, and content-based search and retrieval. The data infrastructure is expected to serve applications in fields such as robotics, psychology, performing arts, animation, and entertainment. The project also develops human expertise in this emerging field by supporting graduate and undergraduate students, including students from underrepresented groups, providing experience in conducting infrastructure development, integrating knowledge from multiple disciplines. These students will interact regularly with the team’s international partners. Public events that create broad public engagement in the work will focus on numerous applications to human-robot interaction. The infrastructure will stimulate focused research projects and agendas in affective computing, AI, including artificial emotional intelligence and human-AI interaction, computer vision, social/assistive robotics, virtual agents, psychiatric telemedicine, human-centered design, machine/deep learning, ethics in computing, and related communities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目的目标是解锁已经在互联网上的视频中发现的关于人类表达的丰富信息。多学科项目团队将收集在线提供的人类运动视频，并使用运动分析专家和非专家来确定人类运动的特征，这些特征可用于驱动算法，这些算法将尝试对人类运动者表达的情感进行分类。这些特征将在数据上形成标签，包括上下文、人口统计学、来自运动分析的技术概念和情感。这项工作将采取前所未有的多学科方法，为情感的身体表达的计算建模创建数据基础设施。为了确保基础设施与人机交互研究的兼容性，该团队将进行面向公众的可行性研究。该团队还将聘请咨询委员会，并继续与计算机和信息科学与工程研究界多个子学科的活跃研究人员合作，设计，创建，测试和传播数据基础设施，并组织年度用户社区研讨会和基准测试挑战。数据基础设施预计将促进技术创新和突破，在数据驱动的人体情感表达建模和影响，一个高度复杂的问题，在医疗保健中的应用，例如，为心理健康、制造业等提供机器人和诊断工具，具有社会意识的自动叉车和安全监控系统，安全，例如，监控和消费电子产品，例如，改善与家用机器人的交互。身体运动表达了重要的信息，包括传达情感，这对未来的人机交互至关重要。与人工智能（AI）的其他领域（如图像识别）一样，大规模数据驱动的方法有望揭示人类身体表达的复杂，微妙和上下文性质的新见解。然而，关于身体表情的计算识别（情感计算、人工智能和人机交互的一个领域）的研究正在努力成熟，因为研究人员必须复制许多相同的工作密集型步骤，从而产生不同的努力和费用。这个NSF项目旨在创建一个大规模，高质量，多方面，注释，开放和可扩展的数据基础设施，用于在各种环境中对人体表情的计算理解。它将利用团队在人工智能，计算机视觉，情感计算，表达机器人，情感识别，心理学，运动分析，统计和数据挖掘，数据伦理和艺术方面的专业知识来创建（1）一个数据共享基础设施，以满足研究主观体验，情感和身体运动的需求，（2）一个众包注释视频数据集，以及（3）用于严格的可靠性验证、再现性和透明度评估以及基于内容的搜索和检索的工具和软件的集合。数据基础设施预计将服务于机器人、心理学、表演艺术、动画和娱乐等领域的应用。该项目还通过支持研究生和本科生，包括来自代表性不足群体的学生，提供基础设施开发经验，整合多学科知识，发展这一新兴领域的人力专业知识。这些学生将定期与团队的国际合作伙伴互动。创造广泛公众参与工作的公共活动将集中在人机交互的众多应用上。该基础设施将刺激情感计算，人工智能，包括人工情感智能和人机交互，计算机视觉，社交/辅助机器人，虚拟代理，精神病远程医疗，以人为本的设计，机器/深度学习，计算伦理，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准。

项目成果

A Machine Learning Paradigm for Studying Pictorial Realism: How Accurate are Constable's Clouds?

• DOI: 10.1109/tpami.2023.3324743
• 发表时间: 2022-02
• 期刊: IEEE Transactions on Pattern Analysis and Machine Intelligence
• 影响因子: 23.6
• 作者: Zhuomin Zhang;Elizabeth C. Mansfield;Jia Li;John Russell;George S. Young;Catherine Adams;Kevin A Bowley;James Z. Wang

Tutorial on Movement Notation: An Interdisciplinary Methodology for HRI to Reveal the Bodily Expression of Human Counterparts via Collecting Annotations from Dancers in a Shared Data Repository

动作注释教程：HRI 的跨学科方法，通过在共享数据存储库中收集舞者的注释来揭示人类对应者的身体表达

• 发表时间: 2024
• 期刊: Proceedings of the Annual ACM/IEEE International Conference on Human Robot Interaction Companion
• 作者: LaViers, Amy;Maguire, Cat;Wang, James Z.;Tsachor, Rachelle
• 通讯作者: Tsachor, Rachelle