CAREER: Social Intelligence with Contextual Ambidexterity for Long-Term Human-Robot Interaction and Intervention (LT-HRI2)

职业：具有情境二元性的社交智能，用于长期人机交互和干预（LT-HRI2）

• 批准号: 1846658
• 负责人: Chung Hyuk Park
• 金额: $ 54.89万
• 依托单位: George Washington University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-03-01 至 2025-02-28
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1846658&HistoricalAwards=false
• 关键词: CAREER; Social; Intelligence; Contextual; Ambidexterity

Autism Spectrum Disorder (ASD) is the most common neuro-developmental disorder among adolescents in the United States, with an estimated prevalence rate of one in 59. The major characteristics include deficits in social, sensory, and emotional processing. Boys have about four times higher rate of diagnosis than girls. General interventions such as behavior and speech/language therapies have been developed, and socially assistive robots have shown effective and positive outcomes in communicating with and delivering interactive interventions to individuals with ASD. However, a major obstacle to broadening the impact of robotic assistance and intervention in real-world settings is the lack of a robotic framework that can adaptively learn diverse interaction skills over time and associate with new social contexts as the human counterpart develops. Furthermore, recent studies are showing that girls are under-diagnosed and thus under-served due to their higher sensitivity in social cues resulting in social camouflage that often causes their symptoms to go undetected. Thus, there is a demonstrated need for intelligent socially-assistive robotic systems that can cope with the developmental processes and gender-specific characteristics of children with ASD. To address this need, the project aims to develop a novel socio-emotional human-robot interaction framework to provide interactive emotional regulation and guidance through robot-initiated conversation and gestures in response to the individual's unique socio-emotional cues. These cues can be recognized through analysis of voice signals, facial expressions, gestures and conversation. The successfully developed framework, which will be implemented on virtual and physical robotic platforms, will promote healthier and emotionally-balanced living for individuals with socio-emotional processing disorders or deficits. The knowledge gained through this project will be directly fused into educational activities for the future generation and students from underrepresented groups to develop creative mindsets and analytic skills in science, technology, engineering, art, and math (STEAM). Collaborative outreach activities include: applying the technology developed in the George Washington Autism & Neurodevelopmental Disorders Institute (ANDI) facility; working with the Kennedy Krieger Institute through Robotic Design workshops, Summer Robotics camps and providing Robotic interactions in school environments; and participating in the Take2 Summer Camp program that offers a 4-week therapeutic camp for children who have difficultly functioning in the social world.The PI's long-term career research goal is to understand the fundamental principles of human interactions and behaviors and translate these mechanisms into computational modeling and algorithms for a novel assistive robotic framework. Toward this goal, this project will develop a socially assistive robotic framework with contextual ambidexterity that is perceptive of personal socio-emotional states, capable of learning social skills, emotionally interactive, and gender-smart for long-term human-robot interaction and intervention (LT-HRI). Contextual ambidexterity, which investigates the key metrics for synchronizing two strategies--how to best exploit given functionality and resources in performing a task and how to efficiently explore new skills and knowledge to gain social intelligence over time--is very applicable to the robotic agent that needs to be able to determine best selections from programmed skills (exploitation) when faced with well-perceived situations but needs to also be able to learn new skills and contexts (exploration) when faced with unknown situations. To evaluate the efficacy of the framework in the real world, an active learning robotic agent will be developed to assist in socio-emotional LT-HRI for adolescents with autism spectrum disorder (ASD). The Research Plan is organized under four tasks. The FIRST TASK is to achieve advanced emotional perception tailored to an individual's unique socio-emotional cues. A voice-based-emotion-estimation module will be developed and combined with a widely used facial expression analysis module to control an interactive communication module that can interact with the user to acquire more accurate emotional characteristics if needed. The SECOND TASK is to learn social gestures and contexts from personal interactions and communications. Algorithms have been developed to capture the real-time sequence of a human gesture and to generate "gesture-features" that will be passed to a module that will check to see if the feature is already learned or, if not, will be registered as a new gesture. Once the gesture is learned, the robot will practice the learned behavior and observe the user's response to learn/update the social context of the gesture. The THIRD TASK is to develop an efficient framework for modeling emotional interaction and regulation between a human user and a robotic agent and to develop controllable algorithms for effective rapport formation. Emotional agents include human emotion, robotic emotion and a target emotional goal for emotional regulation and therapy. In the "rapport forming" phase, the robot's emotional goal is designed to approach the human's emotional goal and establish a common bond of empathy. In the "emotional guidance" phase, if the "rapport phase" is failing, the robot will take a proactive role in moving the human's emotional state toward the target emotion. The FOURTH TASK is to develop a gender-smart robotic intervention system, i.e., an optimal interaction policy for gender-specific social environments. Once the robot has perceived the user's emotional state, the robot can select subsequent actions to increase the user's engagement/participation level. Gender-smart behavior planning will be based on Partial Observable Markov Decision Process(POMDP) models of each user's interaction patterns/preferences, which can be used to maximize the total expected reward and determine the next action to be taken by the robot. POMDP models results can be incorporated into gender-specific representations (e.g. heatmaps) that can be used to anticipate gender differences. A set of experiments with human participants (neurotypical adolescents and adolescents with ASD, ages 10-19) will be designed to validate algorithms and assess the overall effectiveness of the robotic framework. The specifications, training data, and algorithm outcomes produced by the project will be openly disseminated for researchers as open-source Robot Operating System (ROS) packages and repositories.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

自闭症谱系障碍（ASD）是美国青少年中最常见的神经发育障碍，估计患病率为59分之一。主要特征包括社会、感觉和情绪处理方面的缺陷。男孩的诊断率是女孩的四倍。一般的干预措施，如行为和言语/语言治疗已经开发出来，社交辅助机器人在与ASD患者沟通和提供互动干预方面已经显示出有效和积极的结果。然而，在现实环境中扩大机器人辅助和干预影响的一个主要障碍是缺乏一个机器人框架，该框架可以随着时间的推移适应性地学习各种互动技能，并随着人类同伴的发展与新的社会环境相关联。此外，最近的研究表明，女孩的诊断不足，因此得不到充分的服务，因为她们对社会线索更敏感，导致社会伪装，往往导致她们的症状未被发现。因此，需要智能社交辅助机器人系统，以应对自闭症儿童的发育过程和性别特征。为了满足这一需求，该项目旨在开发一种新的社会情感人机交互框架，通过机器人发起的对话和手势来响应个人独特的社会情感线索，提供交互式情感调节和指导。这些线索可以通过分析语音信号、面部表情、手势和对话来识别。成功开发的框架将在虚拟和物理机器人平台上实施，将促进有社会情感处理障碍或缺陷的个人更健康和情感平衡的生活。通过该项目获得的知识将直接融入下一代和来自代表性不足群体的学生的教育活动中，以培养科学，技术，工程，艺术和数学（STEAM）的创造性思维和分析技能。合作推广活动包括：应用乔治华盛顿自闭症和神经发育障碍研究所（ANDI）设施开发的技术；与肯尼迪克里格研究所合作，通过机器人设计研讨会，夏季机器人营地，并在学校环境中提供机器人互动；并参加Take2夏令营项目，该项目为在社交世界中有困难的儿童提供为期4周的治疗营。PI的长期职业研究目标是了解人类互动和行为的基本原理，并将这些机制转化为新型辅助机器人框架的计算建模和算法。为了实现这一目标，该项目将开发一种具有情境双灵巧性的社交辅助机器人框架，该框架能够感知个人社会情绪状态，能够学习社交技能，情感互动，并具有性别智能，用于长期人机交互和干预（LT-HRI）。上下文怀二心,它调查了同步两种策略的关键指标——如何在执行任务时最好地利用给定的功能和资源，以及如何有效地探索新技能和知识以获得社会智能——这非常适用于机器人代理，当面对良好感知的情况时，它需要能够从编程技能（开发）中做出最佳选择，但同时也需要能够学习新技能和背景（探索）未知的情况。为了评估该框架在现实世界中的有效性，将开发一个主动学习机器人代理来协助自闭症谱系障碍（ASD）青少年的社会情感LT-HRI。研究计划分为四个任务。第一项任务是根据个人独特的社会情感线索实现高级情绪感知。将开发基于语音的情绪估计模块，并结合广泛使用的面部表情分析模块来控制交互通信模块，该模块可以与用户交互，根据需要获取更准确的情绪特征。第二项任务是从个人互动和沟通中学习社交手势和语境。已经开发了算法来捕捉人类手势的实时序列，并生成“手势特征”，这些特征将被传递给一个模块，该模块将检查该特征是否已经被学习，如果没有，将被注册为一个新的手势。一旦学会了手势，机器人就会练习学会的行为，并观察用户的反应，以学习/更新手势的社会背景。第三项任务是开发一个有效的框架，用于模拟人类用户和机器人代理之间的情感交互和调节，并开发有效建立融洽关系的可控算法。情感主体包括人类情感、机器人情感和用于情感调节和治疗的目标情感。在“融洽关系形成”阶段，机器人的情感目标被设计成接近人类的情感目标，并建立一种共情的共同纽带。在“情绪引导”阶段，如果“融洽阶段”失败，机器人将主动将人类的情绪状态推向目标情绪。第四个任务是开发一个性别智能机器人干预系统，即针对特定性别社会环境的最佳交互策略。一旦机器人感知到用户的情绪状态，机器人就可以选择后续的动作来提高用户的参与度。性别智能行为规划将基于每个用户交互模式/偏好的部分可观察马尔可夫决策过程（POMDP）模型，该模型可用于最大化总预期奖励并确定机器人要采取的下一步行动。POMDP模型的结果可以纳入特定性别的表示（如热图），可用于预测性别差异。将设计一组人类参与者（10-19岁的神经正常青少年和自闭症青少年）的实验，以验证算法并评估机器人框架的整体有效性。该项目产生的规范、训练数据和算法结果将作为开源机器人操作系统（ROS）软件包和存储库公开分发给研究人员。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

The Effects of Robot Voices and Appearances on Users’ Emotion Recognition and Subjective Perception

机器人声音和外观对用户情绪识别和主观感知的影响

• DOI: 10.1142/s0219843623500019
• 发表时间: 2023
• 期刊: International Journal of Humanoid Robotics
• 影响因子: 1.5
• 作者: Ko, Sangjin;Barnes, Jaclyn;Dong, Jiayuan;Park, Chung Hyuk;Howard, Ayanna;Jeon, Myounghoon
• 通讯作者: Jeon, Myounghoon

A MultiModal Social Robot Toward Personalized Emotion Interaction

• 发表时间: 2021-10
• 期刊: ArXiv
• 作者: Baijun Xie;C. Park

Dance with a Robot: Encoder-Decoder Neural Network for Music-Dance Learning

与机器人共舞：用于音乐舞蹈学习的编码器-解码器神经网络

• DOI: 10.1145/3371382.3378372
• 发表时间: 2020
• 期刊: IEEE/ACM International Conference on Human-Robot Interaction
• 作者: Baijun Xie;C. Park
• 通讯作者: C. Park

Musical Emotion Recognition with Spectral Feature Extraction Based on a Sinusoidal Model with Model-Based and Deep-Learning Approaches

• DOI: 10.3390/app10030902
• 发表时间: 2020-02-01
• 期刊: APPLIED SCIENCES-BASEL
• 影响因子: 2.7
• 作者: Xie, Baijun;Kim, Jonathan C.;Park, Chung Hyuk
• 通讯作者: Park, Chung Hyuk

"Can You Guess My Moves?: Playing Charades with a Humanoid Robot Employing Mutual Learning with Emotional Intelligence

• DOI: 10.1145/3568294.3580170
• 发表时间: 2023-03
• 期刊: Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction
• 作者: Baijun Xie;C. Park