SGER: Social-Emotional Intelligence Prosthetic

这项研究将开发第一个已知的可穿戴设备，能够在实时人类互动中感知和报告社交情感信息。使用一个小型可穿戴相机以及定制的视频和图案分析算法，该系统将自动读取和解释佩戴者与之互动的人的面部表情和头部运动，并通过声音或触觉反馈将其推论传达给佩戴者。该系统建立在Rana el Kaliouby开发的“读心术”计算模型的基础上，该模型形成了一个自动系统的基础，用于从视频流中的头部和面部显示实时推断精神状态。该项目汇集并扩展了三个关键科学和技术领域-情感计算、可穿戴计算和实时机器感知-的最新进展和方法，该项目将显著提高机器根据参与一系列自然互动的人的非语言表达实时推断认知情感状态的新生能力。拟议的工作是为预计将成为一系列日益智能化的假肢的第一个原型所做的工作。这一设计和策略将为理解社会情感线索在自然互动中的作用开辟新的研究路线。对非语言线索的自动感知和对心理状态的推断是机器视觉和机器学习领域中的研究人员继续面临的问题。如果系统被要求实时执行并概括到新的情况和人，这些问题就变得更加复杂，对于辅助技术来说，这是必要的要求。该项目解决了关于机器感知是否足够可靠和足够舒适，以改善人与人之间的交流来增强社会互动的公开研究问题。由于需要在照明条件多变、处理和功率有限的移动环境中运行，这一挑战变得更加复杂。作为第一项旨在增强人的社会情绪智力技能损害的假肢辅助技术，该项目有可能带来一系列新的研究。一个具有社会-情绪感知能力的系统可能会帮助越来越多的自闭症谱系障碍患者在自然而不是结构化的环境中感知交流，增强他们在社会环境中学习和发展的能力。该系统还将为协作系统的研究人员提供一个新的工具，以更仔细地检查自然日常交流中非语言线索的交流。这项跨学科工作的成果可以被用于人机交互、可穿戴计算、机器人以及未来具有社交情绪智能的技术。将通过同行评议的出版物和会议介绍以及通过吸引更多少数群体进入科学和技术领域的论坛，在所有这些社区分发成果。这项工作通过让学生深入参与拟议的研究活动来促进对他们的培训和教育，并致力于让代表人数不足的少数群体更多地参与尖端科学研究。

This research will develop the first known wearable device capable of perceiving and reporting on social-emotional information in real-time human interaction. Using a small wearable camera and customized video and pattern analysis algorithms, the system will automatically read and interpret the facial expressions and head movements of the person the wearer is interacting with and will communicate its inferences to the wearer via sound or tactile feedback. The system builds on and extends the computational model of "mind-reading" developed by Rana el Kaliouby, which forms the basis of an automated system for the inference of mental states from head and facial displays in a video stream in real time. Bringing together and expanding upon recent advances and methodologies in three key scientific and technical areas - affective computing, wearable computing, and real time machine perception - the project will significantly advance the nascent ability of machines to infer cognitive-affective states in real-time from nonverbal expressions of people engaged in a range of natural interactions. The proposed work is for the first prototype of what is expected to become a sequence of increasingly intelligent prosthetics. The design and strategy will open up new lines of inquiry in understanding the role of social-emotional cues in natural interaction. The automated perception of nonverbal cues and inference of mental states are problems that continue to challenge researchers in the domains of machine vision and machine learning. These problems become even more complex if the system is required to execute in real-time and to generalize to novel situations and people, a necessary requirement in the case of an assistive technology. The project addresses open research questions pertaining to whether machine perception can be reliable enough and comfortable enough to augment social interactions in a way that improves human to human communication. The challenge is further complicated by the requirement of operating in a mobile environment with varied lighting conditions and limited processing and power. Being the first prosthetic assistive technology that is designed to augment a person's impairment in socio-emotional intelligence skills, the project has the potential to lead to a line of new research. A system with social-emotional perceptual abilities can potentially assist the growing number of individuals diagnosed with Autism Spectrum Disorder, in perceiving communication in a natural rather than a structured environment, bootstrapping their ability to learn and develop in social settings. The system will also provide a new tool for researchers on collaborative systems to more carefully examine the communication of nonverbal cues in natural everyday communication. The results of this interdisciplinary work can be leveraged in human-computer interaction, wearable computing, robotics and in future technologies with social-emotional intelligence. Results will be distributed in all of these communities through peer-reviewed publications and conference presentations, as well as through forums for attracting more minorities to scientific and technical fields. This work promotes the training and education of students by deeply involving them in the proposed research activities, with a commitment to growing the participation of underrepresented minorities in cutting-edge scientific research