SGER: Social-Emotional Intelligence Prosthetic

这项研究将开发出第一个已知的可穿戴设备，能够在实时人类互动中感知和报告社会情感信息。通过一个小型可穿戴摄像头和定制的视频和模式分析算法，该系统将自动读取和解释与佩戴者互动的人的面部表情和头部动作，并通过声音或触觉反馈将其推断传达给佩戴者。该系统建立并扩展了Rana el Kaliouby开发的“读心术”计算模型，该模型构成了一个自动系统的基础，用于实时从视频流中的头部和面部显示推断心理状态。该项目汇集并扩展了三个关键科学和技术领域的最新进展和方法——情感计算、可穿戴计算和实时机器感知——该项目将显著推进机器的新生能力，即从参与一系列自然互动的人们的非语言表达中实时推断认知情感状态。这项提议的工作是第一个原型，预计将成为一系列越来越智能的假肢。该设计和策略将为理解社会情感线索在自然互动中的作用开辟新的探究路线。非语言线索的自动感知和心理状态的推断一直是机器视觉和机器学习领域研究人员面临的挑战。如果系统需要实时执行并推广到新的情况和人员，这些问题就会变得更加复杂，这是辅助技术的必要要求。该项目解决了有关机器感知是否足够可靠和舒适的开放性研究问题，以改善人与人之间的沟通方式来增强社会互动。由于需要在不同的照明条件和有限的处理和功率的移动环境中操作，这一挑战进一步复杂化。作为第一个旨在增强人类社会情商技能缺陷的假肢辅助技术，该项目有可能引发一系列新的研究。一个具有社会情感感知能力的系统可以潜在地帮助越来越多的被诊断为自闭症谱系障碍的个体，在自然而不是结构化的环境中感知交流，引导他们在社会环境中学习和发展的能力。该系统还将为协作系统的研究人员提供一个新的工具，以更仔细地检查自然日常交流中的非语言线索的交流。这项跨学科工作的结果可以在人机交互、可穿戴计算、机器人和未来的社交情商技术中得到利用。研究结果将通过同行评议的出版物和会议报告，以及通过吸引更多少数民族进入科学和技术领域的论坛，在所有这些社区中分发。这项工作通过使学生深入参与拟议的研究活动来促进对学生的培训和教育，并致力于增加代表性不足的少数群体参与尖端科学研究

This research will develop the first known wearable device capable of perceiving and reporting on social-emotional information in real-time human interaction. Using a small wearable camera and customized video and pattern analysis algorithms, the system will automatically read and interpret the facial expressions and head movements of the person the wearer is interacting with and will communicate its inferences to the wearer via sound or tactile feedback. The system builds on and extends the computational model of "mind-reading" developed by Rana el Kaliouby, which forms the basis of an automated system for the inference of mental states from head and facial displays in a video stream in real time. Bringing together and expanding upon recent advances and methodologies in three key scientific and technical areas - affective computing, wearable computing, and real time machine perception - the project will significantly advance the nascent ability of machines to infer cognitive-affective states in real-time from nonverbal expressions of people engaged in a range of natural interactions. The proposed work is for the first prototype of what is expected to become a sequence of increasingly intelligent prosthetics. The design and strategy will open up new lines of inquiry in understanding the role of social-emotional cues in natural interaction. The automated perception of nonverbal cues and inference of mental states are problems that continue to challenge researchers in the domains of machine vision and machine learning. These problems become even more complex if the system is required to execute in real-time and to generalize to novel situations and people, a necessary requirement in the case of an assistive technology. The project addresses open research questions pertaining to whether machine perception can be reliable enough and comfortable enough to augment social interactions in a way that improves human to human communication. The challenge is further complicated by the requirement of operating in a mobile environment with varied lighting conditions and limited processing and power. Being the first prosthetic assistive technology that is designed to augment a person's impairment in socio-emotional intelligence skills, the project has the potential to lead to a line of new research. A system with social-emotional perceptual abilities can potentially assist the growing number of individuals diagnosed with Autism Spectrum Disorder, in perceiving communication in a natural rather than a structured environment, bootstrapping their ability to learn and develop in social settings. The system will also provide a new tool for researchers on collaborative systems to more carefully examine the communication of nonverbal cues in natural everyday communication. The results of this interdisciplinary work can be leveraged in human-computer interaction, wearable computing, robotics and in future technologies with social-emotional intelligence. Results will be distributed in all of these communities through peer-reviewed publications and conference presentations, as well as through forums for attracting more minorities to scientific and technical fields. This work promotes the training and education of students by deeply involving them in the proposed research activities, with a commitment to growing the participation of underrepresented minorities in cutting-edge scientific research