NCS-FO:Collaborative Research:Decoding and Reconstructing the Neural Basis of Real World Social Perception

NCS-FO：合作研究：解码和重建现实世界社会感知的神经基础

• 批准号: 1734868
• 负责人: Max G'Sell
• 金额: $ 49.01万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-08-01 至 2021-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1734868&HistoricalAwards=false
• 关键词: NCS; FO; Collaborative; Research; Decoding

Social and affective perception is the critical input that governs how we interact with others during everyday life. Consequently, having a model of the neurobiological basis of social and affective perception is critical for understanding the neural basis of human behavior. The overwhelming majority of our understanding of the neural basis of social and affective perception comes from studies done in artificial lab settings, which cannot capture the richness, complexity, and salience of real-world social interactions. This project aims to fill this gap in knowledge. To accomplish this goal, the researchers will record electrical brain activity from patients undergoing neurosurgical treatment for epilepsy. To determine the region of the brain responsible for their seizures, these patients are implanted with electrodes in various parts of their brain and then they spend 1-2 weeks in the hospital during which they interact with doctors, nurses, friend and family visitors, etc. This award will support research into using the recordings from their brains to understand how these patients perceive and understand the actions, emotions, and communication during these interactions on a moment-to-moment basis. The results of these studies have the potential to transform our understanding of social and affective perception by illuminating the neural basis of these processes during real life, meaningful interactions. The lack of models of the neural basis of natural, real world social and affective perception is a critical impediment to understanding these processes and ultimately a developing treatments for debilitating neurological and psychiatric disorders of social and affective perception, such as autism, post traumatic stress disorder, etc. In addition, through education, mentoring, and teaching, this award will provide an avenue for new researchers to take advantage of the rare and valuable opportunity for basic neuroscientific research provided by direct recordings from the human brain. This research is supported by the EHR Core Research Program, providing funding for fundamental research in STEM learning and learning environments, broadening participation in STEM, and STEM workforce development. Models of social visual perception developed using unnatural stimuli often assume that neurons have unchanging response sensitivity and are organized into bottom-up hierarchies. While some recent models acknowledge the role of feedback, they remain simplistic with a relatively limited number of core systems and often neglect of the role of social context and dynamic prior knowledge. These models are unlikely to fully generalize to natural social vision where the system can rapidly and actively adapt its response to optimize processing of rich and complex natural visual input. The PI and colleagues will combine intracranial EEG (iEEG) recordings captured during long stretches of natural visual behavior with cutting-edge computer vision, machine learning, and statistical analyses to understand the neural basis of natural, real-world visual perception. The goal of their program of research is to develop the first fully ecologically validated models of social perception. The researchers will use recent advances in iEEG in combination with cutting-edge gaze tracking technology, video analysis tools, and big data statistical and machine learning tools to understand the rapid, complex neural information processing that occurs during real-world social vision. The project will involve decoding the spatiotemporal patterns of neural activity and reconstruct the expressive features of people they see at these different levels on a moment-to-moment basis. The multidisciplinary nature of this project provides an excellent environment for students and postdocs to be trained in computational methods, statistics, and neuroscience. Given the rapid advance of high-level computational and statistical methods in neuroscience, this multidisciplinary training is critical for modern neuroscientists. Enhanced understanding of the mechanisms involved in social cognition has implications for teaching and learning. For example, knowing more about how people form impressions of one another can inform teachers' abilities to recognize and respond to students and other stakeholders in educational settings.This project is funded by Integrative Strategies for Understanding Neural and Cognitive Systems (NSF-NCS), a multidisciplinary program jointly supported by the Directorates for Computer and Information Science and Engineering (CISE), Education and Human Resources (EHR), Engineering (ENG), and Social, Behavioral, and Economic Sciences (SBE).

社交和情感感知是决定我们在日常生活中如何与他人互动的关键输入。因此，拥有社会和情感感知的神经生物学基础的模型对于理解人类行为的神经基础至关重要。我们对社交和情感感知的神经基础的绝大多数理解来自于在人工实验室环境中进行的研究，这些研究无法捕捉真实世界社交互动的丰富性、复杂性和显着性。该项目旨在填补这一知识空白。为了实现这一目标，研究人员将记录接受癫痫神经外科治疗的患者的脑电活动。为了确定导致癫痫发作的大脑区域，这些患者在大脑的不同部分植入电极，然后在医院度过1-2周，在此期间他们与医生、护士、朋友和家人访客等进行互动。该奖项将支持研究，利用他们大脑的录音来了解这些患者如何感知和理解这些互动过程中的行为、情绪和交流。这些研究的结果有可能改变我们对社会和情感感知的理解，通过阐明现实生活中这些过程的神经基础，有意义的互动。缺乏自然、现实世界社会和情感知觉的神经基础模型是理解这些过程的关键障碍，并最终成为开发治疗社交和情感知觉的衰弱神经和精神障碍的方法，如自闭症、创伤后应激障碍等。此外，通过教育、指导和教学，该奖项将为新的研究人员提供一条途径，利用来自人脑的直接记录为基础神经科学研究提供的难得和宝贵的机会。这项研究得到了EHR核心研究计划的支持，为STEM学习和学习环境中的基础研究提供资金，扩大对STEM的参与，以及STEM劳动力发展。使用非自然刺激开发的社会视觉感知模型通常假设神经元具有不变的反应敏感性，并被组织成自下而上的层次结构。虽然最近的一些模型承认反馈的作用，但它们仍然过于简单化，核心系统的数量相对有限，而且往往忽视了社会背景和动态先验知识的作用。这些模型不太可能完全推广到自然社会视觉，在自然社会视觉中，系统可以快速和主动地调整其响应，以优化对丰富和复杂的自然视觉输入的处理。PI和他的同事们将把在自然视觉行为的长时间段中捕捉到的颅内EEG(IEEG)记录与尖端计算机视觉、机器学习和统计分析相结合，以了解自然、真实世界视觉感知的神经基础。他们研究计划的目标是开发第一个完全经过生态验证的社会感知模型。研究人员将利用iEEG的最新进展，结合尖端的凝视跟踪技术、视频分析工具、大数据统计和机器学习工具，了解现实世界社会视觉过程中发生的快速、复杂的神经信息处理。该项目将涉及解码神经活动的时空模式，并重建他们在不同水平上看到的人的表达特征。这个项目的多学科性质为学生和博士后提供了一个很好的环境，让他们接受计算方法、统计学和神经科学方面的培训。鉴于神经科学中高级计算和统计方法的快速发展，这种多学科培训对现代神经科学家来说至关重要。加强对社会认知机制的理解对教与学具有重要意义。例如，更多地了解人们如何形成彼此的印象可以提高教师识别和回应学生和教育环境中其他利益相关者的能力。该项目由理解神经和认知系统的综合策略(NSF-NCS)资助，NSF-NCS是由计算机和信息科学与工程(CEISE)、教育和人力资源(EHR)、工程(ENG)以及社会、行为和经济科学(SBE)等部门联合支持的多学科项目。

项目成果

Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data

• DOI: 10.18653/v1/2021.acl-long.322
• 发表时间: 2021-06
• 作者: P. Liang;Terrance Liu;Anna Cai;Michal Muszynski;Ryo Ishii;Nicholas Allen;R. Auerbach;D. Brent;R. Salakhutdinov;Louis-Philippe Morency

Deep Gamblers: Learning to Abstain with Portfolio Theory

深度赌徒：通过投资组合理论学习戒赌

• 发表时间: 2019
• 期刊: NeurIPS
• 作者: Ziyin Liu;Zhikang Wang;Paul Pu Liang;Russ R. Salakhutdinov;Louis-Philippe Morency and Masahito Ueda
• 通讯作者: Louis-Philippe Morency and Masahito Ueda

Multimodal Behavioral Markers Exploring Suicidal Intent in Social Media Videos

• DOI: 10.1145/3340555.3353718
• 发表时间: 2019-10
• 期刊: 2019 International Conference on Multimodal Interaction
• 作者: Ankit Shah;Vasu Sharma;Vaibhav Vaibhav-Vaibhav;Mahmoud Alismail;Louis-Philippe Morency

MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French.

MOSEAS：西班牙语、葡萄牙语、德语和法语的多模态语言数据集。

• 发表时间: 2020
• 作者: Zadeh, A.;Cao, Y.;Hessner, S.;Liang, P.;Poria, S.;Morency, L.-P.
• 通讯作者: Morency, L.-P.

Edge Convolutional Network for Facial Action Intensity Estimation

• DOI: 10.1109/fg.2018.00034
• 发表时间: 2018-05
• 期刊: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)
• 作者: Liandong Li;T. Baltrušaitis;Bo Sun;Louis-Philippe Morency