Collaborative Research: RUI: Uncovering the Neural Dynamics of Scene Categorization through Electroencephalography, Machine Learning, and Neuromodulation

合作研究：RUI：通过脑电图、机器学习和神经调节揭示场景分类的神经动力学

• 批准号: 1736394
• 负责人: Bruce Hansen
• 金额: $ 18.67万
• 依托单位: Colgate University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-08-01 至 2023-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1736394&HistoricalAwards=false
• 关键词: Collaborative; Research; RUI; Uncovering; Neural

A long-standing problem in cognitive neuroscience is understanding how we can categorize a novel scene in about the same amount of time that it takes to blink one's eyes. Categorization aids both identifying objects and locating them in cluttered scenes, and thus allows for intelligent action in the world. How do we derive semantically meaningful categories from the raw image pixels? Currently, there is experimental support for multiple mechanisms supporting scene categorization, such as through recognizing the scene's objects or other visual features such as spatial layout, color, or texture. Crucially, substantial correlations exist between all of these proposed features. This make it difficult to disentangle their relative contributions to categorization. For example, if two scenes share an object, they will often also share the texture features associated with that object. In this work, the PI (Dr. Bruce C Hansen, Colgate University) and co-PI (Dr. Michelle R Greene, Bates College) seek to disentangle the contribution of such features, and also to determine when these features become available for use, and how they combine to support scene categorization. By understanding the temporal dynamics of the brain activity related to scene categorization, it will be possible to obtain critical insights into how people rapidly but flexibly extract information from the environment. This work forms a bridge across several disciplines including psychology, cognitive neuroscience, computer vision, and machine learning. As such, the project will engage undergraduate students in truly interdisciplinary training that is at the cutting edge of multiple fields.This project will make use of high-density EEG combined with machine learning, computational modeling behavioral measures, and advanced neuromodulation to determine how and when the behaviorally relevant features support scene categorization. First, the work will link the encoding of these features to visual event related potentials (vERPs) and also to category information using multivariate classification techniques from machine learning. Taken together, these techniques will allow the PIs to determine the unique contributions of each feature to category-related brain activity over time. A hallmark of intelligent action is flexibility. Therefore, the project will also investigate the flexibility of feature use by manipulating the diagnosticity of information available to observers. These studies will provide insights regarding feature space usage as a function of task demands, as well as the impact of such demands on the time course of feature space availability as indexed by vERPs. Lastly, the project will test for a potential causal role of vERPs to categorization through the use of advanced neuromodulation techniques.

认知神经科学中一个长期存在的问题是，我们如何在眨眼的时间内对一个新场景进行分类。分类既有助于识别对象，也有助于在混乱的场景中定位它们，从而允许在世界中采取智能行动。我们如何从原始图像像素中获得语义上有意义的类别？目前，有实验支持多种机制支持场景分类，例如通过识别场景的对象或其他视觉特征，如空间布局，颜色或纹理。至关重要的是，所有这些被提出的特征之间存在着实质性的相关性。这使得很难理清它们对分类的相对贡献。例如，如果两个场景共享一个对象，它们通常也会共享与该对象相关的纹理特征。在这项工作中，PI （Bruce C Hansen博士，科尔盖特大学）和联合PI （Michelle R Greene博士，贝茨学院）试图理清这些特征的贡献，并确定这些特征何时可用，以及它们如何结合起来支持场景分类。通过了解与场景分类相关的大脑活动的时间动态，将有可能获得人们如何快速而灵活地从环境中提取信息的关键见解。这项工作在心理学、认知神经科学、计算机视觉和机器学习等多个学科之间架起了一座桥梁。因此，该项目将使本科生参与真正跨学科的培训，在多个领域的前沿。本项目将利用高密度脑电图结合机器学习、计算建模行为测量和高级神经调节来确定行为相关特征如何以及何时支持场景分类。首先，这项工作将把这些特征的编码与视觉事件相关电位（verp）联系起来，并使用机器学习的多元分类技术将这些特征与类别信息联系起来。综合起来，这些技术将允许pi确定每个特征随时间对与类别相关的大脑活动的独特贡献。聪明行动的一个特点是灵活。因此，该项目还将通过操纵观察者可用信息的诊断性来研究特征使用的灵活性。这些研究将提供关于特征空间使用作为任务需求的函数的见解，以及这些需求对由verp索引的特征空间可用性的时间过程的影响。最后，该项目将通过使用先进的神经调节技术来测试verp对分类的潜在因果作用。

项目成果

Uncovering the Spatiotemporal Dynamics of Goal-driven Efficient Coding with a Brain-supervised Sparse coding Network

利用脑监督稀疏编码网络揭示目标驱动高效编码的时空动态

• DOI: 10.32470/ccn.2022.1127-0
• 发表时间: 2022
• 期刊: Conference on Cognitive Computational Neuroscience
• 作者: Hansen, B.C.

Dynamic Electrode-to-Image (DETI) mapping reveals the human brain's spatiotemporal code of visual information.

• DOI: 10.1371/journal.pcbi.1009456
• 发表时间: 2021-09
• 期刊: PLoS computational biology
• 影响因子: 4.3
• 作者: Hansen BC;Greene MR;Field DJ
• 通讯作者: Field DJ

Disentangling the Independent Contributions of Visual and Conceptual Features to the Spatiotemporal Dynamics of Scene Categorization

• DOI: 10.1523/jneurosci.2088-19.2020
• 发表时间: 2020-07-01
• 期刊: JOURNAL OF NEUROSCIENCE
• 影响因子: 5.3
• 作者: Greene, Michelle R.;Hansen, Bruce C.
• 通讯作者: Hansen, Bruce C.

Towards a state-space geometry of neural responses to natural scenes: A steady-state approach

• DOI: 10.1016/j.neuroimage.2019.116027
• 发表时间: 2019-11-01
• 期刊: NEUROIMAGE
• 影响因子: 5.7
• 作者: Hansen, Bruce C.;Field, David J.;Miskovic, Vladimir
• 通讯作者: Miskovic, Vladimir