CRCNS US-German Research Proposal: Inception loops for interpretable tuning in macaque area V4

一个长期存在的假说认为，大脑的视觉区域沿着一个层次结构，呈现出越来越复杂的人类可解释的特征。此外，灵长类动物的视觉是一个主动的过程，通过眼睛的一系列短暂注视来获取有关场景的信息。尽管几十年的研究，表征神经元的响应特性沿着视觉皮层层次和调谐动态与自由观看仍然是远远没有完成。两个主要原因是大脑中信息处理的非线性性质，以及视觉输入本身的高维性。为了解决这些固有的挑战，研究人员将应用一种名为Inception Loops的创新方法，该方法将大数据和人工智能（AI）相结合，以研究信息在视觉感知过程中如何沿着皮层呈现的动态。这项工作将揭示生命最大奥秘之一的机制-感知和认知的生物学基础。对视觉感知的算法理解将超越神经科学的影响，开发具有更人性化能力的智能AI。该团队致力于通过专注于神经科学和人工智能的教育推广来扩大科学参与。外联活动包括公共活动、学徒、研究实习和课程。该团队还将参与关于大脑研究、人工智能、社会和伦理的公开讨论。Inception Loops是研究人员开发的一种创新方法，联合收割机了多神经元记录和深度学习（DL）预测模型。这使得可以在体内验证的神经调谐的系统的计算机表征成为可能。在目标1中，研究人员将记录猕猴V4在注视期间对呈现的自然刺激的神经反应。神经元的响应将使用DL模型建模，使用从渲染过程中提取的视觉场景的图像和几何特征。这将使他们能够系统地表征神经元的非线性调谐功能，特别是单细胞不变性，在可解释的几何场景特征，如倾斜，表面曲率和对象身份。在目标2中，研究人员将使用DL模型以及交错固定和自由观看试验的实验范式来研究V4神经元的空间感受野和调谐功能如何在自然观看条件下被自然场景中的扫视和显著特征调制。显着性将确定从眼睛运动在自由观看实验相同的图像和建模使用一个国家的最先进的DL显着性模型。他们假设，即使没有执行扫视，感受野也会被图像中的显著特征所吸引。总之，这两个目标将产生一个更全面和可解释的理解V4中潜在的几何场景特征的表示，以及它们如何在自由观看期间被显著特征和扫视调制。德国（BMBF）该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响进行评估，被认为值得支持审查标准。

A long-standing hypothesis posits that along a hierarchy of visual areas of the brain, increasingly complex humanly interpretable features are represented. Moreover, vision in primates is an active process, where information about a scene is acquired through sequences of short fixations of the eyes. Despite decades of study, the characterization of response properties of neurons along the visual cortical hierarchy and the tuning dynamics associated with free viewing is still far from complete. Two major reasons are the non-linear nature of information processing in the brain, and the high dimensionality of the visual input itself. To address these inherent challenges, the investigators will apply an innovative method called Inception Loops, that combines big data and artificial intelligence (AI) to study the dynamics of how information is represented along the cortex during visual perception. This work will shed light on the mechanisms of one of the greatest mysteries of life -- the biological basis of perception and cognition. An algorithmic understanding of visual perception will have impact beyond neuroscience in developing smarter AI with more humanlike capabilities. The team is committed to broadening participation in science through educational outreach focused on neuroscience and AI. Outreach activities include public events, apprenticeships, research internships, and courses. The team will also engage in public discussions about brain research, AI, society and ethics.Inception Loops, an innovative method developed by the investigators, combine multi-neuronal recordings and deep learning (DL) predictive models. This enables a systematic in silico characterization of neural tuning which can be verified in vivo. In aim 1, the investigators will record neural responses from macaque V4 to rendered naturalistic stimuli during fixation. The responses of neurons will be modeled with DL models, using the images and geometric features of the visual scene extracted from the rendering process. This will enable them to systematically characterize the nonlinear tuning functions of the neurons, in particular single cell invariances, in terms of interpretable geometric scene features such as slant, surface curvature and object identities. In aim 2, the investigators will use DL models together with an experimental paradigm of interleaved fixating and free viewing trials to study how the spatial receptive fields and tuning functions of V4 neurons are modulated by saccades and salient features in natural scenes under natural viewing conditions. Saliency will be determined from eye movements in free viewing experiments on the same images and modeled using a state-of-the-art DL saliency model. They hypothesize that receptive fields are attracted towards salient features in an image even when saccades are not executed. Together, these two aims will yield a more comprehensive and interpretable understanding of the representation of latent geometric scene features in V4 and how they are modulated by salient features and saccades during free viewing.A companion project is being funded by the Federal Ministry of Education and Research, Germany (BMBF).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.