EAGER: Top-down processes to extract meaning from images

人类快速处理图像和场景，以便了解周围世界正在发生的事情。人类在这方面做得非常好，以至于他们比现有的计算模型更能理解场景中存在什么元素、它们的位置或它们所参与的动作。计算模型的一个局限性是，它们不像人类那样提供对场景各个组成部分的详细解释。例如，计算模型可以成功地将图像标记为包含马，但是人类也会自然地识别马的较小组件，例如眼睛、耳朵、嘴、马内、腿、尾巴等。识别这些单独的组件及其关系是人类视觉处理的重要部分。视觉理解的这种差异为构建与人类相似地看待和解释世界的人工计算系统带来了挑战。这些基本的局限性与以下事实有关：现有的计算系统主要依赖于所谓的“自下而上的处理”，即从简单到复杂的视觉特征的顺序处理，这并没有考虑人类认知如何影响对图像的有意义的识别。我们的主要目标是调查的计算原理和神经生物学系统，允许整合的认知经验的视觉处理。我们结合联合收割机对人类的心理学研究，脑组织的神经生理学记录和计算工作，以建立一个能够从图像中提取复杂含义的综合模型，其方式更接近人类的能力。这项研究将对理解大脑如何处理图像以及相关的神经回路产生广泛的影响。此外，从该项目中获得的见解可以在广泛的领域中应用，包括机器人视觉，自动导航，监控和自动临床图像理解。作为该项目的一部分，我们将建立一个基于研究产品的夏季课程，我们将在大脑，思想和机器的界面上培养下一代学者。在人类大脑中，信息既从低级视觉区域流向高级视觉区域，也在整个腹侧视觉皮层中以相反的方向流动。这种双向处理在皮层计算中起着重要作用。然而，由自上而下的组件实现的功能在视觉认知中形成了一个开放的问题。了解前馈处理的局限性（从较低到较高的视觉区域）将揭示先验知识与自下而上输入相结合的机制，并指导开发新算法，从感官输入中提取有用的意义。深度网络模型的应用现在在多个领域的机器学习中发挥着重要作用，但这些应用无法捕捉视觉处理的基本方面。我们的研究项目探讨了现有的前馈识别模型构成第一阶段的可能性，导致类别候选人的初始激活，这是不完整的，往往是不准确的。然后，第一阶段触发特定于类的过程的应用，该过程恢复对可见场景的更丰富和更准确的解释，并拒绝初始的错误候选。该建议包括三个主要部分：㈠心理物理学实验，以评估人类如何从图像中提取意义的准确性和速度; ㈡沿着人类腹侧视觉皮层进行侵入性神经生理学记录，以了解参与从一组最小图像的新数据中提取意义的神经回路;（iii）将自下而上的计算与自上而下的信号相结合的计算模型，以从图像中提取含义。研究工作将与教育和外联活动相结合，旨在传播科学见解，并将尖端研究纳入本科生和研究生的培训机会。

Humans rapidly process images and scenes so they can understand what is occurring in the world around them. Humans do this so well that they outperform existing computational models' ability to understand what elements exist in the scene, their location, or the actions they are involved in. One limitation of computational models is that they do not provide a detailed interpretation of a scene's individual components the way that humans do. For example, the computational model may successfully label an image as containing a horse, but humans will also naturally identify smaller components of the horse, such as the eyes, ears, mouth, mane, legs, tail, and so on. Identifying these individual components and their relationships is an essential part of human visual processing. Such differences in visual understanding create a challenge for constructing artificial computational systems that see and interpret the world similarly to humans. These fundamental limitations are related to the fact that existing computational systems rely primarily on what is called 'bottom-up processing', the sequential processing of visual features from simple to complex ones, which does not account for how human cognition influences meaningful recognition of the image. Our main goal is to investigate the computational principles and neurobiological systems that allow for integration of cognitive experience within visual processing. We combine psychological studies in humans, neurophysiological recordings of brain tissue, and computational work to build an integrative model capable of extracting complex meaning from images, in a way that more closely resembles human capabilities. The research will have broad implications in understanding how the brain processes images and the neural circuits that are involved. Additionally, the insights obtained from this project could have applications in a broad range of domains including robot vision, automatic navigation, surveillance, and automatic clinical image understanding. As part of the project we will establish a summer course based on the research products in which we will train the next generation of scholars at the interface of brains, minds, and machines. In the human brain, information flows both from low to higher visual areas, as well as in the opposite direction throughout ventral visual cortex. This bi-directional processing has a fundamental role in cortical computations. Yet, the functions implemented by the top-down components form an open problem in visual cognition. Understanding the limitations of feed-forward processing (from lower to higher visual regions) will shed light on the mechanisms by which prior knowledge is integrated with bottom-up inputs, and guide development of new algorithms for extracting useful meaning from sensory input. Applications of deep network models now play a significant role in machine learning across multiple domains, but these fail to capture fundamental aspects of visual processing. Our research program examines the possibility that existing feed-forward recognition models constitute a first stage leading to the initial activation of category candidates, which is incomplete and often inaccurate. The first stage then triggers the application of class-specific processes, which recover a richer and more accurate interpretation of the visible scene, and reject initial false candidates. The proposal involves three main components: (i) Psychophysics experiments to evaluate the accuracy and speed of how humans extract meaning from images; (ii) Invasive neurophysiological recordings along the human ventral visual cortex to understand the neural circuits involved in extracting meaning from a novel data set of minimal images; (iii) A computational model that integrates bottom-up computations with top-down signals to extract meaning from images. The research efforts will be combined with educational and outreach activities aimed at disseminating the scientific insights and incorporating cutting-edge research into training opportunities for undergraduate and graduate students.