CRCNS Research Proposal: Predictive Coding Network for Human Vision

该项目旨在推进有关人类视觉的科学知识，并利用神经科学来增强计算机视觉的人工智能。视觉是人类如何看待和探索世界的核心。大约有十几个大脑区域协同工作，在几分之一秒内处理视觉信息。据推测，这些大脑区域积极预测一个人的视觉环境，并使用预测错误来更新其内部表示并指导行动。然而，目前尚不清楚大脑如何执行识别和预测的计算，以及机器是否有可能模仿大脑并在复杂，嘈杂和不确定的情况下识别和预测视觉输入。该项目将从计算、心理学和神经科学的角度解决这些问题，并提供新的模型、数据和工具，促进人工智能和神经科学之间的协同作用。研究人员将设计一个基于大脑预测编码的模型，并测试其执行计算机视觉任务的能力，以及解释人类行为和大脑对自然视觉刺激的反应。研究人员将首先开发一个被称为预测编码网络的深度神经网络。与现有的前馈神经网络（目前占主导地位的视觉模型）不同，预测编码网络具有与大脑中的神经处理相关的几个定义特征。它是双向的，自下而上和自上而下处理信息。它是循环的，利用相同的架构进行动态计算。它是并行的，允许信息处理在不同层内和不同层之间并行发生。它既是判别式的，又是生成式的，在一个框架中调和了图像识别和合成。预测编码网络将根据基准数据集进行评估。它被假设为达到具有竞争力的性能与许多更少的参数比最先进的状态。然后，研究人员将测试模型的行为给定的自然图像以各种方式退化和/或呈现不同的持续时间。假设该模型在运行越来越长的时间后将更加稳健和准确，并在类似条件下达到与人类感知相似的时间-准确性权衡。为了验证这一假设，研究人员将进行人类行为实验，并将模型的行为与人类行为进行比较。此外，研究人员还将测试该模型解释大脑对自然图像和视频的反应的能力，这些反应是通过功能性磁共振成像和颅内脑电图来测量的。该模型被假设为能够预测大脑的动态活动和表示给定的自然刺激。该项目的成功完成预计将提供一个可学习和可计算的端到端的大脑启发视觉模型。该模型将赋予机器自适应和强大的视觉，并为理解生物视觉的计算基础提供工具。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to advance scientific knowledge about human vision and use neuroscience to enhance artificial intelligence for computer vision. Vision is central to how humans see and explore the world. About a dozen brain regions work together to process visual information within a fraction of a second. It is hypothesized that these brain regions actively predict one's visual surroundings and use errors of prediction to update their internal representations and guide actions. However, it is not clear how the brain performs computations for recognition and prediction, and whether it is possible for a machine to mimic the brain and recognize and predict visual input in complex, noisy, and uncertain circumstances. This project will address these questions from computational, psychological, and neuroscientific perspectives and deliver new models, data, and tools that promote the synergy between artificial intelligence and neuroscience.Investigators will design a model based on predictive coding in the brain, and test its ability to perform computer vision tasks and explain human behaviors and brain responses to naturalistic visual stimuli. The investigators will first develop a deep neural network referred to as the predictive coding network. Unlike existing feedforward neural networks, the currently predominant vision models, the predictive coding network has several defining features relevant to neural processing in the brain. It is bi-directional, processing information both bottom-up and top-down. It is recurrent, utilizing the same architecture for dynamic computation. It is parallel, allowing information processing to occur in parallel both within and across different layers. It is both discriminative and generative, reconciling image recognition and synthesis in a single framework. The predictive coding network will be evaluated against benchmark data sets. It is hypothesized to reach competitive performance with many fewer parameters than the state of the art. Then, the investigators will test the model's behaviors given naturalistic images degraded in various ways and/or presented for various durations. It is hypothesized that the model will be more robust and accurate after running for increasingly longer times and reach a time-accuracy tradeoff like human perception under similar conditions. To test this hypothesis, the investigators will perform human behavioral experiments and compare the model's behaviors against human behaviors. Further, the investigators will test the model's ability to explain brain responses to naturalistic images and videos, measured with functional magnetic resonance imaging and intracranial electroencephalography. The model is hypothesized to be able to predict the brain's dynamic activity and representation given naturalistic stimuli. The successful completion of this project is expected to deliver a brain-inspired vision model learnable and computable end-to-end. This model will empower machines with adaptive and robust vision and provide a tool for understanding the computational basis of biological vision.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.