CRCNS Research Proposal: Predictive Coding Network for Human Vision

该项目旨在推进人类视觉的科学知识，并利用神经科学来增强计算机视觉的人工智能。视觉是人类如何看待和探索世界的核心。大约十几个大脑区域协同工作，在几分之一秒内处理视觉信息。据推测，这些大脑区域积极地预测一个人的视觉环境，并利用预测的错误来更新他们的内部表征和指导行动。然而，目前尚不清楚大脑是如何进行识别和预测的计算的，以及机器是否有可能模仿大脑，并在复杂、嘈杂和不确定的环境中识别和预测视觉输入。该项目将从计算、心理学和神经科学的角度解决这些问题，并提供新的模型、数据和工具，促进人工智能和神经科学之间的协同作用。研究人员将设计一个基于大脑预测编码的模型，并测试其执行计算机视觉任务的能力，并解释人类行为和大脑对自然视觉刺激的反应。研究人员将首先开发一个称为预测编码网络的深度神经网络。与现有的前馈神经网络（目前主要的视觉模型）不同，预测编码网络具有几个与大脑神经处理相关的定义特征。它是双向的，自下而上和自上而下都处理信息。它是循环的，利用相同的体系结构进行动态计算。它是并行的，允许信息处理在不同层内和层间并行进行。它是判别和生成，调和图像识别和合成在一个单一的框架。预测编码网络将根据基准数据集进行评估。它的假设是达到具有竞争力的性能与更少的参数比艺术的状态。然后，研究人员将测试模型的行为，给出以各种方式退化和/或呈现不同持续时间的自然图像。假设模型运行时间越长，鲁棒性和准确性越高，在相似条件下达到与人类感知相似的时间-精度权衡。为了验证这一假设，研究人员将进行人类行为实验，并将模型的行为与人类行为进行比较。此外，研究人员将测试该模型解释大脑对自然图像和视频反应的能力，用功能性磁共振成像和颅内脑电图来测量。该模型被假设能够预测大脑在自然刺激下的动态活动和表征。该项目的成功完成有望提供一个端到端可学习和可计算的大脑启发视觉模型。该模型将使机器具有自适应和鲁棒性视觉，并为理解生物视觉的计算基础提供工具。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to advance scientific knowledge about human vision and use neuroscience to enhance artificial intelligence for computer vision. Vision is central to how humans see and explore the world. About a dozen brain regions work together to process visual information within a fraction of a second. It is hypothesized that these brain regions actively predict one's visual surroundings and use errors of prediction to update their internal representations and guide actions. However, it is not clear how the brain performs computations for recognition and prediction, and whether it is possible for a machine to mimic the brain and recognize and predict visual input in complex, noisy, and uncertain circumstances. This project will address these questions from computational, psychological, and neuroscientific perspectives and deliver new models, data, and tools that promote the synergy between artificial intelligence and neuroscience.Investigators will design a model based on predictive coding in the brain, and test its ability to perform computer vision tasks and explain human behaviors and brain responses to naturalistic visual stimuli. The investigators will first develop a deep neural network referred to as the predictive coding network. Unlike existing feedforward neural networks, the currently predominant vision models, the predictive coding network has several defining features relevant to neural processing in the brain. It is bi-directional, processing information both bottom-up and top-down. It is recurrent, utilizing the same architecture for dynamic computation. It is parallel, allowing information processing to occur in parallel both within and across different layers. It is both discriminative and generative, reconciling image recognition and synthesis in a single framework. The predictive coding network will be evaluated against benchmark data sets. It is hypothesized to reach competitive performance with many fewer parameters than the state of the art. Then, the investigators will test the model's behaviors given naturalistic images degraded in various ways and/or presented for various durations. It is hypothesized that the model will be more robust and accurate after running for increasingly longer times and reach a time-accuracy tradeoff like human perception under similar conditions. To test this hypothesis, the investigators will perform human behavioral experiments and compare the model's behaviors against human behaviors. Further, the investigators will test the model's ability to explain brain responses to naturalistic images and videos, measured with functional magnetic resonance imaging and intracranial electroencephalography. The model is hypothesized to be able to predict the brain's dynamic activity and representation given naturalistic stimuli. The successful completion of this project is expected to deliver a brain-inspired vision model learnable and computable end-to-end. This model will empower machines with adaptive and robust vision and provide a tool for understanding the computational basis of biological vision.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.