CRCNS Research Proposal: Predictive Coding Network for Human Vision

CRCNS 研究提案：人类视觉预测编码网络

• 批准号: 2112773
• 负责人: Zhongming Liu
• 金额: $ 104.6万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2112773&HistoricalAwards=false
• 关键词: CRCNS; Research; Proposal; Predictive; Coding

This project aims to advance scientific knowledge about human vision and use neuroscience to enhance artificial intelligence for computer vision. Vision is central to how humans see and explore the world. About a dozen brain regions work together to process visual information within a fraction of a second. It is hypothesized that these brain regions actively predict one's visual surroundings and use errors of prediction to update their internal representations and guide actions. However, it is not clear how the brain performs computations for recognition and prediction, and whether it is possible for a machine to mimic the brain and recognize and predict visual input in complex, noisy, and uncertain circumstances. This project will address these questions from computational, psychological, and neuroscientific perspectives and deliver new models, data, and tools that promote the synergy between artificial intelligence and neuroscience.Investigators will design a model based on predictive coding in the brain, and test its ability to perform computer vision tasks and explain human behaviors and brain responses to naturalistic visual stimuli. The investigators will first develop a deep neural network referred to as the predictive coding network. Unlike existing feedforward neural networks, the currently predominant vision models, the predictive coding network has several defining features relevant to neural processing in the brain. It is bi-directional, processing information both bottom-up and top-down. It is recurrent, utilizing the same architecture for dynamic computation. It is parallel, allowing information processing to occur in parallel both within and across different layers. It is both discriminative and generative, reconciling image recognition and synthesis in a single framework. The predictive coding network will be evaluated against benchmark data sets. It is hypothesized to reach competitive performance with many fewer parameters than the state of the art. Then, the investigators will test the model's behaviors given naturalistic images degraded in various ways and/or presented for various durations. It is hypothesized that the model will be more robust and accurate after running for increasingly longer times and reach a time-accuracy tradeoff like human perception under similar conditions. To test this hypothesis, the investigators will perform human behavioral experiments and compare the model's behaviors against human behaviors. Further, the investigators will test the model's ability to explain brain responses to naturalistic images and videos, measured with functional magnetic resonance imaging and intracranial electroencephalography. The model is hypothesized to be able to predict the brain's dynamic activity and representation given naturalistic stimuli. The successful completion of this project is expected to deliver a brain-inspired vision model learnable and computable end-to-end. This model will empower machines with adaptive and robust vision and provide a tool for understanding the computational basis of biological vision.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目旨在促进人类视觉的科学知识，并利用神经科学来增强计算机视觉的人工智能。视觉是人类观察和探索世界的核心。大约12个大脑区域一起工作，在不到一秒钟的时间内处理视觉信息。据假设，这些大脑区域主动预测一个人的视觉环境，并使用预测错误来更新其内部表示和指导行动。然而，目前尚不清楚大脑是如何进行识别和预测的计算，也不清楚机器是否有可能在复杂、嘈杂和不确定的环境中模仿大脑，识别和预测视觉输入。该项目将从计算、心理学和神经科学的角度解决这些问题，并提供新的模型、数据和工具，促进人工智能和神经科学之间的协同。研究人员将设计一个基于大脑预测编码的模型，并测试其执行计算机视觉任务的能力，并解释人类行为和大脑对自然视觉刺激的反应。研究人员将首先开发一种被称为预测编码网络的深度神经网络。与目前占主导地位的前馈神经网络不同，预测编码网络具有几个与大脑中的神经处理相关的定义特征。它是双向的，自下而上和自上而下地处理信息。它是循环的，使用相同的体系结构进行动态计算。它是并行的，允许信息处理在不同层内和不同层之间并行进行。它兼具鉴别性和生成性，在单一的框架内协调了图像识别和合成。将对照基准数据集对预测编码网络进行评估。它被假设用比最先进的技术状态少得多的参数达到有竞争力的性能。然后，研究人员将测试以不同方式降级和/或呈现不同持续时间的自然主义图像的模型行为。假设模型在运行越来越长的时间后将更加稳健和准确，并达到类似条件下人类感知的时间精度折衷。为了验证这一假设，研究人员将进行人类行为实验，并将模型的行为与人类行为进行比较。此外，研究人员将测试该模型解释大脑对自然图像和视频的反应的能力，这些图像和视频是通过功能磁共振成像和颅内脑电图测量的。该模型被假设为能够预测大脑在自然刺激下的动态活动和表现。这个项目的成功完成有望提供一个端到端的可学习和可计算的大脑启发视觉模型。该模型将使机器具有自适应和健壮的视觉，并为理解生物视觉的计算基础提供工具。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning

• 发表时间: 2021-11
• 期刊: ArXiv
• 作者: Yizhen Zhang;Minkyu Choi;Kuan Han;Zhongming Liu