Testing the limits of deep neural network models of human vision with optimized stimuli

用优化的刺激测试人类视觉深度神经网络模型的极限

• 批准号: 1948004
• 负责人: Nikolaus Kriegeskorte
• 金额: $ 81.9万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-07-01 至 2024-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1948004&HistoricalAwards=false
• 关键词: Testing; limits; deep; neural; network

The sense of vision gives people an instant picture of the world, enabling rapid recognition of objects and others, and to understand their relationships and the layout of the scene. Machine vision is essential to many applications of artificial intelligence, but cannot yet emulate the richness and robustness of human vision. The study of human vision and the development of machine vision have mutually shaped each other in a virtuous cycle. The design of deep neural networks, the kind of artificial neural networks that now dominate machine vision, was inspired by neurobiological principles. In turn, neuroscientists have recently found that deep neural networks trained to recognize objects provide the best current model of human and primate vision. However, current artificial neural networks still fail to capture visual recognition capabilities of the human brain. The goal of this project is to learn what computational mechanisms best explain human vision. To achieve this, the research will develop and apply a novel methodology — "controversial stimuli." Controversial stimuli are computer-generated visual images optimized to cause two neural network models to disagree about their content. Presenting such a stimulus to a human observer will identify neural network models that mimic human vision. These stimuli will systematically compare artificial neural network models to human brains, and find ways to improve the models. This project will generate scientific insights on human vision and engineering insights for machine vision. The project will also generate outreach activities to improve public understanding of the power and limitations of neural networks, and their relationship to human intelligence.The project will develop methods for the synthesis of controversial stimuli, images optimized to adjudicate among alternative deep neural network models of human vision with brain and behavioral data. An initial behavioral experiment will challenge humans to classify and rate various controversial stimuli. Further experiments will design and employ synthetic stimuli for adjudicating between deep neural network models on the basis of brain activity measurements. Hemodynamic responses to the stimuli will be measured in the human ventral visual stream with functional magnetic resonance imaging (fMRI). Each of the fMRI experiments will test a different aspect of deep neural network modeling of visual neural responses, including the distinction between discriminative and generative image classifiers. The stimulus synthesis algorithms that will be developed in this project, as well as the resulting stimuli and corresponding fMRI and behavioral datasets, will be shared with the scientific community as an open resource.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

视觉给人一种即时的世界图景，使人们能够快速识别物体和其他物体，并理解它们之间的关系和场景的布局。机器视觉对人工智能的许多应用都是必不可少的，但还不能像人类视觉那样丰富和健壮。人类视觉的研究和机器视觉的发展在良性循环中相互塑造。深度神经网络的设计受到神经生物学原理的启发，这种人工神经网络现在主导着机器视觉。反过来，神经学家最近发现，经过训练以识别物体的深层神经网络提供了目前人类和灵长类视觉的最佳模型。然而，目前的人工神经网络仍然无法捕捉到人脑的视觉识别能力。这个项目的目标是学习什么计算机制最能解释人类的视觉。为了实现这一目标，这项研究将开发并应用一种新的方法--“有争议的刺激”。有争议的刺激是计算机生成的视觉图像，经过优化后，会导致两个神经网络模型对其内容产生分歧。向人类观察者展示这样的刺激将识别出模仿人类视觉的神经网络模型。这些刺激将系统地将人工神经网络模型与人脑进行比较，并找到改进模型的方法。该项目将产生对人类视觉的科学见解和对机器视觉的工程见解。该项目还将开展推广活动，以提高公众对神经网络的能力和局限性，以及它们与人类智能的关系的了解。该项目将开发合成有争议的刺激的方法，优化图像，以根据大脑和行为数据在人类视觉的替代深度神经网络模型中进行判断。最初的行为实验将挑战人类对各种有争议的刺激进行分类和评级。进一步的实验将设计和使用合成刺激，以便在大脑活动测量的基础上在深度神经网络模型之间进行判断。使用功能性磁共振成像(FMRI)在人体腹侧视觉流中测量对刺激的血流动力学反应。每个fMRI实验都将测试视觉神经反应的深度神经网络建模的不同方面，包括辨别性和生成性图像分类器之间的区别。将在这个项目中开发的刺激合成算法，以及产生的刺激和相应的功能磁共振成像和行为数据集，将作为开放资源与科学界共享。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Controversial stimuli: Pitting neural networks against each other as models of human cognition

• DOI: 10.1073/pnas.1912334117
• 发表时间: 2020-11-24
• 期刊: PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA
• 影响因子: 11.1
• 作者: Golan, Tal;Raju, Prashant C.;Kriegeskorte, Nikolaus
• 通讯作者: Kriegeskorte, Nikolaus

Distinguishing representational geometries with controversial stimuli: Bayesian experimental design and its application to face dissimilarity judgments

区分代表性几何与有争议的刺激：贝叶斯实验设计及其在面对相异性判断中的应用

• 发表时间: 2022
• 期刊: SVRHM 2022 Workshop @ NeurIPS
• 作者: Golan, Tal;Guo, Wenxuan;Kriegeskorte, Nikolaus
• 通讯作者: Kriegeskorte, Nikolaus