Testing the limits of deep neural network models of human vision with optimized stimuli

视觉使人们对世界有了即时的了解，能够快速识别物体和其他物体，并了解它们之间的关系和场景的布局。机器视觉对人工智能的许多应用至关重要，但还不能模仿人类视觉的丰富性和鲁棒性。人类视觉的研究和机器视觉的发展形成了一个良性循环。深度神经网络的设计，即现在主导机器视觉的人工神经网络，是受到神经生物学原理的启发。反过来，神经科学家最近发现，经过训练来识别物体的深度神经网络提供了当前人类和灵长类动物视觉的最佳模型。然而，目前的人工神经网络仍然无法捕捉人类大脑的视觉识别能力。这个项目的目标是学习什么计算机制能最好地解释人类的视觉。为了实现这一目标，这项研究将开发并应用一种新的方法——“有争议的刺激”。有争议的刺激是计算机生成的视觉图像，经过优化，使两个神经网络模型对其内容产生分歧。将这样的刺激呈现给人类观察者将识别出模仿人类视觉的神经网络模型。这些刺激将系统地将人工神经网络模型与人脑进行比较，并找到改进模型的方法。这个项目将产生关于人类视觉的科学见解和机器视觉的工程见解。该项目还将开展外展活动，以提高公众对神经网络的能力和局限性以及它们与人类智能的关系的理解。该项目将开发合成有争议刺激的方法，优化图像，以在具有大脑和行为数据的人类视觉的替代深度神经网络模型中进行裁决。最初的行为实验将挑战人类对各种有争议的刺激进行分类和评级。进一步的实验将设计和使用合成刺激，在大脑活动测量的基础上判断深度神经网络模型。对刺激的血流动力学反应将通过功能性磁共振成像（fMRI）在人体腹侧视觉流中测量。每个fMRI实验将测试视觉神经反应的深度神经网络建模的不同方面，包括判别和生成图像分类器之间的区别。本项目将开发的刺激合成算法，以及由此产生的刺激和相应的fMRI和行为数据集，将作为开放资源与科学界共享。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The sense of vision gives people an instant picture of the world, enabling rapid recognition of objects and others, and to understand their relationships and the layout of the scene. Machine vision is essential to many applications of artificial intelligence, but cannot yet emulate the richness and robustness of human vision. The study of human vision and the development of machine vision have mutually shaped each other in a virtuous cycle. The design of deep neural networks, the kind of artificial neural networks that now dominate machine vision, was inspired by neurobiological principles. In turn, neuroscientists have recently found that deep neural networks trained to recognize objects provide the best current model of human and primate vision. However, current artificial neural networks still fail to capture visual recognition capabilities of the human brain. The goal of this project is to learn what computational mechanisms best explain human vision. To achieve this, the research will develop and apply a novel methodology — "controversial stimuli." Controversial stimuli are computer-generated visual images optimized to cause two neural network models to disagree about their content. Presenting such a stimulus to a human observer will identify neural network models that mimic human vision. These stimuli will systematically compare artificial neural network models to human brains, and find ways to improve the models. This project will generate scientific insights on human vision and engineering insights for machine vision. The project will also generate outreach activities to improve public understanding of the power and limitations of neural networks, and their relationship to human intelligence.The project will develop methods for the synthesis of controversial stimuli, images optimized to adjudicate among alternative deep neural network models of human vision with brain and behavioral data. An initial behavioral experiment will challenge humans to classify and rate various controversial stimuli. Further experiments will design and employ synthetic stimuli for adjudicating between deep neural network models on the basis of brain activity measurements. Hemodynamic responses to the stimuli will be measured in the human ventral visual stream with functional magnetic resonance imaging (fMRI). Each of the fMRI experiments will test a different aspect of deep neural network modeling of visual neural responses, including the distinction between discriminative and generative image classifiers. The stimulus synthesis algorithms that will be developed in this project, as well as the resulting stimuli and corresponding fMRI and behavioral datasets, will be shared with the scientific community as an open resource.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.