Controversial stimuli: Pitting neural networks against each other as models of human cognition
Controversial stimuli: Pitting neural networks against each other as models of human cognition
复制标题
DOI：
10.1073/pnas.1912334117
复制
发表时间：
2020-11-24
影响因子：
11.1
通讯作者：
Kriegeskorte, Nikolaus
中科院分区：
文献类型：
--
作者：
Golan, Tal;Raju, Prashant C.;Kriegeskorte, Nikolaus
不同的科学理论可以做出类似的预测。为了在理论之间做出判断，我们必须设计实验，使理论做出不同的预测。在这里，我们考虑将深度神经网络作为人类视觉识别模型进行比较的问题。为了有效地比较模型预测人类反应的能力，我们合成了有争议的刺激：不同模型产生不同反应的图像。我们将这种方法应用于两个视觉识别任务，手写数字（MNIST）和小自然图像（CIFAR-10）中的对象。对于每个任务，我们合成有争议的刺激，以最大限度地提高采用不同架构和识别算法的模型之间的分歧。人类受试者观看了数百个这些刺激，以及自然的例子，并判断每个数字/物体类别在每个图像中存在的概率。我们量化了每个模型预测人类判断的准确程度。性能最好的模型是MNIST的生成分析合成模型（基于变分自动编码器）和CIFAR-10的混合判别生成联合能量模型。这些对图像分布进行建模的深度神经网络（DNN）比纯粹的区分性DNN表现得更好，后者只学习将图像映射到标签。没有一个候选模型完全解释了人类的反应。有争议的刺激概括了对抗性示例的概念，避免了假设地面真实模型的需要。与自然图像不同，有争议的刺激不受刺激分布模型的约束，因此提供了严重的分布外测试，揭示了模型的归纳偏见。因此，有争议的刺激提供了强有力的探针模型和人类感知之间的差异。
Distinct scientific theories can make similar predictions. To adjudicate between theories, we must design experiments for which the theories make distinct predictions. Here we consider the problem of comparing deep neural networks as models of human visual recognition. To efficiently compare models' ability to predict human responses, we synthesize controversial stimuli: images for which different models produce distinct responses. We applied this approach to two visual recognition tasks, handwritten digits (MNIST) and objects in small natural images (CIFAR-10). For each task, we synthesized controversial stimuli to maximize the disagreement among models which employed different architectures and recognition algorithms. Human subjects viewed hundreds of these stimuli, as well as natural examples, and judged the probability of presence of each digit/object category in each image. We quantified how accurately each model predicted the human judgments. The best-performing models were a generative analysis-by-synthesis model (based on variational autoencoders) for MNIST and a hybrid discriminative-generative joint energy model for CIFAR-10. These deep neural networks (DNNs), which model the distribution of images, performed better than purely discriminative DNNs, which learn only to map images to labels. None of the candidate models fully explained the human responses. Controversial stimuli generalize the concept of adversarial examples, obviating the need to assume a ground truth model. Unlike natural images, controversial stimuli are not constrained to the stimulus distribution models are trained on, thus providing severe out-of-distribution tests that reveal the models' inductive biases. Controversial stimuli therefore provide powerful probes of discrepancies between models and human perception.