This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizations
This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizations
复制标题
DOI：
10.48550/arxiv.2310.18589
复制
发表时间：
2023-10
期刊：
影响因子：
--
通讯作者：
Chiyu Ma;Brandon Zhao;Chaofan Chen;Cynthia Rudin
中科院分区：
文献类型：
--
作者：
Chiyu Ma;Brandon Zhao;Chaofan Chen;Cynthia Rudin
我们提出了ProtoConcepts，这是一种结合深度学习和使用原型部件的基于案例的推理的可解释图像分类方法。基于原型的图像分类中的现有工作使用“这看起来像”的推理过程，该过程通过找到原型部分并结合来自这些原型的证据来解剖测试图像以进行最终分类。然而，所有现有的原型基于部分的图像分类器只提供一对一的比较，其中单个训练图像块作为原型与我们的测试图像的一部分进行比较。通过这些单图像比较，通常难以识别正在比较的基本概念（例如，是比较颜色还是形状？'').我们所提出的方法修改了基于原型的网络的架构，而是学习使用多个图像补丁可视化的原型概念。具有相同原型的多个可视化使我们能够更容易地识别该原型所捕获的概念（例如，“测试图像和相关的训练补丁都是相同的蓝色阴影”），并允许我们的模型创建更丰富，更可解释的视觉解释。我们的实验表明，我们的“这看起来像那些”的推理过程可以作为一种修改，以广泛的现有的原型图像分类网络，同时实现基准数据集的精度相当。
We present ProtoConcepts, a method for interpretable image classification combining deep learning and case-based reasoning using prototypical parts. Existing work in prototype-based image classification uses a ``this looks like that'' reasoning process, which dissects a test image by finding prototypical parts and combining evidence from these prototypes to make a final classification. However, all of the existing prototypical part-based image classifiers provide only one-to-one comparisons, where a single training image patch serves as a prototype to compare with a part of our test image. With these single-image comparisons, it can often be difficult to identify the underlying concept being compared (e.g., ``is it comparing the color or the shape?''). Our proposed method modifies the architecture of prototype-based networks to instead learn prototypical concepts which are visualized using multiple image patches. Having multiple visualizations of the same prototype allows us to more easily identify the concept captured by that prototype (e.g., ``the test image and the related training patches are all the same shade of blue''), and allows our model to create richer, more interpretable visual explanations. Our experiments show that our ``this looks like those'' reasoning process can be applied as a modification to a wide range of existing prototypical image classification networks while achieving comparable accuracy on benchmark datasets.