SUMMIT: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizations
SUMMIT: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizations
复制标题
DOI：
10.1109/tvcg.2019.2934659
复制
发表时间：
2020-01-01
影响因子：
5.2
通讯作者：
Chau, Duen Horng (Polo)
中科院分区：
文献类型：
--
作者：
Hohman, Fred;Park, Haekyu;Chau, Duen Horng (Polo)
深度学习越来越多地被用于决策任务。然而，理解神经网络如何产生最终预测仍然是一个根本性的挑战。现有的解释图像神经网络预测的工作往往侧重于解释单个图像或神经元的预测。由于预测通常是从数百万张图像上优化的数百万权重计算出来的，这样的解释很容易错过更大的图景。我们提出了Summit，这是一个交互式系统，它可扩展和系统地总结和可视化深度学习模型学习到的特征，以及这些特征是如何相互作用进行预测的。Summit引入了两种新的可扩展的摘要技术：(1)激活聚合发现重要神经元，(2)神经元影响聚合识别这些神经元之间的关系。顶峰将这些技术结合在一起，创建了新颖的属性图，揭示并总结了对模型结果做出贡献的关键神经元关联和子结构。Summit可扩展到大数据，例如具有120万个图像的ImageNet数据集，并利用神经网络功能可视化和数据集示例来帮助用户将大型、复杂的神经网络模型提取为紧凑、交互的可视化。我们介绍了神经网络探索场景，其中Summit帮助我们发现了对流行的大规模图像分类器的学习表示的多个令人惊讶的见解，并为未来的神经网络体系结构设计提供了信息。峰会可视化可以在现代网络浏览器上运行，并且是开源的。
Deep learning is increasingly used in decision-making tasks. However, understanding how neural networks produce final predictions remains a fundamental challenge. Existing work on interpreting neural network predictions for images often focuses on explaining predictions for single images or neurons. As predictions are often computed from millions of weights that are optimized over millions of images, such explanations can easily miss a bigger picture. We present SUMMIT, an interactive system that scalably and systematically summarizes and visualizes what features a deep learning model has learned and how those features interact to make predictions. SUMMIT introduces two new scalable summarization techniques: (1) activation aggregation discovers important neurons, and (2) neuron-influence aggregation identifies relationships among such neurons. SUMMIT combines these techniques to create the novel attribution graph that reveals and summarizes crucial neuron associations and substructures that contribute to a model's outcomes. SUMMIT scales to large data, such as the ImageNet dataset with 1.2M images, and leverages neural network feature visualization and dataset examples to help users distill large, complex neural network models into compact, interactive visualizations. We present neural network exploration scenarios where SUMMIT helps us discover multiple surprising insights into a prevalent, large-scale image classifier's learned representations and informs future neural network architecture design. The SUMMIT visualization runs in modern web browsers and is open-sourced.