Rethinking the performance comparison between SNNS and ANNS
Rethinking the performance comparison between SNNS and ANNS
复制标题
重新思考 SNN 和 ANN 之间的性能比较
DOI：
10.1016/j.neunet.2019.09.005
复制
发表时间：
2020-01-01
期刊：
影响因子：
7.8
通讯作者：
Xie, Yuan
中科院分区：
文献类型：
--
作者：
Deng, Lei;Wu, Yujie;Xie, Yuan
人工神经网络（ann）是人工智能的流行路径，通过成熟的模型、各种基准、开源数据集和强大的计算平台取得了显着的成功。脉冲神经网络（snn）是一类很有前途的模拟大脑神经元动力学的模型，在脑启发计算方面受到了广泛的关注，并被广泛应用于神经形态设备。然而，长期以来，人们对snn在实际应用中的价值一直存在争论和怀疑。除了从峰值驱动处理中获得的低功耗属性外，snn的性能通常不如ann，特别是在应用精度方面。最近，研究人员试图通过借鉴人工神经网络的学习方法（如反向传播）来训练高精度的SNN模型来解决这个问题。随着网络规模的不断扩大，该领域的快速发展不断产生惊人的结果，其增长路径似乎与深度学习的发展相似。尽管这些方法赋予snn接近ann精度的能力，但snn的天然优势和超越ann的方法可能会由于使用面向ann的工作负载和简单的评估指标而丢失。在本文中，我们以视觉识别任务为案例研究来回答“什么工作负载是snn的理想工作负载以及如何评估snn是有意义的”的问题。我们使用不同类型的数据集（面向人工神经网络和面向snn）、不同的处理模型、信号转换方法和学习算法设计了一系列对比测试。我们提出了关于应用精度和内存和计算成本的综合指标来评估这些模型，并进行了广泛的实验。我们证明了这样一个事实：在面向人工神经网络的工作负载上，snn无法击败它们的人工神经网络对手；而在面向snn的工作负载上，snn可以完全更好地执行。我们进一步证明，在snn中存在应用精度和执行成本之间的权衡，这将受到仿真时间窗口和触发阈值的影响。基于这些丰富的分析，我们为每个场景推荐最合适的模型。据我们所知，这是第一个使用系统比较来明确揭示从ann到snn的直接工作负载移植是不明智的工作，尽管许多工作都在这样做，而且全面的评估确实很重要。最后，我们强调迫切需要为snn建立一个具有更广泛任务、数据集和指标的基准测试框架。(C) 2019 Elsevier Ltd.版权所有。
Artificial neural networks (ANNs), a popular path towards artificial intelligence, have experienced remarkable success via mature models, various benchmarks, open-source datasets, and powerful computing platforms. Spiking neural networks (SNNs), a category of promising models to mimic the neuronal dynamics of the brain, have gained much attention for brain inspired computing and been widely deployed on neuromorphic devices. However, for a long time, there are ongoing debates and skepticisms about the value of SNNs in practical applications. Except for the low power attribute benefit from the spike-driven processing, SNNs usually perform worse than ANNs especially in terms of the application accuracy. Recently, researchers attempt to address this issue by borrowing learning methodologies from ANNs, such as backpropagation, to train high-accuracy SNN models. The rapid progress in this domain continuously produces amazing results with ever-increasing network size, whose growing path seems similar to the development of deep learning. Although these ways endow SNNs the capability to approach the accuracy of ANNs, the natural superiorities of SNNs and the way to outperform ANNs are potentially lost due to the use of ANN-oriented workloads and simplistic evaluation metrics.In this paper, we take the visual recognition task as a case study to answer the questions of "what workloads are ideal for SNNs and how to evaluate SNNs makes sense". We design a series of contrast tests using different types of datasets (ANN-oriented and SNN-oriented), diverse processing models, signal conversion methods, and learning algorithms. We propose comprehensive metrics on the application accuracy and the cost of memory & compute to evaluate these models, and conduct extensive experiments. We evidence the fact that on ANN-oriented workloads, SNNs fail to beat their ANN counterparts; while on SNN-oriented workloads, SNNs can fully perform better. We further demonstrate that in SNNs there exists a trade-off between the application accuracy and the execution cost, which will be affected by the simulation time window and firing threshold. Based on these abundant analyses, we recommend the most suitable model for each scenario. To the best of our knowledge, this is the first work using systematical comparisons to explicitly reveal that the straightforward workload porting from ANNs to SNNs is unwise although many works are doing so and a comprehensive evaluation indeed matters. Finally, we highlight the urgent need to build a benchmarking framework for SNNs with broader tasks, datasets, and metrics. (C) 2019 Elsevier Ltd. All rights reserved.