Rich Information is Affordable: A Systematic Performance Analysis of Second-order Optimization Using K-FAC
Rich Information is Affordable: A Systematic Performance Analysis of Second-order Optimization Using K-FAC
复制标题
DOI：
10.1145/3394486.3403265
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
Yuichiro Ueno;Kazuki Osawa;Yohei Tsuji;Akira Naruse;Rio Yokota
中科院分区：
文献类型：
--
作者：
Yuichiro Ueno;Kazuki Osawa;Yohei Tsuji;Akira Naruse;Rio Yokota
来自一阶和二阶导数的丰富信息矩阵在深度学习的理论和实际问题中有许多潜在的应用。然而，计算这些信息矩阵是非常昂贵的，这种巨大的成本目前限制了其应用于有关泛化，超参数调整和深度神经网络优化的重要问题。信息矩阵最具挑战性的用例之一是将其用作优化器的预处理器，因为信息矩阵需要在每一步都进行更新。在这项工作中，我们在ImageNet上训练ResNet-50期间计算Fisher信息矩阵时进行了逐步的性能分析，并表明开销可以减少到与执行单个SGD步骤的成本相同。我们还表明，由此产生的Fisher预处理优化器可以收敛在1/3的时期数相比，SGD，同时实现相同的Top-1验证精度。这是第一次使用K-FAC实现这样的精度，同时减少训练时间以匹配SGD。
Rich information matrices from first and second-order derivatives have many potential applications in both theoretical and practical problems in deep learning. However, computing these information matrices is extremely expensive and this enormous cost is currently limiting its application to important problems regarding generalization, hyperparameter tuning, and optimization of deep neural networks. One of the most challenging use cases of information matrices is their use as a preconditioner for the optimizers, since the information matrices need to be updated every step. In this work, we conduct a step-by-step performance analysis when computing the Fisher information matrix during training of ResNet-50 on ImageNet, and show that the overhead can be reduced to the same amount as the cost of performing a single SGD step. We also show that the resulting Fisher preconditioned optimizer can converge in 1/3 the number of epochs compared to SGD, while achieving the same Top-1 validation accuracy. This is the first work to achieve such accuracy with K-FAC while reducing the training time to match that of SGD.