Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
复制标题
DOI：
--
复制
发表时间：
2021-07
期刊：
影响因子：
--
通讯作者：
Xiaolong Ma;Geng Yuan;Xuan Shen;Tianlong Chen;Xuxi Chen;Xiaohan Chen;Ning Liu;Minghai Qin
中科院分区：
文献类型：
--
作者：
Xiaolong Ma;Geng Yuan;Xuan Shen;Tianlong Chen;Xuxi Chen;Xiaohan Chen;Ning Liu;Minghai Qin
长期以来，文献中对“中奖彩票”的实验设置和鉴定标准存在争议和不一致。为了调和这一点，我们重新审视彩票假设的定义，具有全面和更严格的条件。根据我们的新定义，我们展示了具体的证据来澄清中奖彩票是否存在于主要的DNN架构和/或应用中。通过大量的实验，我们对中奖彩票与各种实验因素之间的相关性进行了定量分析，并实证研究了我们观察到的模式。我们发现关键的训练超参数，如学习率和训练时间，以及架构特征，如容量和剩余连接，都与中奖彩票是否和何时被识别高度相关。在此基础上，我们总结了一个针对具体建筑特征的参数设置准则，希望能够促进彩票假设这一课题的研究进展。我们的代码是公开的：https://github.com/boone891214/sanity-check-LTH。
There have been long-standing controversies and inconsistencies over the experiment setup and criteria for identifying the"winning ticket"in literature. To reconcile such, we revisit the definition of lottery ticket hypothesis, with comprehensive and more rigorous conditions. Under our new definition, we show concrete evidence to clarify whether the winning ticket exists across the major DNN architectures and/or applications. Through extensive experiments, we perform quantitative analysis on the correlations between winning tickets and various experimental factors, and empirically study the patterns of our observations. We find that the key training hyperparameters, such as learning rate and training epochs, as well as the architecture characteristics such as capacities and residual connections, are all highly correlated with whether and when the winning tickets can be identified. Based on our analysis, we summarize a guideline for parameter settings in regards of specific architecture characteristics, which we hope to catalyze the research progress on the topic of lottery ticket hypothesis. Our codes are publicly available at: https://github.com/boone891214/sanity-check-LTH.