The Non-IID Data Quagmire of Decentralized Machine Learning
The Non-IID Data Quagmire of Decentralized Machine Learning
复制标题
DOI：
--
复制
发表时间：
2019-10
期刊：
影响因子：
--
通讯作者：
Kevin Hsieh;Amar Phanishayee;O. Mutlu;Phillip B. Gibbons
中科院分区：
文献类型：
--
作者：
Kevin Hsieh;Amar Phanishayee;O. Mutlu;Phillip B. Gibbons
许多大规模机器学习（ML）应用程序需要对在不同设备和位置生成的数据集进行分散式学习。这些数据集对分散式学习提出了重大挑战，因为它们的不同上下文导致设备/位置之间的数据分布严重倾斜。在本文中，我们通过对一种常见类型的数据偏斜进行分散式DNN训练的详细实验研究来更好地理解这一挑战：跨设备/位置的数据标签的偏斜分布。我们的研究表明：（i）偏斜的数据标签是分散式学习的一个基本和普遍的问题，导致许多ML应用程序，DNN模型，训练数据集和分散式学习算法的严重准确性损失;（ii）这个问题对于具有批量归一化的DNN模型特别具有挑战性;（iii）数据偏斜的程度是问题难度的关键决定因素。基于这些发现，我们提出了SkewScout，这是一种系统级的方法，可以将分散式学习算法的通信频率调整为数据分区之间的（偏斜引起的）准确性损失。我们还表明，组归一化可以恢复批量归一化的准确性损失。
Many large-scale machine learning (ML) applications need to perform decentralized learning over datasets generated at different devices and locations. Such datasets pose a significant challenge to decentralized learning because their different contexts result in significant data distribution skew across devices/locations. In this paper, we take a step toward better understanding this challenge by presenting a detailed experimental study of decentralized DNN training on a common type of data skew: skewed distribution of data labels across devices/locations. Our study shows that: (i) skewed data labels are a fundamental and pervasive problem for decentralized learning, causing significant accuracy loss across many ML applications, DNN models, training datasets, and decentralized learning algorithms; (ii) the problem is particularly challenging for DNN models with batch normalization; and (iii) the degree of data skew is a key determinant of the difficulty of the problem. Based on these findings, we present SkewScout, a system-level approach that adapts the communication frequency of decentralized learning algorithms to the (skew-induced) accuracy loss between data partitions. We also show that group normalization can recover much of the accuracy loss of batch normalization.