Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing-猫眼课题宝

DOI：

发表时间：

2021-06

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

M. Khodak;Renbo Tu;Tian Li;Liam Li;Maria-Florina Balcan;Virginia Smith;Ameet Talwalkar

调整超参数是机器学习管道中至关重要但艰巨的一部分。在联合学习中，超参数优化更具挑战性，在这种学习中，通过异质设备的分布式网络学习模型。在这里，需要在设备上保持数据并进行本地培训的需求使得很难有效地培训和评估配置。在这项工作中，我们调查了联合的超参数调整的问题。我们首先确定关键挑战，并显示如何适应标准方法以形成联合环境的基准。然后，通过与重量共享的神经结构搜索技术建立新的联系，我们引入了一种新方法FedEx，以加速联合联盟的高参数调整，该调整适用于广泛使用的联合联合优化方法，例如FedAvg和最近的变体。从理论上讲，我们表明，在跨设备的在线凸优化的设置中，联邦快递变体可以正确调整设备学习率。从经验上讲，我们表明，联邦快递可以在莎士比亚，女权主义者和CIFAR-10基准上胜过联盟的超级参数调谐的天然基线，从而使用相同的培训预算获得更高的准确性。

Tuning hyperparameters is a crucial but arduous part of the machine learning pipeline. Hyperparameter optimization is even more challenging in federated learning, where models are learned over a distributed network of heterogeneous devices; here, the need to keep data on device and perform local training makes it difficult to efficiently train and evaluate configurations. In this work, we investigate the problem of federated hyperparameter tuning. We first identify key challenges and show how standard approaches may be adapted to form baselines for the federated setting. Then, by making a novel connection to the neural architecture search technique of weight-sharing, we introduce a new method, FedEx, to accelerate federated hyperparameter tuning that is applicable to widely-used federated optimization methods such as FedAvg and recent variants. Theoretically, we show that a FedEx variant correctly tunes the on-device learning rate in the setting of online convex optimization across devices. Empirically, we show that FedEx can outperform natural baselines for federated hyperparameter tuning by several percentage points on the Shakespeare, FEMNIST, and CIFAR-10 benchmarks, obtaining higher accuracy using the same training budget.

Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing

文献摘要

相似文献