Hierarchical Distributed Machine Learning

分布式机器学习是一个新兴的研究领域，用户数据不能集中在单个站点，而是必须分布在多个设备上，这些设备必须协同参与模型训练和推理任务。近年来，工业界和学术界都对解决这一领域的各种挑战非常感兴趣，并且已经开发了许多为分布式训练和预测提供支持的开源软件库。然而，仍有一些重要的挑战有待解决。在现有的系统中，用户节点传输的是基于本地可用训练数据计算的原始梯度向量，这可能会泄露敏感信息。目前的梯度矢量加密解决方案基于安全的多方计算技术，这些技术具有较高的通信复杂度，并且不能扩展到大量参与用户。其次，需要训练的模型可能包含数百万个参数，原始梯度向量的传输将占用大量带宽。提出的研究将通过以下两种关键方法来解决这些挑战：(1)应用密码学原理来开发隐私保护方法；(2)在用户之间开发分层聚类以提高效率。我们的研究方法还将纳入公平标准，即通过设计，性别、人口统计学、性别、种族、民族和其他因素等变量不会影响所提议算法的决策。通过与我们的行业合作伙伴——日立解决方案（Hitachi Solutions）和Filament AI的合作，拟议的研究将对工业自动化、医疗保健、自动驾驶汽车和智能建筑等多个领域产生现实影响。参与研究项目的HQP将是一个多元化的团队，将积极参与一些推广EDI计划的外展活动，接受机器学习和通信系统前沿研究方面的培训，并将与行业合作伙伴密切互动。

Distributed Machine Learning is an emerging research area where user data cannot be centrally located at a single site but must be distributed across multiple devices that must cooperatively participate in the model training and inference tasks. In recent years there has been a significant interest from both the industry and academia to address various challenges in this area and a number of open source software libraries that provide support for distributed training and prediction have been developed. Nevertheless, a number of important challenges still remain to be addressed. In current systems the user nodes transmit raw gradient vectors computed on the locally available training data, which can leak sensitive information. Current solutions that encrypt the gradient vectors are based on secure multi-party computation techniques that have high communication complexity and do not scale to a large number of participating users. Secondly the models that should be trained may consist of millions of parameters and the transmission of raw gradient vectors would be bandwidth intensive. The proposed research will address these challenges by following two key approaches: (1) apply principles from cryptography to develop privacy preserving methods and (2) develop hierarchical clustering among users to improve efficiency. Our research methodology will also incorporate fairness criteria i.e., by design, variates such as sex, demographics, gender, race, ethnicity and other factors will not influence the decision making of the proposed algorithms. In collaboration with our industry partners --- Hitachi Solutions and Filament AI --- the proposed research will lead to real-world impact on a number of sectors including industrial automation, healthcare, self-driving cars and smart buildings. The HQP involved in the research project will be a diverse team that will be actively involved in a number of outreach activities promoting EDI initiatives, be trained in cutting edge research in machine learning and communication systems and will closely interact with both the industry partners.