Hierarchical Distributed Machine Learning

分布式机器学习是一个新兴的研究领域，其中用户数据不能集中位于单个站点，而是必须分布在多个设备上，这些设备必须协同参与模型训练和推理任务。近年来，工业界和学术界都对解决这一领域的各种挑战产生了浓厚的兴趣，并且已经开发了许多为分布式训练和预测提供支持的开源软件库。然而，仍有一些重要挑战有待解决。在当前的系统中，用户节点传输在本地可用的训练数据上计算的原始梯度向量，这可能泄漏敏感信息。加密梯度向量的当前解决方案是基于安全多方计算技术，其具有高通信复杂度并且不能扩展到大量参与用户。其次，应该训练的模型可能由数百万个参数组成，并且原始梯度向量的传输将是带宽密集型的。拟议的研究将通过以下两个关键方法来解决这些挑战：（1）应用密码学原理来开发隐私保护方法;（2）在用户之间开发层次聚类以提高效率。我们的研究方法也将纳入公平标准，即，通过设计，诸如性别、人口统计、性别、种族、民族和其它因素的变量将不会影响所提出的算法的决策。通过与我们的行业合作伙伴Hitachi Solutions和Filament AI合作，拟议的研究将对工业自动化、医疗保健、自动驾驶汽车和智能建筑等多个领域产生现实影响。参与研究项目的HQP将是一个多元化的团队，将积极参与一些推广EDI倡议的外联活动，接受机器学习和通信系统前沿研究的培训，并将与行业合作伙伴密切互动。

Distributed Machine Learning is an emerging research area where user data cannot be centrally located at a single site but must be distributed across multiple devices that must cooperatively participate in the model training and inference tasks. In recent years there has been a significant interest from both the industry and academia to address various challenges in this area and a number of open source software libraries that provide support for distributed training and prediction have been developed. Nevertheless, a number of important challenges still remain to be addressed. In current systems the user nodes transmit raw gradient vectors computed on the locally available training data, which can leak sensitive information. Current solutions that encrypt the gradient vectors are based on secure multi-party computation techniques that have high communication complexity and do not scale to a large number of participating users. Secondly the models that should be trained may consist of millions of parameters and the transmission of raw gradient vectors would be bandwidth intensive. The proposed research will address these challenges by following two key approaches: (1) apply principles from cryptography to develop privacy preserving methods and (2) develop hierarchical clustering among users to improve efficiency. Our research methodology will also incorporate fairness criteria i.e., by design, variates such as sex, demographics, gender, race, ethnicity and other factors will not influence the decision making of the proposed algorithms. In collaboration with our industry partners --- Hitachi Solutions and Filament AI --- the proposed research will lead to real-world impact on a number of sectors including industrial automation, healthcare, self-driving cars and smart buildings. The HQP involved in the research project will be a diverse team that will be actively involved in a number of outreach activities promoting EDI initiatives, be trained in cutting edge research in machine learning and communication systems and will closely interact with both the industry partners.