Hierarchical Distributed Machine Learning

点击翻译按钮获取中文摘要

Distributed Machine Learning is an emerging research area where user data cannot be centrally located at a single site but must be distributed across multiple devices that must cooperatively participate in the model training and inference tasks. In recent years there has been a significant interest from both the industry and academia to address various challenges in this area and a number of open source software libraries that provide support for distributed training and prediction have been developed. Nevertheless, a number of important challenges still remain to be addressed. In current systems the user nodes transmit raw gradient vectors computed on the locally available training data, which can leak sensitive information. Current solutions that encrypt the gradient vectors are based on secure multi-party computation techniques that have high communication complexity and do not scale to a large number of participating users. Secondly the models that should be trained may consist of millions of parameters and the transmission of raw gradient vectors would be bandwidth intensive. The proposed research will address these challenges by following two key approaches: (1) apply principles from cryptography to develop privacy preserving methods and (2) develop hierarchical clustering among users to improve efficiency. Our research methodology will also incorporate fairness criteria i.e., by design, variates such as sex, demographics, gender, race, ethnicity and other factors will not influence the decision making of the proposed algorithms. In collaboration with our industry partners --- Hitachi Solutions and Filament AI --- the proposed research will lead to real-world impact on a number of sectors including industrial automation, healthcare, self-driving cars and smart buildings. The HQP involved in the research project will be a diverse team that will be actively involved in a number of outreach activities promoting EDI initiatives, be trained in cutting edge research in machine learning and communication systems and will closely interact with both the industry partners.