Distributed Optimization for Machine Learning on Decentralized Data and Features

点击翻译按钮获取中文摘要

The need to scale up machine learning, in the presence of a rapid growth in both data volumes and model complexity, has sparked broad interests in developing distributed machine learning systems as well as parallel optimization algorithms. Most existing studies focus on partitioning computation among a tightly coupled cluster of machines either in a data-parallel fashion, to deal with large amounts of training samples, or a model-parallel fashion, to deal with large models, such as deep neural networks. In contrast, the goal of this research program is to glean insights and build models when the dataset (features, samples, labels or a combination of them) used for machine learning is inherently decentralized and owned by multiple participants/domains. Our long-term vision is to design reliable distributed algorithms and systems that can build models from decentralized data, without letting participants share original data with each other or to a central site. By effectively leveraging data from other domains, each participant is expected to enhance its predicting power over a model built only based on its local data, while the joint model built in a decentralized way is expected to approach and approximate the global model if all data were collected centrally. In the meantime, the sharing of model parameters among parties should also be minimized to preserve privacy and reduce communication overhead. Toward these objectives, we will introduce generic composite model structures that can jointly reap insights from data in different decentralization scenarios, including decentralization by features, by samples, by labels or decentralization by a combination of them. We will design theoretically inspired distributed optimization algorithms to solve these problems, develop effective communication compression techniques to reduce overhead, and also study implementation issues for specific applications. Specifically, our algorithms will be inspired by the recent advancements in the convergence of ADMM, stochastic gradient descent (SGD) and proximal SGD in an asynchronous and blockwise setting. Our communication compression techniques will be inspired by the opportunity to suppress model parameter transfers in flat regions of the optimization objective function, whereas existing literature mainly considers significance filters for gradients.  Finally, we will use the proposed model architectures and algorithms to solve various real-world applications in cross-domain recommender systems, multitasked natural language understanding and collaborative mobile edge computing. We will design specific model composition and decomposition structures as well as distributed algorithms based on the data decentralization pattern inherent in each problem.