Network-Centric Methods for Distributed Machine Learning and Optimization

本研究计划旨在开发大规模分布式机器学习和优化的方法。目前，社会在各个领域产生了大量的数据，从科学发现的工具（例如，大型强子对撞机，斯隆数字巡天）到在线社交网络（例如，Twitter和Facebook）。过去几十年来发展起来的机器学习、数据挖掘和模式识别方法现在被用来处理这些数据并建立有用的模型。这些方法已经产生了重大影响，并被用于检测和减轻欺诈性金融活动，搜索有效的新药和药品，过滤垃圾邮件，以及根据用户偏好定向投放广告。为了继续以加速的速度处理收集的数据，需要新的方法来利用分布式计算资源，例如云中的可用资源。该提案的短期目标是针对这种设置中出现的以网络为中心的问题：计算集群中节点之间传输信息时产生的延迟如何影响性能，如何确定哪些计算机应该交换信息以尽快完成训练，以及应该交换哪些信息。这些目标将通过理论分析和基于分散异步八卦算法的计算框架的开发、实现和评估来实现。我们的科学方法结合了通信网络、分布式信号处理、网络控制和信息论的技术，以及优化理论、图论、概率和随机过程的数学工具。本提案的长期目标是理解和描述分布式机器学习和优化中出现的基本限制和权衡；也就是说，在给定数据集上训练机器学习模型的速度有多快，以及机器学习如何从使用分布式处理中获得最佳收益。

This research proposal aims to develop methods for large-scale distributed machine learning and optimization. Society currently generates tremendous amounts of data across all sectors, from tools enabling scientific discovery (e.g., large hadron collider, Sloan digital sky survey) to online social networks (e.g., Twitter and Facebook). Machine learning, data mining, and pattern recognition methods developed over the past few decades are now used to process such data and build useful models. These methods have already had significant impacts and are being used to detect and mitigate fraudulent financial activity, to search for effective new drugs and pharmaceuticals, to filter spam email, and to target advertising based on user preferences. In order to continue processing data at the accelerated rate it is being gathered, new methods are needed to exploit distributed computing resources such as those available in the cloud. The short-term objectives of this proposal target network-centric issues arising in this setting: how do delays incurred when information is transmitted between nodes in a compute cluster effect performance, how to determine which computers should exchange information to accomplish training as fast as possible, and what information should be exchanged. These objectives will be realized both through theoretical analysis and through the development, implementation, and evaluation of a computational framework based on decentralized asynchronous gossip algorithms. Our scientific approach combines techniques from communication networks, distributed signal processing, networked control, and information theory, together with mathematical tools from optimization theory, graph theory, probability and stochastic processes. The long-term goals of this proposal are to understand and characterize fundamental limits and tradeoffs arising in distributed machine learning and optimization; namely, how fast can training machine learning models be trained on a given dataset and how can machine learning optimally benefit from the use of distributed processing.