Network-Centric Methods for Distributed Machine Learning and Optimization

该研究方案旨在开发大规模分布式机器学习和优化的方法。社会目前在所有部门都产生了海量数据，从支持科学发现的工具(例如大型强子对撞机、斯隆数字巡天)到在线社交网络(例如Twitter和Facebook)。过去几十年发展起来的机器学习、数据挖掘和模式识别方法现在被用来处理这些数据并建立有用的模型。这些方法已经产生了重大影响，正被用于检测和减少欺诈性金融活动，搜索有效的新药和药品，过滤垃圾邮件，并根据用户偏好定向广告。为了继续以收集数据的加速速度处理数据，需要新的方法来利用分布式计算资源，例如云中可用的资源。该提案的短期目标针对在这种情况下产生的以网络为中心的问题：在计算集群中的节点之间传输信息时产生的延迟如何影响性能、如何确定哪些计算机应该交换信息以尽可能快地完成培训，以及应该交换什么信息。这些目标将通过理论分析和基于分散的异步八卦算法的计算框架的开发、实施和评估来实现。我们的科学方法结合了通信网络、分布式信号处理、网络控制和信息论的技术，以及最优化理论、图论、概率和随机过程的数学工具。这项建议的长期目标是理解和表征分布式机器学习和优化中出现的基本限制和权衡；即，在给定的数据集上训练机器学习模型的速度有多快，以及机器学习如何从使用分布式处理中获得最佳好处。

This research proposal aims to develop methods for large-scale distributed machine learning and optimization. Society currently generates tremendous amounts of data across all sectors, from tools enabling scientific discovery (e.g., large hadron collider, Sloan digital sky survey) to online social networks (e.g., Twitter and Facebook). Machine learning, data mining, and pattern recognition methods developed over the past few decades are now used to process such data and build useful models. These methods have already had significant impacts and are being used to detect and mitigate fraudulent financial activity, to search for effective new drugs and pharmaceuticals, to filter spam email, and to target advertising based on user preferences. In order to continue processing data at the accelerated rate it is being gathered, new methods are needed to exploit distributed computing resources such as those available in the cloud. The short-term objectives of this proposal target network-centric issues arising in this setting: how do delays incurred when information is transmitted between nodes in a compute cluster effect performance, how to determine which computers should exchange information to accomplish training as fast as possible, and what information should be exchanged. These objectives will be realized both through theoretical analysis and through the development, implementation, and evaluation of a computational framework based on decentralized asynchronous gossip algorithms. Our scientific approach combines techniques from communication networks, distributed signal processing, networked control, and information theory, together with mathematical tools from optimization theory, graph theory, probability and stochastic processes. The long-term goals of this proposal are to understand and characterize fundamental limits and tradeoffs arising in distributed machine learning and optimization; namely, how fast can training machine learning models be trained on a given dataset and how can machine learning optimally benefit from the use of distributed processing.