Scaling-Up Distributed Processing of Data Streams for Machine Learning
Scaling-Up Distributed Processing of Data Streams for Machine Learning
复制标题
DOI：
10.1109/jproc.2020.3021381
复制
发表时间：
2020-05
影响因子：
20.6
通讯作者：
M. Nokleby;Haroon Raja;W. Bajwa
中科院分区：
文献类型：
--
作者：
M. Nokleby;Haroon Raja;W. Bajwa
机器学习在许多领域的新兴应用-包括在线社交网络、遥感、物联网(IoT)系统、智能电网等-涉及不断收集数据流样本并从中学习。将流数据实时合并到学习的机器学习模型中，对于改进这些应用中的推理是必不可少的。此外，这些应用程序通常涉及由于物理原因(例如物联网系统和智能电网)而固有地收集在地理上分布的实体处的数据，或者出于内存、存储、计算和/或隐私原因而故意分布在多个计算机器上的数据。在这种分布式流传输环境中训练机器学习模型需要在物理实体之间的通信链路上以协作方式解决随机优化(SO)问题。当流数据速率与单个计算实体的处理能力和/或通信链路的速率相比较高时，这提出了一个具有挑战性的问题：在计算能力和/或通信速率的限制下，如何最好地利用传入数据来进行机器学习模型的分布式训练？近几十年来，出现了大量关于分布式在线优化的研究，以解决这一问题和相关问题。本文回顾了最近发展起来的在计算和带宽受限条件下的大规模分布式SO的方法，重点是收敛分析，它明确地解释了计算、通信和流传输速率之间的不匹配，并提供了阶数最优收敛的充分条件。特别是，它着重于解决：1)分布式随机凸问题和2)分布式主成分分析，这是一个具有允许全局收敛的几何结构的非凸问题。对于这种方法，本文讨论了面对高速率流数据时，分布式算法设计方面的最新进展。此外，它还回顾了这些方法背后的理论保证，这些保证表明，存在这样的制度，在这些制度下，系统可以从以阶数最优速率进行的流数据的分布式处理中学习-几乎像在一台超级强大的机器上处理所有数据一样快。
Emerging applications of machine learning in numerous areas—including online social networks, remote sensing, Internet-of-Things (IoT) systems, smart grids, and more—involve continuous gathering of and learning from streams of data samples. Real-time incorporation of streaming data into the learned machine learning models is essential for improved inference in these applications. Furthermore, these applications often involve data that are either inherently gathered at geographically distributed entities due to physical reasons, for example, IoT systems and smart grids, or that are intentionally distributed across multiple computing machines for memory, storage, computational, and/or privacy reasons. Training of machine learning models in this distributed, streaming setting requires solving stochastic optimization (SO) problems in a collaborative manner over communication links between the physical entities. When the streaming data rate is high compared with the processing capabilities of individual computing entities and/or the rate of the communications links, this poses a challenging question: How can one best leverage the incoming data for distributed training of machine learning models under constraints on computing capabilities and/or communications rate? A large body of research in distributed online optimization has emerged in recent decades to tackle this and related problems. This article reviews recently developed methods that focus on large-scale distributed SO in the compute- and bandwidth-limited regimes, with an emphasis on convergence analysis that explicitly accounts for the mismatch between computation, communication, and streaming rates and provides sufficient conditions for order-optimum convergence. In particular, it focuses on methods that solve: 1) distributed stochastic convex problems and 2) distributed principal component analysis, which is a nonconvex problem with the geometric structure that permits global convergence. For such methods, this article discusses recent advances in terms of distributed algorithmic designs when faced with high-rate streaming data. Furthermore, it reviews theoretical guarantees underlying these methods that show that there exist regimes in which systems can learn from distributed processing of streaming data at order-optimal rates—nearly as fast as if all the data were processed at a single superpowerful machine.