CCF-BSF: AF: Small: Convex and Non-Convex Distributed Learning

从机器翻译和语音识别，到自动驾驶汽车的控制，再到蛋白质结构预测和药物设计，机器学习在解决许多困难的科学、工程和人工智能任务方面，是一种越来越重要的方法。机器学习的核心思想是使用示例和数据来自动训练系统执行某些任务。因此，机器学习的成功取决于大量训练数据的可用性和我们处理这些数据的能力。最近机器学习的成功很大程度上是由现在可以收集的大量数据（文本、图像、视频等）推动的。但所有这些数据也需要被处理和学习——事实上，数据洪流已经在很大程度上转移了瓶颈，从数据的可用性到我们处理数据的能力。特别是，所涉及的大量数据不能再在单个计算机上存储和处理。因此，分布式机器学习是现代大规模机器学习的关键要素，在许多相互通信的计算机上处理和学习数据。该项目的目标是为研究分布式机器学习提供一个严格的框架，并通过它开发分布式学习的有效方法，并从理论上理解这些方法的好处，以及分布式学习的固有局限性。pi方法的一个核心组件是将分布式学习建模为随机优化问题，其中不同的机器接收来自相同源分布的样本，从而允许专门利用不同机器上数据之间的相关性的方法和分析。这对于研究多台计算机的可用性如何帮助减少学习的计算成本至关重要。此外，该项目还包括更具有挑战性的情况，即不同机器上的数据性质存在显著差异（例如，当不同的机器服务于不同的地理区域时，或者当每台机器都是个人设备，从单个用户收集数据时）。在这种情况下，所提出的研究方法是将分布式学习与个性化或自适应相结合，pi认为这不仅可以提高学习性能，还可以更好地利用分布式计算。这是一项国际合作，由美以两国科学基金会（BSF）联合资助。该项目汇集了两位在机器学习和优化相关主题上广泛合作的pi。

Machine learning is an increasingly important approach in tackling many difficult scientific, engineering and artificial intelligence tasks, ranging from machine translation and speech recognition, through control of self driving cars, to protein structure prediction and drug design.  The core idea of machine learning is to use examples and data to automatically train a system to perform some task.  Accordingly, the success of machine learning is tied to availability of large amounts of training data and our ability to process it.  Much of the recent success of machine learning is fueled by the large amounts of data (text, images, videos, etc) that can now be collected. But all this data also needs to be processed and learned from---indeed this data flood has shifted the bottleneck, to a large extent, from availability of data to our ability to process it. In particular, the amounts of data involved can no longer be stored and handled on single computers.  Consequently, distributed machine learning, where data is processed and learned from on many computers that communicate with each other, is a crucial element of modern large scale machine learning.The goal of this project is to provide a rigorous framework for studying distributed machine learning, and through it develop efficient methods for distributed learning and a theoretical understanding of the benefits of these methods, as well as the inherent limitations of distributed learning.  A central component in the PIs' approach is to model distributed learning as a stochastic optimization problem, where different machines receive samples drawn from the same source distribution, thus allowing methods and analysis that specifically leverage the relatedness between data on different machines.  This is crucial for studying how availability of multiple computers can aid in reducing the computational cost of learning. Furthermore, the project also encompasses the more challenging case where there are significant differences between the nature of the data on different machines (for instance, when different machines serve different geographical regions, or when each machine is a personal device, collecting data from a single user).  In such a situation, the proposed approach to be studied is to integrate distributed learning with personalization or adaptation, which the PIs argue can not only improve learning performance, but also better leverage distributed computation.This is an international collaboration, made possible through joint funding with the US-Israel Binational Science Foundation (BSF). The project brings together two PIs that have worked together extensively on related topics in machine learning and optimization.