XPS: FULL: Broad-Purpose, Aggressively Asynchronous and Theoretically Sound Parallel Large-scale Machine Learning

许多人工智能（AI）应用，如图像理解和自然语言处理，都依赖于机器学习（ML）方法来自动从大数据（大学习）中提取有价值的知识。高效的机器学习不仅需要高级数学模型和算法的专业知识，还需要大型计算机集群的经验，其中机器故障，内存/网络瓶颈，机器间延迟等问题必须通过复杂的系统编程来妥善处理。这种对“双重技能”的需求通常会阻止大规模AI向广泛的用户社区民主化，并且需要一个新的框架，该框架将ML和集群的分布式计算环境与类似单机的简单界面连接起来，允许ML从业者对后端细节不可知，并能够快速原型化或在集群上部署ML程序。这种需要的解决办法仍然很少。在这个项目中，PI为分布式系统上的ML开发了一个新的通用框架，提供了高效和理论上合理的协议（例如，通信、调度和分区功能）来编排异构计算机集群以变得可编程并且像单个大型计算机一样工作，正确执行分布式机器学习程序，速度比Hadoop和Spark等当前系统快几个数量级。有了这个新的框架，数据科学家将能够在不需要专门的工程和基础设施团队的情况下，对海量数据进行复杂模型的机器学习分析，从而使大学习更容易为社会所接受。具体而言，在四年的时间里，拟议的研究重点是三个技术目标：（1）构建大学习的系统框架，通过开发一种新的架构，支持大型ML程序的数据和模型并行执行，使用智能调度器，参数服务器和一致性控制器，可配置为提供模型/数据并行化，同步方案，负载平衡，容错和多实例租赁;（2）构建多级抽象编程接口，支持大规模应用程序的基本和高级ML算法的简单并行编程;（3）基于独特的见解，如块一致性和有界同步下的容错性，在所提出的系统上对分布式ML算法进行理论分析。目标是开发一个系统框架，以实现ML程序的通用，自动和有效的并行化。

Many artificial intelligence (AI) applications such as image understanding and natural language processing rely on Machine Learning (ML) methods to automatically extract valuable knowledge from Big Data (Big Learning). Efficient ML requires not only expertise in advanced mathematical models and algorithms, but also experiences with large computer clusters where issues such as machine failures, memory/network bottlenecks, inter-machine latencies must be properly handled through complex system programming. Such demand on "dual skill" often prevents democratizing large-scale AI to wide user communities, and necessitates a new framework that bridges ML and the distributed computing environment of a cluster with a single-machine-like simple interface, allowing ML practitioners to be agnostic about the backend details, and able to quickly prototype or deploy ML programs on clusters. Solutions to such a need remain rare. In this project the PIs develop a new general purpose framework for ML on distributed systems, offering highly efficient and theoretically justified protocols (e.g. communication, scheduling, and partitioning functions) to orchestrate a heterogeneous computer cluster to become programmable and act like a single big computer, and execute distributed ML programs correctly and at a speed orders of magnitude faster than current systems such as Hadoop and Spark. With this new framework, data scientists will be able to conduct ML analytics with complex models on massive data without the need for dedicated engineering and infrastructure teams, allowing Big Learning more readily accessible to society. Specifically, over a four year span, the proposed research focuses on three technical aims: (1) Building a System Framework for Big Learning, by developing a new architecture that supports both data- and model-parallel execution of large ML programs, using intelligent scheduler, parameter server, and consistency controller that are configurable to provide flexible options for model/data parallelization, synchronization schemes, load balance, fault tolerance, and multi-instance tenancy; (2) Building a Multi-Level-Abstraction Programming Interface, which supports easy parallel programming of both basic and advanced ML algorithms for large-scale applications; and (3)Conducting theoretical analysis of distributed ML algorithms on the proposed system, based on unique insights such as block consistency and error-tolerance under bounded synchronism. The goal is to develop a system framework to achieve general, automatic, and effective parallelization of ML programs.