XPS: FULL: Broad-Purpose, Aggressively Asynchronous and Theoretically Sound Parallel Large-scale Machine Learning

许多人工智能（AI）应用，如图像理解和自然语言处理，都依赖于机器学习（ML）方法从大数据（大学习）中自动提取有价值的知识。高效的机器学习不仅需要高级数学模型和算法方面的专业知识，还需要大型计算机集群的经验，其中机器故障、内存/网络瓶颈、机器间延迟等问题必须通过复杂的系统编程来妥善处理。这种对“双重技能”的需求往往阻碍了大规模人工智能向广泛用户社区的民主化，并且需要一个新的框架，将ML和集群的分布式计算环境连接起来，使用类似单机的简单界面，允许ML从业者不知道后端细节，并能够快速原型化或在集群上部署ML程序。解决这种需求的办法仍然很少。在这个项目中，pi为分布式系统上的机器学习开发了一个新的通用框架，提供了高效和理论上合理的协议（例如通信，调度和分区功能），以编排异构计算机集群，使其成为可编程的，并像单个大型计算机一样运行，并正确执行分布式机器学习程序，其速度比当前的系统（如Hadoop和Spark）快几个数量级。有了这个新框架，数据科学家将能够在不需要专门的工程和基础设施团队的情况下，对大量数据进行复杂模型的ML分析，从而使大学习更容易为社会所接受。具体而言，在四年的时间跨度内，提出的研究重点是三个技术目标：(1)通过开发支持大型机器学习程序的数据和模型并行执行的新架构，构建大学习的系统框架，使用智能调度器，参数服务器和可配置的一致性控制器，为模型/数据并行，同步方案，负载平衡，容错和多实例租赁提供灵活的选项；(2)构建多层抽象编程接口，支持大规模应用中基本和高级ML算法的轻松并行编程；(3)基于有界同步下的块一致性和容错等独特见解，对所提系统的分布式ML算法进行理论分析。目标是开发一个系统框架来实现机器学习程序的通用、自动和有效的并行化。

Many artificial intelligence (AI) applications such as image understanding and natural language processing rely on Machine Learning (ML) methods to automatically extract valuable knowledge from Big Data (Big Learning). Efficient ML requires not only expertise in advanced mathematical models and algorithms, but also experiences with large computer clusters where issues such as machine failures, memory/network bottlenecks, inter-machine latencies must be properly handled through complex system programming. Such demand on "dual skill" often prevents democratizing large-scale AI to wide user communities, and necessitates a new framework that bridges ML and the distributed computing environment of a cluster with a single-machine-like simple interface, allowing ML practitioners to be agnostic about the backend details, and able to quickly prototype or deploy ML programs on clusters. Solutions to such a need remain rare. In this project the PIs develop a new general purpose framework for ML on distributed systems, offering highly efficient and theoretically justified protocols (e.g. communication, scheduling, and partitioning functions) to orchestrate a heterogeneous computer cluster to become programmable and act like a single big computer, and execute distributed ML programs correctly and at a speed orders of magnitude faster than current systems such as Hadoop and Spark. With this new framework, data scientists will be able to conduct ML analytics with complex models on massive data without the need for dedicated engineering and infrastructure teams, allowing Big Learning more readily accessible to society. Specifically, over a four year span, the proposed research focuses on three technical aims: (1) Building a System Framework for Big Learning, by developing a new architecture that supports both data- and model-parallel execution of large ML programs, using intelligent scheduler, parameter server, and consistency controller that are configurable to provide flexible options for model/data parallelization, synchronization schemes, load balance, fault tolerance, and multi-instance tenancy; (2) Building a Multi-Level-Abstraction Programming Interface, which supports easy parallel programming of both basic and advanced ML algorithms for large-scale applications; and (3)Conducting theoretical analysis of distributed ML algorithms on the proposed system, based on unique insights such as block consistency and error-tolerance under bounded synchronism. The goal is to develop a system framework to achieve general, automatic, and effective parallelization of ML programs.