CNS Core: Small: Toward Globally-Optimal Resource Distribution and Computation Acceleration in Multi-Tenant and Heterogeneous Machine Learning Systems

CNS 核心：小型：在多租户和异构机器学习系统中实现全局最优资源分配和计算加速

• 批准号: 2008248
• 负责人: Eric Xing
• 金额: $ 49.99万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2008248&HistoricalAwards=false
• 关键词: CNS; Core; Small; Toward; Globally

In the era of large-scale deep learning (DL) and massive data, existing hardware systems have struggled to effectively accommodate heavy and complex computing workload due to difficulties in scheduling highly dynamic, heterogeneous, and competing tasks from many users over many machines in a cluster or data-center environment. This project aims to develop a "1-click" demand-aware and responsive software system capable of simultaneously training a wide spectrum of DL tasks, using a new resource management architecture that automatically and adaptively chooses the most effective distributed training/serving techniques and their hyperparameters to achieve best overall efficiency of multiple tasks in such environment.This interdisciplinary project innovates in distributed systems design, DL algorithm design, and related industrial applications and theoretical analyses, with the following thrusts: 1: Develop a framework for "ML-aware" resource management and scheduling of multiple simultaneously running training tasks. 2: Develop principled strategies for resource management and scheduling for serving, streaming, and heterogeneous-task settings. 3: Optimize memory resources for training large-parameter models by developing holistic approaches to maximize computation throughput subject to device memory bounds. A limited-scope but rigorous and practical theoretical analysis of some of the proposed architectures will also be performed. This project addresses the needs from the academic and industrial communities and will have a broad impact on both. It will provide easy-to-use tools that reduce the time to set-up and facilitate large-scale experimentation, while reducing the required costs, whether measured in cluster access quotas or dollars spent on cloud services. The impact on commercial practitioners will be even greater, by improving their productivity by an order of magnitude or more, as they must contend with heterogeneous computing and network resources that are shared among many users as well as the need to run many jobs on a regular basis.The team will release and/or open-source the code at http://sailing-lab.wixsite.com/sailing-pmls to benefit researchers and practitioners, to share their lessons learned to advocate more research in machine learning (ML) systems problems, and also to democratize high-performance ML systems and make them accessible to non-ML-educated software developers and society at large, such as industrial and manufacturing, healthcare, biology, social science, and finance, where results may have a catalytic impact. The team will publish results at a variety of top tier conferences, including machine learning (NIPS, ICML), systems (OSDI, SOSP, USENIX), and data mining (KDD, WWW).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在大规模深度学习（DL）和海量数据的时代，现有的硬件系统一直在努力有效地适应繁重和复杂的计算工作负载，这是由于在集群或数据中心环境中调度来自许多用户的高度动态、异构和竞争性任务的困难。该项目旨在开发一个“一键”的需求感知和响应的软件系统，能够同时训练广泛的DL任务，使用一种新的资源管理架构，自动和自适应地选择最有效的分布式训练/服务技术及其超参数，以实现在这种环境中多个任务的最佳整体效率。DL算法的设计，以及相关的工业应用和理论分析，具有以下几个方面的重点：1：开发一个“ML感知”的资源管理和调度的多个同时运行的训练任务的框架。2：为服务、流和异构任务设置制定资源管理和调度的原则性策略。第三章：通过开发整体方法来优化内存资源，以训练大参数模型，从而最大限度地提高受设备内存限制的计算吞吐量。一个有限的范围，但严格和实用的理论分析，一些建议的架构也将执行。该项目解决了学术界和工业界的需求，并将对两者产生广泛的影响。它将提供易于使用的工具，减少设置时间并促进大规模实验，同时降低所需的成本，无论是以集群访问配额还是花费在云服务上的美元来衡量。对商业从业者的影响将更大，通过提高他们的生产力一个数量级或更多，因为他们必须应对异构计算和网络资源，这些资源在许多用户之间共享，以及需要定期运行许多作业。该团队将在www.example.com上发布和/或开源代码http://sailing-lab.wixsite.com/sailing-pmls以使研究人员和从业者受益，分享他们的经验教训，倡导对机器学习（ML）系统问题进行更多的研究，并使高性能ML系统民主化，使其能够被未受过ML教育的软件开发人员和整个社会所访问，例如工业和制造业，医疗保健，生物学，社会科学和金融，其结果可能具有催化作用。该团队将在各种顶级会议上发布结果，包括机器学习（NIPS，ICML），系统（OSDI，SOSP，USENIX）和数据挖掘（KDD，WWW）。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

项目成果

Cuttlefish: Low-Rank Model Training without All the Tuning

• DOI: 10.48550/arxiv.2305.02538
• 发表时间: 2023-05
• 期刊: ArXiv
• 作者: Hongyi Wang;Saurabh Agarwal;Pongsakorn U-chupala;Yoshiki Tanaka;Eric P. Xing;Dimitris Papailiopoulos

On Optimizing the Communication of Model Parallelism

• DOI: 10.48550/arxiv.2211.05322
• 发表时间: 2022-11
• 期刊: ArXiv
• 作者: Yonghao Zhuang;Hexu Zhao;Lianmin Zheng;Zhuohan Li;Eric P. Xing;Qirong Ho;Joseph E. Gonzalez

MPCFormer: fast, performant and private Transformer inference with MPC

• DOI: 10.48550/arxiv.2211.01452
• 发表时间: 2022-11
• 期刊: ArXiv
• 作者: Dacheng Li;Rulin Shao;Hongyi Wang;Han Guo;Eric P. Xing;Haotong Zhang

Making Scalable Meta Learning Practical

• DOI: 10.48550/arxiv.2310.05674
• 发表时间: 2023-10
• 期刊: ArXiv
• 作者: Sang Keun Choe;Sanket Vaibhav Mehta;Hwijeen Ahn;W. Neiswanger;Pengtao Xie;Emma Strubell;Eric P. Xing

Betty: An Automatic Differentiation Library for Multilevel Optimization

• DOI: 10.48550/arxiv.2207.02849
• 发表时间: 2022-07
• 期刊: ArXiv
• 作者: Sang Keun Choe;W. Neiswanger;P. Xie;Eric P. Xing