Collaborative Research: CNS Core: Small: Optimizing Large-Scale Heterogeneous ML Platforms

大规模人工智能和机器学习（AI/ML）平台在当前的数据革命中发挥着至关重要的作用。为了最大限度地减少用户的工作量，需要在可能异构的计算集群上部署复杂工作流的端到端解决方案。然而，这种“按钮式”部署背后的调度和资源管理问题是具有挑战性的。如果不加以解决，这些昂贵的系统将严重得不到充分利用，导致不必要的电力消耗和温室气体排放。该项目将为分布式大规模AI/ML系统制定有效的资源分配政策，以应对挑战。具体来说，该项目将通过分布式优化加速和并行化AI/ML平台中主导工作负载的大规模优化和推理任务，为异构设置中的掉队者提供容错性和鲁棒性。在分布式优化的基础上，该项目将进一步调度AI/ML工作流，并在子任务之间具有优先级约束。最后，在被分配的资源是可交换的情况下，在作业之间公平有效地分配异构资源，这对于具有图形处理单元（gpu）和其他加速器的AI/ML平台至关重要。该项目将为学术界和工业界使用的AI/ML平台的调度和资源分配提供新的基本算法。算法思想将在核心、经典模型的背景下发展，因此将比AI/ML平台应用更广泛，例如，网络、存储、供应链管理等。该项目将通过为初中学生开发数学速成课程、为初中和高中学生开发暑期课程、为本科生开发暑期研究课程等活动，扩大科学、技术、工程、数学领域代表性不足群体的参与。该项目将在项目网站https://adamwierman.com/optimizing-large-scale-heterogeneous-ml-platforms/上向研究社区提供其软件工件、数据集和研究结果。工件将至少维护10年。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Large-scale artificial intelligence and machine learning (AI/ML) platforms are playing a vital role in the current data revolution. To minimize efforts from users, an end-to-end solution is desired to deploy complex workflow over possibly heterogeneous computing clusters. However, the scheduling and resource management problems behind such “push-button” deployment are challenging. If left unsolved, these costly systems will be severely under-utilized, leading to unnecessary electricity consumption and greenhouse gas emissions. This project will develop efficient resource allocation policies for distributed, large-scale AI/ML systems to tackle the challenges. Specifically, this project will accelerate and parallelize the large-scale optimization and inference tasks that dominate workloads in AI/ML platforms via distributed optimization that provides fault tolerance and robustness to stragglers in heterogeneous settings. Built upon the distributed optimization, the project will further schedule AI/ML workflows with precedence constraints among sub-tasks. Finally, heterogeneous resources are allocated among jobs fairly and efficiently in the case where the resources being allocated are exchangeable, which is key for AI/ML platforms with graphic processing units (GPUs) and other accelerators. The project will provide new fundamental algorithms for scheduling and resource allocation in AI/ML platforms used across academia and industry. The algorithmic ideas will be developed in the context of core, classical models and so will apply more broadly than AI/ML platforms, e.g., to networking, storage, supply chain management, and beyond. The project will seek to broaden the participation of underrepresented groups in Science, Technology, Engineering and Mathematics by planned activities including the development of accelerated mathematics programs for middle school students, summer programs for middle-school and high-school students, and summer research programs for undergraduate students.The project will make its software artifacts, datasets, and research results available to the research community on the project website at https://adamwierman.com/optimizing-large-scale-heterogeneous-ml-platforms/ Artifacts will be maintained for a minimum of 10 years.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.