CSR: Small: Latency-controlled Reduction of Data Center Expenses for Handling Bursty ML Inference Requests

数据中心经常必须过度配置用于在线机器学习（ML）推理的gpu，因为推理请求可能以突发的方式大量到达。这种过度供应导致数据中心不必要的高资本支出（CapEx）。该项目旨在设计一个整体管理框架，以延迟保证和最小化资本支出来处理突发ML推理请求。首先，所提出的框架将在同一GPU上有效地共同定位ML推理和训练工作负载，并具有延迟保证，以提高GPU利用率。因此，gpu可以主要用于突发请求时的推理，然后主要用于之后的训练，这可以减少所需gpu的数量，从而减少数据中心的资本支出。其次，将设计一种新的任务调度算法，将负相关的ML任务合并到相同的gpu上，以进一步降低资本支出。第三，为了避免昂贵的电力设施升级，通过利用大多数数据中心已经配备的现有储能设备在突发时提供额外的能源，数据中心的电力/冷却资本支出也将减少。最后，所有的软件、硬件和电源/冷却设施将作为一个整体框架进行协调，以有效地将数据中心管理为一个大型仓库级计算机。随着ML云服务变得越来越流行，快速增长的计算业务需求正在推动数据中心以更高的性能和更低的成本运行其gpu和服务器。该项目的成功将对数据中心设计产生积极影响，允许数据中心安全地处理ML推理突发，并具有理想的延迟保证和最小化的资本支出。目前，为了在保证延迟的情况下以越来越大的规模处理突发的机器学习工作负载，数据中心通常必须1)过度配置其推理gpu，并将其与用于训练的gpu分开，以最大限度地降低可能危及推理延迟的资源竞争风险；2)升级相应的电源/冷却设施，以容纳越来越多的gpu和服务器，为偶尔出现的最坏情况做好准备。这种资本投资的增加可能会给运营自己数据中心的ML/AI公司带来严重的负担。对于依赖云的初创企业和较小的ML/AI公司来说，较高的数据中心资本支出可能导致更高的云服务费用。提出的框架可以为数据中心提供延迟控制的资本支出减少，从而使那些蓬勃发展的ML/AI公司有更好的成功机会。此外，该项目还可以使其他执行密集型GPU计算的工作负载受益，例如高性能计算、大数据分析和云虚拟现实。该项目的成果将转化为俄亥俄州立大学新的和现有的本科和研究生课程。本项目将积极招募来自计算机领域代表性不足群体的学生。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Data centers must often over-provision the GPUs used for online machine learning (ML) inference because inference requests can arrive in a large number as bursts. Such over-provisioning results in unnecessarily high capital expenses (CapEx) for data centers. This project aims to design a holistic management framework that handles bursty ML inference requests with latency guarantees and minimized CapEx. First, the proposed framework will co-locate ML inference and training workloads efficiently on the same GPUs, with latency guarantees, for improving GPU utilization. As a result, GPUs can be used mostly for inference when a request burst comes, and then mostly for training afterwards, which can reduce the number of GPUs needed, and thus the data center’s CapEx. Second, a novel task scheduling algorithm will be designed to consolidate negatively correlated ML tasks onto the same GPUs for further reducing CapEx. Third, data center power/cooling CapEx will be reduced as well, by exploiting existing energy storage devices already equipped in most data centers to supply additional energy during a burst, in order to avoid expensive power facility upgrades. Finally, all the software, hardware, and power/cooling facilities will be coordinated as a holistic framework to efficiently manage a data center as one massive warehouse-scale computer.As ML cloud services are becoming increasingly popular, the fast-growing business demands for computing are driving data centers to run their GPUs and servers with higher performance and lower costs. The success of this project would positively impact data center design by allowing data centers to safely handle ML inference bursts, with desirable latency guarantees and minimized CapEx. Currently, in order to deal with bursty ML workloads at an increasing scale with latency guarantees, a data center must often 1) over-provision their inference GPUs and keep them separated from GPUs used for training in order to minimize the risk of resource competition that might jeopardize inference latency; and 2) upgrade the corresponding power/cooling facilities for hosting the increasing number of GPUs and servers to prepare for occasional worst-case scenarios. Such capital investment increases can impose a serious burden on ML/AI companies that run their own data centers. For start-ups and smaller ML/AI companies that rely on the cloud, the higher data center CapEx can lead to higher cloud service bills. The proposed framework can provide latency-controlled CapEx reduction for data centers, thus allowing those booming ML/AI companies to have a better chance of success. Further, this project could benefit other workloads that also perform intensive GPU computing, such as high-performance computing, big data analytics, and cloud virtual reality. The results of the project will be transitioned into new and existing undergraduate and graduate courses at The Ohio State University. Students from groups underrepresented in computing will be actively recruited in this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.