CSR: Small: Latency-controlled Reduction of Data Center Expenses for Handling Bursty ML Inference Requests

数据中心必须经常过度配置用于在线机器学习（ML）推理的GPU，因为推理请求可能会以突发方式大量到达。这种过度配置会导致数据中心不必要的高资本支出（CapEx）。该项目旨在设计一个整体管理框架，以处理具有延迟保证和最小化CapEx的突发ML推理请求。首先，所提出的框架将在相同的GPU上有效地协同定位ML推理和训练工作负载，并保证延迟，以提高GPU利用率。因此，GPU可以在请求突发时主要用于推理，然后主要用于之后的训练，这可以减少所需的GPU数量，从而减少数据中心的CapEx。其次，将设计一种新的任务调度算法，将负相关的ML任务合并到相同的GPU上，以进一步降低资本支出。第三，通过利用大多数数据中心已经配备的现有能源存储设备在突发期间提供额外能源，数据中心电力/冷却资本支出也将减少，以避免昂贵的电力设施升级。最后，所有的软件、硬件和电源/冷却设施将作为一个整体框架进行协调，以便将数据中心作为一个大型仓库规模的计算机进行有效管理。随着ML云服务越来越受欢迎，快速增长的业务计算需求正在推动数据中心以更高的性能和更低的成本运行其GPU和服务器。该项目的成功将对数据中心设计产生积极影响，允许数据中心安全地处理ML推理突发，并具有理想的延迟保证和最小化的资本支出。目前，为了在延迟保证的情况下处理越来越大规模的突发ML工作负载，数据中心必须经常1）过度配置其推理GPU，并将其与用于训练的GPU分开，以最小化可能危及推理延迟的资源竞争的风险;以及2）升级相应的电源/冷却设施，以托管越来越多的GPU和服务器，为偶尔出现的最坏情况做好准备。这种资本投资的增加可能会给运营自己数据中心的ML/AI公司带来严重的负担。对于依赖云的初创企业和小型ML/AI公司来说，更高的数据中心资本支出可能会导致更高的云服务费用。拟议的框架可以为数据中心提供延迟控制的资本支出减少，从而使那些蓬勃发展的ML/AI公司有更好的成功机会。此外，该项目还可以使其他执行密集型GPU计算的工作负载受益，例如高性能计算，大数据分析和云虚拟现实。该项目的成果将过渡到新的和现有的本科和研究生课程在俄亥俄州州立大学。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Data centers must often over-provision the GPUs used for online machine learning (ML) inference because inference requests can arrive in a large number as bursts. Such over-provisioning results in unnecessarily high capital expenses (CapEx) for data centers. This project aims to design a holistic management framework that handles bursty ML inference requests with latency guarantees and minimized CapEx. First, the proposed framework will co-locate ML inference and training workloads efficiently on the same GPUs, with latency guarantees, for improving GPU utilization. As a result, GPUs can be used mostly for inference when a request burst comes, and then mostly for training afterwards, which can reduce the number of GPUs needed, and thus the data center’s CapEx. Second, a novel task scheduling algorithm will be designed to consolidate negatively correlated ML tasks onto the same GPUs for further reducing CapEx. Third, data center power/cooling CapEx will be reduced as well, by exploiting existing energy storage devices already equipped in most data centers to supply additional energy during a burst, in order to avoid expensive power facility upgrades. Finally, all the software, hardware, and power/cooling facilities will be coordinated as a holistic framework to efficiently manage a data center as one massive warehouse-scale computer.As ML cloud services are becoming increasingly popular, the fast-growing business demands for computing are driving data centers to run their GPUs and servers with higher performance and lower costs. The success of this project would positively impact data center design by allowing data centers to safely handle ML inference bursts, with desirable latency guarantees and minimized CapEx. Currently, in order to deal with bursty ML workloads at an increasing scale with latency guarantees, a data center must often 1) over-provision their inference GPUs and keep them separated from GPUs used for training in order to minimize the risk of resource competition that might jeopardize inference latency; and 2) upgrade the corresponding power/cooling facilities for hosting the increasing number of GPUs and servers to prepare for occasional worst-case scenarios. Such capital investment increases can impose a serious burden on ML/AI companies that run their own data centers. For start-ups and smaller ML/AI companies that rely on the cloud, the higher data center CapEx can lead to higher cloud service bills. The proposed framework can provide latency-controlled CapEx reduction for data centers, thus allowing those booming ML/AI companies to have a better chance of success. Further, this project could benefit other workloads that also perform intensive GPU computing, such as high-performance computing, big data analytics, and cloud virtual reality. The results of the project will be transitioned into new and existing undergraduate and graduate courses at The Ohio State University. Students from groups underrepresented in computing will be actively recruited in this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.