OAC Core: Towards Zero-Carbon Data Movement at the HPC and Cloud Data Centers with GreenDataFlow

随着商业和科学应用以越来越高的速度生成数据，与数据移动相关的碳足迹正成为一个关键问题，尤其是对于高性能计算(HPC)和云数据中心而言。虽然有大量的研究集中在数据传输期间的硬件级别和较低的网络堆栈层的电源管理技术，但对于服务器、HPC中心和云数据中心等计算系统在网络数据传输期间的应用层节能措施关注较少。这一领域的现有战略要么昂贵得令人望而却步，在短期内不切实际，要么为了追求更高的能源效率而牺牲绩效。该项目开发了一种创新的应用层解决方案，该解决方案经济高效，适用于立即部署，而且重要的是，在提高能效的同时不会影响性能。它具有自适应微调多个应用层和核心层传输参数的能力，从而保证计算和网络资源的高效利用。这进而在不影响端到端性能的情况下最大限度地减少了数据传输能耗。这一革命性的高能效数据传输方法凸显了该项目的创新和变革潜力。在该项目中开发的模型、算法和工具有望在端到端数据传输过程中增强性能和降低功耗，潜在地节省千兆瓦时的能源，并为美国经济贡献数百万美元的节省。此外，该项目寻求将研究见解渗透到所有层次的教育中。该项目开发了新的应用层模型、算法和工具，用于(1)预测和调整用于高性能计算机和云数据中心的高能效和高性能数据传输的最佳跨层传输参数组合；(2)基于深度强化学习的方法，能够适应各种网络和终端系统配置中的动态变化的条件；(3)准确估计因活跃的数据中心内和数据中心间网络链路上的数据传输速率变化而导致的伴随网络设备功耗，并动态重新调整传输速率以平衡能量与性能比；以及(4)为HPC管理员和云服务提供商提供一套基于服务级别协议的节能传输算法，以实现动态可调整的性能和能效目标。建议的模型和算法的评估和验证是在现实场景中与德克萨斯理工大学的HPC中心和IBM的分布式云管理小组合作进行的。该项目的研究成果将填补HPC和云数据中心在数据传输能效方面的重大空白。该项目的最终目标是将研究活动转化为强大的、生产质量的软件库，从而为处理大量数据的一系列用户社区减少数据移动的碳足迹。该项目将通过制定研究生和本科生课程、K-12外展计划、夏令营、招募少数群体以及扩大对计算机的参与来实现更广泛的影响。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

As commercial and scientific applications generate data at increasingly high rates, the carbon footprint associated with data movement is becoming a critical concern, particularly for High-Performance Computing (HPC) and Cloud data centers. While there is substantial research focusing on power management techniques at the hardware level and lower networking stack layers during data transfers, little attention has been paid to energy-saving measures at the application layer for computing systems such as servers, HPC centers, and Cloud data centers during network data transmission. The existing strategies in this realm are either prohibitively expensive, impractical in the short term, or sacrifice performance in pursuit of increased energy efficiency. This project develops an innovative application-layer solution, which is cost-effective, practical for immediate deployment, and importantly, does not compromise performance while boosting energy efficiency. It possesses the ability to adaptively fine-tune several application-layer and kernel-layer transfer parameters, thereby guaranteeing efficient utilization of computing and networking resources. This, in turn, minimizes data transfer energy consumption without undermining end-to-end performance. This revolutionary approach to energy-efficient data transfers underscores the innovation and transformative potential of this project. The models, algorithms, and tools developed within this project are poised to augment performance and reduce power consumption during end-to-end data transfers, potentially saving gigawatt hours of energy and contributing millions of dollars in savings to the US economy. Furthermore, this project seeks to permeate research insights across all tiers of education. The well-structured research activities promise to benefit for K-12, undergraduate, and graduate students alike, fostering their academic growth and nurturing future scientists in this critical field.This project develops novel application-layer models, algorithms, and tools for (1) prediction and tuning of the best cross-layer transfer parameter combination for energy-efficient and high-performance data transfers at the HPC and Cloud data centers; (2) a deep reinforcement learning-based approach that can adapt to the dynamically changing conditions in a wide range of network and end system configurations; (3) accurate estimation of the accompanying network device power consumption due to changing data transfer rate on the active intra- and inter-data center network links and dynamic readjustment of the transfer rate to balance the energy vs. performance ratio; and (4) a suite of service level agreement based energy-efficient transfer algorithms to the HPC administrators and Cloud service providers for dynamically adjustable performance and energy efficiency goals. The evaluation and validation of the proposed models and algorithms are performed in realistic scenarios in collaboration with the HPC Center at Texas Tech University and the Distributed Cloud Management group at IBM. The research outcomes of this project will fill a significant gap in the data transfer energy efficiency in HPC and Cloud data centers. This project's eventual goal is to translate the research activities into robust, production-quality software libraries that will reduce the carbon footprint of data movement for a range of user communities dealing with large amounts of data. The project will enable wider broader impacts through the development of graduate and undergraduate curricula, K-12 outreach programs, summer boot camps, the recruitment of minority groups, and broadening participation in computing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.