OAC Core: Towards Zero-Carbon Data Movement at the HPC and Cloud Data Centers with GreenDataFlow

随着商业和科学应用程序以越来越高的速度生成数据，与数据移动相关的碳足迹正成为一个关键问题，特别是对于高性能计算（HPC）和云数据中心。在数据传输过程中，硬件层和较低网络堆栈层的电源管理技术已经得到了大量的研究，但在服务器、高性能计算中心和云数据中心等计算系统的网络数据传输过程中，应用层的节能措施却很少受到关注。这一领域的现有策略要么昂贵得令人望而却步，要么短期内不切实际，要么为了追求更高的能源效率而牺牲性能。该项目开发了一种创新的应用层解决方案，具有成本效益，可立即部署，重要的是，在提高能源效率的同时不影响性能。它具有自适应微调多个应用层和内核层传输参数的能力，从而保证了计算和网络资源的有效利用。这反过来又可以在不损害端到端性能的情况下最大限度地减少数据传输能耗。这种革命性的节能数据传输方法凸显了该项目的创新和变革潜力。该项目中开发的模型、算法和工具将提高端到端数据传输的性能，降低功耗，可能节省千兆瓦时的能源，为美国经济节省数百万美元。此外，该项目旨在将研究见解渗透到所有教育层次。结构良好的研究活动有望使K-12、本科生和研究生受益，促进他们的学术成长，培养这一关键领域的未来科学家。该项目开发了新颖的应用层模型、算法和工具，用于(1)预测和调优最佳跨层传输参数组合，以实现高性能高性能高性能高性能高性能高性能高性能的高性能高性能数据传输；(2)基于深度强化学习的方法，可以适应各种网络和终端系统配置中动态变化的条件；(3)准确估计由于数据中心内和数据中心间的主动网络链路上数据传输速率的变化而伴随的网络设备功耗，并动态调整传输速率以平衡能量与性能比；(4)为高性能计算管理员和云服务提供商提供一套基于服务水平协议的节能传输算法，以实现动态调整的性能和能效目标。我们与得克萨斯理工大学的HPC中心和IBM的分布式云管理小组合作，在实际场景中对所提出的模型和算法进行评估和验证。本项目的研究成果将填补HPC和云数据中心在数据传输能效方面的重大空白。该项目的最终目标是将研究活动转化为健壮的、生产质量的软件库，这将为处理大量数据的用户社区减少数据移动的碳足迹。该项目将通过开发研究生和本科生课程、K-12外展项目、夏季新兵训练营、招募少数群体以及扩大计算机领域的参与，产生更广泛的影响。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

As commercial and scientific applications generate data at increasingly high rates, the carbon footprint associated with data movement is becoming a critical concern, particularly for High-Performance Computing (HPC) and Cloud data centers. While there is substantial research focusing on power management techniques at the hardware level and lower networking stack layers during data transfers, little attention has been paid to energy-saving measures at the application layer for computing systems such as servers, HPC centers, and Cloud data centers during network data transmission. The existing strategies in this realm are either prohibitively expensive, impractical in the short term, or sacrifice performance in pursuit of increased energy efficiency. This project develops an innovative application-layer solution, which is cost-effective, practical for immediate deployment, and importantly, does not compromise performance while boosting energy efficiency. It possesses the ability to adaptively fine-tune several application-layer and kernel-layer transfer parameters, thereby guaranteeing efficient utilization of computing and networking resources. This, in turn, minimizes data transfer energy consumption without undermining end-to-end performance. This revolutionary approach to energy-efficient data transfers underscores the innovation and transformative potential of this project. The models, algorithms, and tools developed within this project are poised to augment performance and reduce power consumption during end-to-end data transfers, potentially saving gigawatt hours of energy and contributing millions of dollars in savings to the US economy. Furthermore, this project seeks to permeate research insights across all tiers of education. The well-structured research activities promise to benefit for K-12, undergraduate, and graduate students alike, fostering their academic growth and nurturing future scientists in this critical field.This project develops novel application-layer models, algorithms, and tools for (1) prediction and tuning of the best cross-layer transfer parameter combination for energy-efficient and high-performance data transfers at the HPC and Cloud data centers; (2) a deep reinforcement learning-based approach that can adapt to the dynamically changing conditions in a wide range of network and end system configurations; (3) accurate estimation of the accompanying network device power consumption due to changing data transfer rate on the active intra- and inter-data center network links and dynamic readjustment of the transfer rate to balance the energy vs. performance ratio; and (4) a suite of service level agreement based energy-efficient transfer algorithms to the HPC administrators and Cloud service providers for dynamically adjustable performance and energy efficiency goals. The evaluation and validation of the proposed models and algorithms are performed in realistic scenarios in collaboration with the HPC Center at Texas Tech University and the Distributed Cloud Management group at IBM. The research outcomes of this project will fill a significant gap in the data transfer energy efficiency in HPC and Cloud data centers. This project's eventual goal is to translate the research activities into robust, production-quality software libraries that will reduce the carbon footprint of data movement for a range of user communities dealing with large amounts of data. The project will enable wider broader impacts through the development of graduate and undergraduate curricula, K-12 outreach programs, summer boot camps, the recruitment of minority groups, and broadening participation in computing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.