EAGER: Using Machine Learning to Increase the Operational Efficiency of Large Distributed Systems

大型分布式系统如今无处不在，是面向广泛客户和应用程序的可持续IT解决方案的一部分。私有云或公共云中的数据中心和高性能计算系统是复杂、高度分布式系统的两个例子：前者几乎每个人每天都在使用，后者被计算科学家用于推进科学和工程。这些复杂系统的高可用性和可靠性对用户体验的质量至关重要。这类系统的有效管理有助于它们的可用性和可靠性，并依赖于对用户集体需求的时间安排的先验知识和对各种系统组件的某些性能测量(例如，使用、温度、功率)的先验知识。该项目旨在提供一种系统的方法，通过开发能够在精细和粗略的时间尺度内高效和准确地预测到来的工作量的神经网络，来提高复杂的分布式系统的运行效率。这种工作负载预测可以通过推动专门旨在增强可靠性的主动式管理策略，显著提高数据中心和高性能系统的运营效率。对于数据中心，重点是积极减少由主动管理虚拟机调整大小和迁移而自动触发的性能票证。对于高性能计算系统，重点是预测硬件故障，以自动提高调度器的效率，直接冷却，并改善性能和内存带宽。

Large, distributed systems are nowadays ubiquitous and part of sustainable IT solutions to a broad range of customers and applications. Data centers in the private or public cloud and high performance computing systems are two examples of complex, highly distributed systems: the former are used by almost everyone on a daily basis, the latter are used by computational scientists for advancing science and engineering. High availability and reliability of these complex systems are important for the quality of user experience. Efficient management of such systems contributes to their availability and reliability, and relies on a priori knowledge of the timing of the collective demands of users and a priori knowledge of certain performance measures (e.g., usage, temperature, power) of various systems components.This project aims to provide a systematic methodology to improve the operational efficiency of complex, distributed systems by developing neural networks that can efficiently and accurately predict the incoming workload within fine and coarse time scales. Such workload prediction can dramatically improve the operational efficiency of data centers and high performance systems by driving proactive management strategies that specifically aim to enhance reliability. For datacenters, the focus is on actively reducing performance tickets that are automatically triggered by pro-actively managing virtual machine resizing and migration. For high performance computing systems the focus is on predicting hardware faults to autonomically improve the scheduler's efficiency, direct cooling, and improve performance and memory bandwidth.