SHF-Small: Robust Methodologies for Effective Data Center Management

尽管数据中心无处不在，但对其有效管理知之甚少。在数据中心中，具有不同且不断变化的资源需求的多个应用的整合是常见的，因为硬件资源丰富，并且更好地利用系统的机会很多，由于应用和系统资源之间的不受管制的性能干扰，降低单个应用性能的机会也很多。 在尊重各个应用程序性能目标的同时，是否有可能最大限度地利用资源，或者同时满足这些相互冲突的措施是一种矛盾的说法？在本项目中，针对上述难题，提出了一种三管齐下的解决方法。首先，将在两年内对数千台数据中心服务器进行详细的大规模性能研究。本研究提供了当前工作负载需求的微观和宏观视图，工作负载资源对基本资源组件（包括CPU、内存、磁盘）的需求及其时间演变。该分析为数据中心中可扩展和高效的资源管理的开发提供了基线。第二，对数据中心基本组件的广泛实验将量化由于整合而导致的不同类别应用程序之间的性能干扰。这个实验推动了一个轻量级的分析器的开发，这是系统和应用程序不可知的。该方法通过标准工具提供的非侵入式低级测量来捕获应用程序资源需求。实验观察结果有可能在微观层面（即，在用作数据中心构建块的特定硬件组件处）和在宏观级别处（即，第三，开发了一种基于分析理论的工具，该工具使用分析器测量的资源需求作为输入，以准确预测同构和异构整合下的应用程序可伸缩性。该模型可用于在微观和宏观层面上预测虚拟化环境下的应用和系统性能，并提供整合建议，以满足预定义的用户或系统指定的性能目标。所提出的方法有可能提高在复杂工作负载下运行的数据中心中的资源分配的有效性，并显示出满足预定义性能目标的分配解决方案的出色潜力。定义用户和系统性能目标。这项研究将通过工业合作，特别是IBM研究和NEC研究实验室，影响实践的状态。更广泛地说，这项研究有可能对生产数据中心的管理产生重大影响。通过该项目，几名学生将做好准备，更好地满足复杂环境中性能建模和资源分配领域的行业需求。

Despite the ubiquity of data centers, little is known about their effective management. Consolidation of multiple applications with diverse and changing resource requirements is common in data centers as hardware resources are abundant and opportunities for better system usage are plenty, as are opportunities to degrade individual application performance due to unregulated performance interference between applications and system resources. Is it possible to maximize resource usage while respecting individual application performance targets or is it an oxymoron to simultaneously meet such conflicting measures? In this project, a solution methodology to the above difficult problem is proposed using a three-pronged approach.First, a detailed large scale performance study on several thousands of data center servers within a time period that spans two years is going to be conducted. This study provides a micro and macro view of current workload requirements, of workload resource demands onbasic resource components including CPU, memory, disk, and their temporal evolution. This analysis provides a baseline for the development of scalable and efficient resource management in data centers.Second, extensive experimentation on basic components of data centers is going to quantify performance interference among different classes of applications due to consolidation. This experimentation drives the development of a light-weight profiler that is system- and application-agnostic. The methodology captures application resource demands via non-intrusive low-level measurements that are provided via standard tools.The experimental observations have the potential to drive the development of resource allocation policies in data centers both at the micro level (i.e., at specific hardware components that are used as data center building blocks) and at the macro level (i.e., at the data center as a whole).Third, a queueing-theory based tool is developed that uses as input the resource demands measured by the profiler to accurately predict application scalability under homogeneous and heterogeneous consolidations. The model can be used to predict the application and system performance under virtualized environments at the micro and macro levels, and provide consolidation suggestions such that pre-defined user- or system-specified performance targets are met.The proposed methodologies have the potential to improve the effectiveness of resource allocation in data centers that operate under complex workloads and show excellent potential for allocation solutions that meet pre-defined user and system performance targets. This research will affect the state-of-the-practice via industrial collaborations, especially IBM Research and NEC Research Labs. More broadly, this research has the potential to make a strong impact in management of in-production data centers. Through this project, several students will be prepared to better meet industry demands in the areas of performance modeling and resource allocation in complex environments.