Collaborative Research: Adaptive Techniques for Achieving End-to-End QoS in the I/O Stack on Petascale Multiprocessors

新兴的高端计算平台，如千万亿次的领导级机器，为复杂的建模和大规模仿真提供了新的视野。这些机器用于执行国家利益的数据密集型应用，如气候建模，宇宙微波背景辐射和天体物理热核闪光。虽然这些系统具有前所未有的峰值计算能力和存储容量，但一个关键挑战涉及可扩展I/O（输入输出）系统软件（也称为I/O堆栈）的设计和实现，该软件可以利用这些系统的功能进行科学发现和工程设计。不幸的是，目前还没有可用的机制来适应I/O堆栈范围的应用程序级QoS（服务质量）规范、监视和管理。该项目研究了一种革命性的方法，使用反馈控制理论，机器学习和优化的I/O堆栈的QoS感知管理。其目标是最大限度地提高I/O性能，从而提高国家利益的大规模应用程序的整体性能。该项目使用（1）机器学习和优化来确定应用程序级QoS到针对单个资源的子QoS的最佳分解，以及（2）反馈控制理论来分配由I/O堆栈管理的共享资源，以便在整个执行过程中满足指定的QoS。该项目使用NCAR、LBNL和ANL系统上的工作负载来测试已开发的I/O堆栈增强功能。它还涉及两项努力，以扩大参与：CISE访问工程周末（VIEW）和美国宇航局-航空航天教育服务项目（NASA-AESP）在中心的科学和学校（CSATS）。

Abstract Emerging high-end computing platforms, such as leadership-class machines at the petascale, provide new horizons for complex modeling and large-scale simulations. These machines are used to execute data intensive applications of national interest such as climate modeling, cosmic microwave background radiation, and astrophysical thermonuclear flashes. While these systems have unprecedented levels of peak computational power and storage capacity, a critical challenge concerns the design and implementation of scalable I/O (input-output) system software (also called I/O stack) that makes it possible to harness the power of these systems for scientific discovery and engineering design. Unfortunately, currently, there are no available mechanisms that accommodate I/O stack-wide, application-level QoS (quality-of-service)specification, monitoring, and management. This project investigates a revolutionary approach to the QoS-aware management of the I/O stack using feedback control theory, machine learning, and optimization. The goal is to maximize I/O performance and thus improve overall performance of large scale applications of national interest. The project uses (1) machine learning and optimization to determine the best decomposition of application-level QoS to sub-QoSs targeting individual resources, and (2) feedback control theory to allocate shared resources managed by the I/O stack such that the specified QoSs are satisfied throughout the execution. The project tests the developed I/O stack enhancements using the workloads at NCAR, LBNL and ANL systems. It also involves two efforts in broadening participation: CISE Visit in Engineering Weekends (VIEW) and NASA-Aerospace Education Services Project (NASA-AESP) at the Center for Science and the Schools (CSATS).