Collaborative Research: Adaptive Techniques for Achieving End-to-End QoS in the I/O Stack on Petascale Multiprocessors

新兴的高端计算平台，如千万亿级的领导级机器，为复杂的建模和大规模模拟提供了新的视野。这些机器被用来执行国家利益的数据密集型应用，如气候建模、宇宙微波背景辐射和天体物理热核闪。虽然这些系统具有前所未有的峰值计算能力和存储容量，但一个关键的挑战是设计和实现可扩展的I/O(输入-输出)系统软件(也称为I/O堆栈)，使其能够利用这些系统的能力进行科学发现和工程设计。遗憾的是，目前还没有可用的机制来适应I/O堆栈范围内、应用程序级别的服务质量(Qos)规范、监控和管理。该项目研究了一种革命性的方法，使用反馈控制理论、机器学习和优化来实现对I/O堆栈的Qos感知管理。目标是最大限度地提高I/O性能，从而提高涉及国家利益的大型应用程序的整体性能。该项目使用(1)机器学习和优化来确定将应用级服务质量最好地分解为针对单个资源的子服务质量，以及(2)反馈控制理论来分配由I/O堆栈管理的共享资源，使得指定的服务质量在整个执行过程中得到满足。该项目使用NCAR、LBNL和ANL系统的工作负载测试开发的I/O堆栈增强功能。它还涉及两项扩大参与的努力：工程周末的CESE访问(VIEW)和科学和学校中心(CSATS)的NASA-航空航天教育服务项目(NASA-AESP)。

Emerging high-end computing platforms, such as leadership-class machines at the petascale, provide new horizons for complex modeling and large-scale simulations. These machines are used to execute data intensive applications of national interest such as climate modeling, cosmic microwave background radiation, and astrophysical thermonuclear flashes. While these systems have unprecedented levels of peak computational power and storage capacity, a critical challenge concerns the design and implementation of scalable I/O (input-output) system software (also called I/O stack) that makes it possible to harness the power of these systems for scientific discovery and engineering design. Unfortunately, currently, there are no available mechanisms that accommodate I/O stack-wide, application-level QoS (quality-of-service)specification, monitoring, and management.This project investigates a revolutionary approach to the QoS-aware management of the I/O stack using feedback control theory, machine learning, and optimization. The goal is to maximize I/O performance and thus improve overall performance of large scale applications of national interest. The project uses (1) machine learning and optimization to determine the best decomposition of application-level QoS to sub-QoSs targeting individual resources, and (2) feedback control theory to allocate shared resources managed by the I/O stack such that the specified QoSs are satisfied throughout the execution. The project tests the developed I/O stack enhancements using the workloads at NCAR, LBNL and ANL systems. It also involves two efforts in broadening participation: CISE Visit in Engineering Weekends (VIEW) and NASA-Aerospace Education Services Project (NASA-AESP) at the Center for Science and the Schools (CSATS).