CRII: CSR: Online Analysis of Disk I/O for Automatic Storage System Optimization

当今的关键应用，包括基因组分析、气候模拟、药物发现、空间观测以及计算化学和高能物理中的数值模拟，本质上都是数据密集型的。存储性能瓶颈是限制数据密集型应用程序性能和可扩展性的主要威胁。该项目的目标是开发一个自优化并行存储系统的通用框架，可以缓解存储性能瓶颈，从而通过加速众多科学领域的创新过程对社会产生重大影响。结果还将通过受益于广泛的并行存储系统，包括磁盘阵列，键值存储和并行/分布式文件系统，提高存储系统的知识水平。更广泛的影响包括通过夏令营指导和培训K-12学生，促进代表性不足的学生参与科学和工程。这项研究开发了新颖的，理论上有根据的，实验验证的方法，用于在线检测和自动消除磁盘I/O瓶颈。具体目标包括开发：（i）在数据流挖掘和社会网络分析理论的指导下，用于连续监控磁盘I/O请求并有效分析它们的新在线方法，以及（ii）在装箱，图着色和网络流理论指导下的自动触发自优化技术，可以仔细规划自适应数据布局以提高磁盘I/O性能。通过分析理论上可能的，在实践中可以实现的，并试图缩小两者之间的差距，使用软件模拟和原型实现进行实验和验证。

Today's critical applications, including genome analysis, climate simulations, drug discovery, space observation, and numerical simulations in computational chemistry and high-energy physics, are all data intensive in nature. Storage performance bottlenecks are major threats limiting the performance and scalability of data intensive applications. The goal of this project is to develop a general framework for self-optimizing parallel storage systems that can alleviate storage performance bottlenecks, and thus can have a considerable impact on society by accelerating the innovation process in a multitude of domains of science. The results will also advance the state of knowledge in storage systems by benefiting a wide range of parallel storage systems including disk arrays, key-value stores, and parallel/distributed file systems. Broader impacts include mentoring and training K-12 students through summer camps and promoting involvement of underrepresented students in science and engineering.This research develops novel, theoretically grounded, and experimentally validated methods for online detection and automatic elimination of disk I/O bottlenecks. Specific goals include the development of: (i) new online methods for continuously monitoring disk I/O requests and analyzing them efficiently, guided by data stream mining and social network analysis theory, and (ii) automatically-triggered self-optimization techniques guided by bin packing, graph coloring, and network flow theory, which can carefully plan an adaptive data layout to improve disk I/O performance. Experimentation and validation using software simulation and prototype implementation are performed by analyzing what is theoretically possible, what can be achieved in practice, and trying to close the gap between the two.