CSR: Small: Automatic Storage and Network Contention Management for Large-scale High-performance Computing Systems

从资源勘探到下一代消费电子产品的设计，高性能计算对科学、工业和环境至关重要。这些高性能计算机系统是最复杂和最昂贵的计算机系统之一，要求以最有效的方式使用它们的资源。许多利用高性能计算的应用程序都是数据密集型的，存储系统性能是系统性能的一个关键方面。然而，存储系统对于存储客户端之间争夺有限的带宽和磁盘访问而引起的争用非常敏感。这是共享存储系统的一个重要问题。该项目为大规模高性能存储提供了一种存储争用自动缓解和减少系统（ASCAR），以提高带宽利用率和资源分配的公平性。ASCAR使用机器学习方法结合几种启发式方法来发现最适合的控制策略。它是一个高度可扩展的全自动存储争用和拥塞管理系统，可以提高旧系统和新系统的效率，而无需更改服务器硬件/软件或现有应用程序。ASCAR使用基于规则的算法从客户端调节I/O流量。它采用无共享设计，并且不需要客户机之间或与中央协调器之间的运行时协调，因为运行时协调缓慢且不可扩展。ASCAR的有效性依赖于交通控制的质量。研究小组设计了一种原型算法“无共享规则生成器”（SHARP），通过系统地探索可能设计的解空间，以无监督的方式生成规则。SHARP从一个初始规则开始，使用类似于随机重新开始爬坡的启发式方法来找到最优参数，而不需要穷举搜索。ASCAR监视系统上运行的工作负载，并使用几种启发式方法来挑选最合适的规则。很明显，计算机系统正变得越来越复杂，人类主导的基于经验的系统优化方法并不是实现这些现代、复杂、高性能计算系统全部潜力的最有效方法。这项研究为系统研究带来了机器学习、人工智能和大数据方法，并可能为广泛的系统带来非常低成本的I/O性能提升。

High performance computing is essential to science, industry, and the environment, from resource exploration to the design of the next generation of consumer electronics. These high performance computer systems are among the most complex and expensive computer systems and require that their resources be used in the most efficient manner. Many of the applications that utilize high performance computing are data-intensive, and storage system performance is a crucial aspect of system performance. However, storage systems are notoriously sensitive to contention caused by competition among storage clients for limited bandwidth and disk access. This is a significant problem for shared storage systems. This project provides an automatic storage contention alleviation and reduction system (ASCAR) for large-scale high-performance storage to increase bandwidth utilization and fairness of resource allocation. ASCAR uses machine learning methods combined with several heuristics to discover the fittest control strategy. It is a highly scalable and fully automatic storage contention and congestion management system, which can improve the efficiency of both legacy and new systems, with no need to change either server hardware/software or existing applications. ASCAR regulates I/O traffic from the client side using a rule based algorithm. It employs a shared-nothing design and requires no runtime coordination between clients or with a central coordinator whatsoever, because runtime coordination is slow and unscalable. The effectiveness of ASCAR relies on the quality of traffic control. The research team has designed a prototype algorithm, the SHAred-nothing Rule Producer (SHARP), which produces rules in an unsupervised manner by systematically exploring the solution space of possible designs. Starting from one initial rule, SHARP uses heuristics similar to random-restart hill climbing to find the optimal parameters without the need for an exhaustive search. ASCAR monitors the workloads running on the system and uses several heuristics to pick up the fittest rules. It is clear that computer systems are getting ever more sophisticated, and human-lead empirical-based approach towards system optimization is not the most efficient way to realize the full potential of these modern, complex, high performance computing systems. This research brings machine learning, artificial intelligence, and big data methods to systems research and could lead to a very low cost I/O performance increase for a wide range of systems.