SDCI NMI Improvement: Maintenance of the Rocks Cluster Toolkit and Enhancements for Scalable, Reliable, and Resilient Clustered Data Storage

如今，越来越多的科学家需要将可靠、可扩展、大规模和高性能的存储紧密集成到他们的计算/分析工作流中。这些研究人员经常要求他们的数据不仅在远程计算集群上可用，而且还在他们专门的实验室内设备、工作站和显示器上可用。不幸的是，许多公司受到了阻碍，因为他们没有基本的存储容量和带宽，但他们的实验室需要这样的基本存储容量和带宽，这通常只有在专门的数据中心才有。例如，生物研究领域的领先者正在通过开发和使用高场强电子显微镜等仪器，使他们的科学向数据密集型转变。高场强电子显微镜每周产生数兆字节的数据。尽管目前专家可以构建专门的并行存储系统，以满足对群集进行计算密集型分析的容量和吞吐量需求，但无论是初始部署还是持续操作，管理工作量都是巨大的。对于许多科学家来说，他们的需求正在迅速进入数据密集型领域，但由于现有解决方案的复杂性或成本(或两者兼而有之)，他们对功能强大且可靠的存储的访问受到限制。十年前，在计算集群中也存在同样的限制，Rock集群工具包已经成功地解决了这一问题。在此奖项中，成熟且广泛使用的Rock群集软件工具包将得到扩展，不仅包括针对计算群集的持续生产支持和工程增强，还将逐步解决与群集存储资源调配、监控和事件生成直接相关的一系列问题。特别是，该奖项的影响将是使当前计算群集部署的简单性还包括(1)网络连接文件服务器群和(2)通过称为Rolls的标准Rock扩展机制的专用并行高性能存储群集。此外，还将开始开发专门针对存储子系统的监控体系结构，以包括Lustre并行和基于NFS的服务器场使用的每磁盘指标、文件系统指标和聚合网络利用率。调查人员将开始设计和开发能够将文件服务器利用率与客户端和远程工作站上运行的作业相关联的机制。

Today an increasing number of scientists need reliable, extensible, large-scale and high-performance storage to be tightly integrated into their computing/analysis workflows. These researchers often require their data to be available not only on remote computational clusters but also on their specialized, in-laboratory, equipment, workstations and displays. Unfortunately, many are being hindered because they do not have, but need in their labs, the fundamental storage capacity and bandwidth this is often available only in specialized data centers. For example, leaders in biological research are revolutionizing their science to be data intensive by developing and using instruments like high field-strength electron microscopes that generate terabytes of data weekly. While the specialized parallel storage systems can be built today by experts to meet both the capacity and throughput needs of a computationally-intensive analysis on clusters, the level of administrative effort is enormous for both initial deployment and ongoing operation. For many scientists, their needs are rapidly entering the realm of data intensive but their access to capable and reliable storage is limited either because of the complexity or expense (or both) of existing solutions. This same limit, which the Rocks cluster toolkit has successfully addressed, existed in computational clusters a decade ago. In this award, the established and widely-used Rocks clustering software toolkit will be expanded to include not only ongoing production support and engineering enhancements for computational clusters but also to progressively address a litany of issues directly related to clustered storage provisioning, monitoring, and event generation. In particular, the impact of this award will be to bring the the current simplicity of compute cluster deployment to also include (1) farms of network-attached file servers and (2) dedicated parallel, high-performance storage clusters through the standard Rocks extension mechanism called Rolls. In addtion, the development of a monitoring architecture targeted specifically at storage subsystems to include per-disk metrics, file-system metrics, and aggregated network utilization for both Lustre Parallel and NFS-based server farms use will be started. The investigators will start the design and development of mechanisms that will enable correlation of file server utilization with jobs running on clients and remote workstations.