SDCI NMI Improvement: Maintenance of the Rocks Cluster Toolkit and Enhancements for Scalable, Reliable, and Resilient Clustered Data Storage

如今，越来越多的科学家需要可靠、可扩展、大规模和高性能的存储设备，以紧密集成到他们的计算/分析工作流程中。这些研究人员经常要求他们的数据不仅可以在远程计算集群上使用，还可以在他们的专用实验室设备、工作站和显示器上使用。不幸的是，许多人受到阻碍，因为他们没有，但需要在他们的实验室，基本的存储容量和带宽，这往往是只有在专门的数据中心。例如，生物研究领域的领导者正在通过开发和使用高场强电子显微镜等每周产生TB数据的仪器，彻底改变他们的科学成为数据密集型。虽然专业的并行存储系统目前可以由专家构建，以满足对集群进行计算密集型分析的容量和吞吐量需求，但对于初始部署和持续操作而言，管理工作量是巨大的。对于许多科学家来说，他们的需求正在迅速进入数据密集型领域，但由于现有解决方案的复杂性或成本（或两者兼而有之），他们对功能强大且可靠的存储的访问受到限制。Rocks集群工具包已经成功解决了这个同样的限制，十年前就存在于计算集群中。 在此次授予中，已建立并广泛使用的Rocks集群软件工具包将得到扩展，不仅包括对计算集群的持续生产支持和工程增强，还将逐步解决与集群存储配置、监控和事件生成直接相关的一系列问题。 特别是，该奖项的影响将使当前计算集群部署的简单性也包括（1）网络连接文件服务器的农场和（2）通过标准的Rocks扩展机制（称为Rolls）的专用并行高性能存储集群。此外，还将开始开发专门针对存储子系统的监控体系结构，以包括每个磁盘指标、文件系统指标以及用于Lustre Parallel和基于NFS的服务器场的聚合网络利用率。调查人员将开始设计和开发机制，使文件服务器利用率与客户机和远程工作站上运行的作业相互关联。

Today an increasing number of scientists need reliable, extensible, large-scale and high-performance storage to be tightly integrated into their computing/analysis workflows. These researchers often require their data to be available not only on remote computational clusters but also on their specialized, in-laboratory, equipment, workstations and displays. Unfortunately, many are being hindered because they do not have, but need in their labs, the fundamental storage capacity and bandwidth this is often available only in specialized data centers. For example, leaders in biological research are revolutionizing their science to be data intensive by developing and using instruments like high field-strength electron microscopes that generate terabytes of data weekly. While the specialized parallel storage systems can be built today by experts to meet both the capacity and throughput needs of a computationally-intensive analysis on clusters, the level of administrative effort is enormous for both initial deployment and ongoing operation. For many scientists, their needs are rapidly entering the realm of data intensive but their access to capable and reliable storage is limited either because of the complexity or expense (or both) of existing solutions. This same limit, which the Rocks cluster toolkit has successfully addressed, existed in computational clusters a decade ago. In this award, the established and widely-used Rocks clustering software toolkit will be expanded to include not only ongoing production support and engineering enhancements for computational clusters but also to progressively address a litany of issues directly related to clustered storage provisioning, monitoring, and event generation. In particular, the impact of this award will be to bring the the current simplicity of compute cluster deployment to also include (1) farms of network-attached file servers and (2) dedicated parallel, high-performance storage clusters through the standard Rocks extension mechanism called Rolls. In addtion, the development of a monitoring architecture targeted specifically at storage subsystems to include per-disk metrics, file-system metrics, and aggregated network utilization for both Lustre Parallel and NFS-based server farms use will be started. The investigators will start the design and development of mechanisms that will enable correlation of file server utilization with jobs running on clients and remote workstations.