CCRI: ENS: Collaborative Research: Open Computer System Usage Repository and Analytics Engine

在科学和工程研究中，大规模、集中管理的计算集群或“超级计算机”在实现各种资源密集型模拟、分析和可视化方面发挥了重要作用，这些模拟、分析和可视化已用于计算机辅助药物发现、汽车和喷气发动机的高强度材料设计以及病媒分析等。这样的集群是由数百到数千台计算机服务器组成的复杂系统，这些服务器之间有快速的网络连接，各种数据存储资源，以及与来自不同领域的数百名其他研究人员共享的高度优化的科学软件。因此，此类系统的整体可靠性依赖于这些高度互连的单个元素的可靠性以及级联故障的特征。虽然计算机系统研究人员和从业人员一直处于设计和部署可靠的计算集群系统的前沿，但这项任务一直受到缺乏公开可用的、来自当前运行的超级计算机的真实故障数据的阻碍。先前的实践在很大程度上涉及繁琐的人工收集和管理用于特定分析的小数据集。该项目将建立无缝、自动化的管道，用于获取、处理和管理来自两个组织（普渡大学和德克萨斯大学奥斯汀分校）的大型计算集群的连续、详细的系统使用、监控和故障数据。这些数据将通过一个可公开访问的门户网站传播，并辅以一套现场分析能力，这将支持和促进可靠计算系统的研究。数据采集管道和分析软件将是开源的，并且设计为易于联合、扩展和采用由其他组织操作的集群系统。集群计算系统是时间敏感、计算密集型研究（如病毒结构建模和药物发现）的关键资源，并且一直处于应对全球流行病的前沿。意外的系统停机时间和对研究人员缺乏可操作的计算故障反馈都会对研究的及时性和效率产生不利影响。该项目将允许这些系统的从业者和管理员开发数据支持的最佳实践，以确保其集群的高可用性和利用率。由此产生的大型公共数据存储库由来自具有不同工作负载的集群的数据组成，这些数据跨越传统的高性能计算，现代基于加速器的计算（例如图形处理单元（gpu））和云风格的应用程序，将允许系统研究社区考虑基于真实系统数据的前瞻性研究问题。该项目将培训一批学生进行现场生产系统的数据分析，这将为他们提供独特的学习经验，与各种利益相关者进行交流。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In science and engineering research, large-scale, centrally managed computing clusters or “supercomputers” have been instrumental in enabling the kinds of resource-intensive simulations, analyses, and visualizations that have been used in computer-aided drug discovery, high strength materials design for cars and jet engines, and disease vector analysis to name a few. Such clusters are complex systems comprised of several hundred to thousand computer servers with fast network connections between them, various data storage resources, and highly optimized scientific software being shared with several hundred other researchers from diverse domains. Consequently, the overall dependability of such systems relies on the dependability of these individual highly interconnected elements as well as the characteristics of cascading failures. While computer systems researchers and practitioners have been at the forefront of designing and deploying dependable computing cluster systems, this task has been hampered by the lack of publicly available, real-world failure data from supercomputers currently in operation. Prior practice has largely involved tedious, manual collection and curation of small sets of data for use in specific analyses. This project will establish seamless, automated pipelines for acquiring, processing, and curating continuous, detailed system usage, monitoring, and failure data from large computing clusters at two organizations, Purdue University and the University of Texas at Austin. This data will be disseminated through a publicly accessible portal and complemented by a suite of in-situ analytics capabilities that will support and spur research in dependable computing systems. The data acquisition pipeline and analytics software will be made open-source and designed for ease of federation, extension, and adoption to cluster systems operated by other organizations.Cluster computing systems are a key resource in time-sensitive, computationally intensive research such as virus structure modeling and drug discovery and have been at the forefront of efforts to tackle global pandemics. Both unanticipated system down-times and lack of actionable feedback to researchers on computational failures can have adverse effects on research timeliness and efficiency. This project will allow the practitioners and administrators of these systems to develop data-backed best practices for ensuring high availability and utilization for their clusters. The resulting large, public data repository consisting of data from clusters with diverse workloads spanning traditional high-performance computing, modern accelerator-based computing (for example on graphics processing units (GPUs)), and cloud-style applications will allow the systems research community to consider forward-looking research questions based on real system data. The project will train a cadre of students in data analysis on live production systems and this will provide them with a unique learning experience, interfacing with a variety of stakeholders.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.