CCRI: ENS: Collaborative Research: Open Computer System Usage Repository and Analytics Engine

在科学和工程研究中，大规模、集中管理的计算集群或“超级计算机”在实现计算机辅助药物发现、汽车和喷气发动机的高强度材料设计以及疾病媒介分析等方面的资源密集型模拟、分析和可视化方面发挥了重要作用。这样的集群是由数百到数千台计算机服务器组成的复杂系统，这些服务器之间具有快速的网络连接，各种数据存储资源，以及与来自不同领域的数百名其他研究人员共享的高度优化的科学软件。因此，这种系统的总体可靠性取决于这些高度互连的各个元件的可靠性以及级联故障的特征。虽然计算机系统研究人员和实践者一直处于设计和部署可靠计算集群系统的前沿，但由于缺乏来自当前运行的超级计算机的公开可用、真实世界的故障数据，这项任务一直受到阻碍。以往的做法在很大程度上涉及单调乏味的手工收集和整理用于具体分析的小组数据。该项目将建立无缝的自动化管道，用于从普渡大学和德克萨斯大学奥斯汀分校这两个组织的大型计算集群中获取、处理和管理连续、详细的系统使用、监控和故障数据。这些数据将通过一个可公开访问的门户网站传播，并辅之以一套现场分析能力，这些能力将支持和促进可靠计算系统的研究。数据采集管道和分析软件将是开源的，旨在便于联合、扩展和采用由其他组织运营的集群系统。集群计算系统是时间敏感、计算密集型研究的关键资源，如病毒结构建模和药物发现，并一直处于应对全球流行病的努力的前沿。意外的系统停机时间和缺乏对研究人员计算故障的可操作反馈都可能对研究的及时性和效率产生不利影响。该项目将使这些系统的从业人员和管理员能够制定以数据为后盾的最佳做法，以确保其集群的高可用性和利用率。由此产生的大型公共数据存储库由来自具有不同工作负载的群集的数据组成，涵盖传统高性能计算、现代基于加速器的计算(例如图形处理单元(GPU))和云式应用程序，这将使系统研究团体能够基于实际系统数据考虑前瞻性研究问题。该项目将培训一批学生进行现场生产系统的数据分析，这将为他们提供独特的学习体验，与各种利益相关者接触。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

In science and engineering research, large-scale, centrally managed computing clusters or “supercomputers” have been instrumental in enabling the kinds of resource-intensive simulations, analyses, and visualizations that have been used in computer-aided drug discovery, high strength materials design for cars and jet engines, and disease vector analysis to name a few. Such clusters are complex systems comprised of several hundred to thousand computer servers with fast network connections between them, various data storage resources, and highly optimized scientific software being shared with several hundred other researchers from diverse domains. Consequently, the overall dependability of such systems relies on the dependability of these individual highly interconnected elements as well as the characteristics of cascading failures. While computer systems researchers and practitioners have been at the forefront of designing and deploying dependable computing cluster systems, this task has been hampered by the lack of publicly available, real-world failure data from supercomputers currently in operation. Prior practice has largely involved tedious, manual collection and curation of small sets of data for use in specific analyses. This project will establish seamless, automated pipelines for acquiring, processing, and curating continuous, detailed system usage, monitoring, and failure data from large computing clusters at two organizations, Purdue University and the University of Texas at Austin. This data will be disseminated through a publicly accessible portal and complemented by a suite of in-situ analytics capabilities that will support and spur research in dependable computing systems. The data acquisition pipeline and analytics software will be made open-source and designed for ease of federation, extension, and adoption to cluster systems operated by other organizations.Cluster computing systems are a key resource in time-sensitive, computationally intensive research such as virus structure modeling and drug discovery and have been at the forefront of efforts to tackle global pandemics. Both unanticipated system down-times and lack of actionable feedback to researchers on computational failures can have adverse effects on research timeliness and efficiency. This project will allow the practitioners and administrators of these systems to develop data-backed best practices for ensuring high availability and utilization for their clusters. The resulting large, public data repository consisting of data from clusters with diverse workloads spanning traditional high-performance computing, modern accelerator-based computing (for example on graphics processing units (GPUs)), and cloud-style applications will allow the systems research community to consider forward-looking research questions based on real system data. The project will train a cadre of students in data analysis on live production systems and this will provide them with a unique learning experience, interfacing with a variety of stakeholders.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.