Collaborative Research: Elements: Simulation-driven Evaluation of Cyberinfrastructure Systems

现在，大多数科学突破和发现都是以尽可能方便、可靠和高效地对大量数据进行复杂处理为前提的。这需要高端互联的计算和存储资源，以及自动化处理这些资源的软件系统。在生产这种“网络基础设施”软件系统方面投入了大量精力。然而，开发和改进这些系统，使其尽可能高效，同时预测未来的网络基础设施机会和需求，是一项公开的挑战。这个项目改变了评估这些系统的方式，这样它们的能力就可以被明智地开发和发展。传统的评估方法是观察这些系统在实际硬件资源上的执行情况。虽然看起来很自然，但这种方法有许多缺点。相反，本项目侧重于模拟这些执行。仿真在改变网络基础设施系统的开发周期方面具有巨大的、尚未开发的潜力。具体来说，这个项目产生的软件元素可以很容易地集成到现有的和未来的系统中，以提供模拟能力。这些功能使开发人员能够将他们的系统进行压榨，并观察其在任意操作条件下的行为，包括超出当前硬件平台和科学应用的操作条件。简而言之，这些能力将使建立一个可靠的实验科学方法成为可能，用于开发网络基础设施系统，支持当前和未来对我们社会发展至关重要的科学努力。网络基础设施一直是密集研究和开发的对象，从而产生了一套丰富的可互操作的软件系统，用于支持科学。一个关键的挑战是开发能够有效执行应用程序工作负载的系统，同时预测未来的网络基础设施机会和需求。该项目旨在改变评估这些系统的方式，以便它们的能力可以基于可靠的、定量的实验科学方法来发展。传统的评估方法是使用成熟的软件堆栈在实际的网络基础设施部署上执行应用程序工作负载。不幸的是，这种方法有几个缺点：现实世界的实验是时间和劳动密集型的，并且它们受限于当前可用的硬件和软件配置。不受这些缺点影响的现实世界实验的另一种选择是模拟，也就是说，实现和使用对感兴趣的软件和硬件堆栈的功能和性能行为进行建模的软件工件。该项目使用模拟来改变网络基础设施系统作为其长期开发周期的一部分进行评估的方式。这是通过软件元素来实现的，用于增强具有模拟能力的生产网络基础设施系统，以便能够对这些系统进行任意执行场景的定量评估。创建这些场景只需要很少的劳动，并且可以准确地模拟执行，并且比现实世界中的执行速度快几个数量级。此外，模拟具有完美的可重复性和可观察性。虽然这种方法是通用的，但它的有效性将通过将其应用于许多生产系统（即工作流管理系统）来证明。该项目利用了多年来对SimGrid和扳手模拟框架的开发投入。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Most scientific breakthroughs and discoveries are now preconditioned on performing complex processing of vast amounts of data as conveniently, reliably, and efficiently as possible. This requires high-end interconnected compute and storage resources, as well as software systems to automate the processing on these resources. An enormous amount of effort has been invested in producing such "cyberinfrastructure" software systems. And yet, developing and evolving these systems so that they are as efficient as possible, while anticipating future cyberinfrastructure opportunities and needs, is an open challenge. This project transforms the way in which these systems are evaluated, so that their capabilities can be developed and evolved judiciously. The traditional evaluation approach is to observe executions of these systems on real-world hardware resources. Although seemingly natural, this approach suffers from many shortcomings. Instead, this project focuses on simulating these executions. Simulation has tremendous, and untapped, potential for transforming the development cycle of cyberinfrastructure systems. Specifically, this project produces software elements that can be easily integrated into existing and future systems to afford them with simulation capabilities. These capabilities make it possible for developers to put their systems through the wringer and observe their behaviors for arbitrary operating conditions, including ones that go beyond current hardware platforms and scientific applications. Simply put, these capabilities will make it possible to establish a solid experimental science approach for the development of cyberinfrastructure systems that support current and future scientific endeavors that are critical to the development of our society.The cyberinfrastructure has been the object of intensive research and development, resulting in a rich set of interoperable software systems that are used to support science. A key challenge is the development of systems that can execute application workloads efficiently, while anticipating future cyberinfrastructure opportunities and needs. This project aims to transform the way in which these systems are evaluated, so that their capabilities can be evolved based on a sound, quantitative experimental science approach. The traditional evaluation approach is to use full-fledged software stacks to execute application workloads on actual cyberinfrastructure deployments. Unfortunately, this approach suffers from several shortcomings: real-world experiments are time- and labor-intensive, and they are limited to currently available hardware and software configurations. An alternative to real-world experiments that does not suffer from these shortcomings is simulation, i.e., the implementation and use of a software artifact that models the functional and performance behaviors of software and hardware stacks of interest. This project uses simulation to transform the way in which cyberinfrastructure systems are evaluated as part of their long-term development cycles. This is achieved via software elements for enhancing production cyberinfrastructure systems with simulation capabilities so as to enable quantitative evaluation of these systems for arbitrary execution scenarios. Creating these scenarios requires little labor, and executions can be simulated accurately and orders of magnitude faster than their real-world counterparts. Furthermore, simulations are perfectly reproducible and observable. While this approach is general, its effectiveness will be demonstrated by applying it to a number of production systems, namely, workflow management systems. This project capitalizes on the years of development invested in the SimGrid and WRENCH simulation frameworks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.