Elements: Software: Distributed Workflows for Cyberexperimentation

计算系统、网络和安全领域的工业和学术研究无法证明和验证研究成果，因为研究过程包括在分布式、大规模、网络化试验台上进行的复杂但主要是实验。该项目将通过在试验台上实现严格和可重复的实验来解决这一问题，方法是将实验定义为可执行的工作流，然后可以重复运行和管理。该项目将开发和部署一个名为Elie的软件框架。Elie由实验表示能力和几个支持服务组成，这些服务使实验工作流健壮、容错、易于共享和可重用。具体地说，Elie包括：(1)一种新的实验表示，称为DOW，它编码实验行为和分配约束，而不仅仅是实验拓扑。(2)捕获工具，捕获研究人员在试验台上的手动操作并促进脚本构建，以及露水生成(3)试验床集成的版本控制，(4)实验编排机制，(5)轻量级故障检测，(6)允许在整个实验生命周期中进行实验管理的图形用户界面。Elie的技术可以被研究人员作为完整的环境使用，也可以作为个人工具使用。总的来说，这些技术在整个实验生命周期中提供了基本的试验台支持，包括设计、运行、分析、监控、存储和共享阶段。Elie的所有技术都是可扩展的，并且是开源的，允许研究社区为它们的开发做出贡献，并根据他们的需求进行定制。该项目与以前的工作根本不同，其目标是与当前的试验台实验方法(手动和脚本试验)共存，并适用于广泛的试验台(通过其翻译和生成工具)。该项目的成果将通过将乏味、重复和对细节敏感的任务转移到试验台来显著提高试验台实验的简易性。这将缩短实验时间，提高结果的质量和可靠性。这项工作将进一步使实验更加健壮，以最少的用户努力实现实验的轻松共享和重复使用，并促进重复性和再现性。总体而言，试验台实验的这种进步将使使用试验台的领域能够进行垂直开发(建立在他人工作的基础上)，如分布式系统、网络和网络安全。共享和再利用还将提高这些领域的科学研究质量，因为它能够在其他研究人员共享的人工制品的基础上创建更大、更复杂的实验。该项目团队还将与其他致力于提高试验台实验严谨性的研究人员进行广泛的用户拓展和建立合作，目标是为ELI建立一个强大的用户和开发人员社区。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Industry and academic research in computing systems, networking and security suffers from the inability to demonstrate and validate research outcomes, because the process of research consists of complex but largely experiments on distributed, large-scale, networked test-beds. This project will address this issue by enabling rigorous and repeatable experimentation on test-beds by enabling experiments to be defined as executable workflows, that can then be repeatably run, as well as managed. This project will develop and deploy a software framework called Elie. Elie consists of an experiment representation capability, along with several supporting services that make experimentation workflows robust, fault tolerant, easily sharable and reusable. Specifically, Elie consists of: (1) A new experiment representation, called DEW, which encodes experiment behavior and allocation constraints instead of just experiment topology. (2) Capture tools, which capture a researcher's manual actions on the testbed and facilitate script-building, and DEW generation (3) Testbed-integrated version control, (4) An experiment orchestration mechanism, (5) Lightweight failure detection, (6) A GUI, which allows for experiment management throughout an experiment lifecycle. Elie's technologies can be used by researchers as a full environment or as individual tools. Collectively these technologies provide basic testbed support throughout an experiment lifecycle including the design, running, analysis, monitoring, storage and sharing stages. All of Elie's technologies are extensible, and open-sourced, to allow the research community to contribute to their development and customize them to their needs. This project fundamentally differs from the prior work in its goal to co-exist with current testbed experimentation approaches (manual and scripted experimentation) and to be applicable to a broad range of testbeds (via its translation and generation tools). Outcomes of this project will significantly improve ease of testbed experimentation, by offloading tedious, repetitive and detail-sensitive tasks to testbeds. This will shorten experiment duration and improve quality and reliability of results. This work will further make experiments more robust, enable easy sharing and reuse of experiments with minimal user effort, and facilitate repeatability and reproducibility. Overall, such advances in testbed experimentation will enable vertical development (building upon the work of others) in fields, which use testbeds, such as distributed systems, networking and cybersecurity. Sharing and reuse will also improve quality of scientific research in these fields, by enabling creation of larger and more complex experiments built on the shared artifacts of other researchers. The project team will also conduct extensive user outreach and building collaborations with other researchers that work on improving rigor of testbed experimentation, with the goal of building a strong user and developer community for Elie.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.