CNS Core: Small: Intelligent Fault Injection to Expose and Reproduce Production-Grade Bugs in Cloud Systems

生产分布式系统的故障代价高昂。尽管在测试分布式系统方面付出了广泛的努力，但即使在使用适当的输入对系统进行测试时，仍很难在测试中发现许多错误。这是因为这些错误是由生产环境中独特的故障事件触发的。已经提出了故障注入来模拟测试过程中的故障，目的是捕获此类错误。然而，现有的解决方案将目标系统视为黑匣子，只使用随机选择注入简单的故障。生产失败通常是由Bug引起的，这些Bug需要在谨慎的时间触发复杂的、特定于系统的故障。该项目采用整体方法来解决当前故障注入测试中的基本限制。该项目开发了特殊的编译器支持，以便能够以精细的粒度和精确的控制注入特定于系统的错误。为了有效地挖掘巨大的故障注入空间，暴露错误，本项目设计了新的故障注入决策算法和机器学习方法。一种新的自适应方法进一步分析生产执行轨迹，以快速再现离线环境中的故障诱导故障。生产分布式系统中的漏洞给社会造成了巨大的经济损失。该项目开发的新故障注入技术将有助于有效地捕获大型分布式系统中广泛的生产级错误，并提高云服务的可用性。这个项目将与开源社区的开发人员密切合作，以改进分布式系统的代码质量和测试实践。该项目开发的软件构件将是开源的，并可在https://github.com/OrderLab.获得项目成果，包括纸质出版物、技术报告和演示文稿，将可免费下载，并在项目完成后至少保存五年。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Failures of production distributed systems are costly. Despite extensive efforts on testing distributed systems, many bugs remain difficult to find in testing even when a system is tested with appropriate input. This is because these bugs are triggered by the unique faulty events in the production environment. Fault injection has been proposed to simulate faults during testing with the goal of catching such bugs. However, existing solutions treat the target systems as a black box and only inject simple faults using random choices. Production failures are often caused by bugs that require complex, system-specific faults at careful timing to trigger.This project takes a holistic approach to address the fundamental limitations in current fault injection testing. The project develops special compiler support to enable the injection of system-specific faults at a fine granularity with precise control. To efficiently explore the large fault injection space and expose bugs, this project designs new fault injection decision algorithms and machine learning methods. A new adaptive method further analyzes production execution traces to quickly reproduce fault-induced failures in offline environment. Bugs in production distributed systems have resulted in substantial financial losses to society. The new fault injection techniques developed in this project will help effectively catch a wide range of production-grade bugs in large distributed systems and improve the availability of cloud services. This project will closely engage with developers in the open-source community to improve the distributed systems code quality and testing practice.The software artifact this project develops will be open sourced and available at https://github.com/OrderLab. The project results, including paper publications, technical reports, and presentations will be made available for free download and be maintained for at least five years beyond the completion of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.