CNS Core: Small: Intelligent Fault Injection to Expose and Reproduce Production-Grade Bugs in Cloud Systems

CNS 核心：小型：智能故障注入以暴露和重现云系统中的生产级错误

• 批准号: 2149664
• 负责人: Peng Huang
• 金额: $ 50万
• 依托单位: Johns Hopkins University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-12-15 至 2023-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2149664&HistoricalAwards=false
• 关键词: CNS; Core; Small; Intelligent; Fault

Failures of production distributed systems are costly. Despite extensive efforts on testing distributed systems, many bugs remain difficult to find in testing even when a system is tested with appropriate input. This is because these bugs are triggered by the unique faulty events in the production environment. Fault injection has been proposed to simulate faults during testing with the goal of catching such bugs. However, existing solutions treat the target systems as a black box and only inject simple faults using random choices. Production failures are often caused by bugs that require complex, system-specific faults at careful timing to trigger.This project takes a holistic approach to address the fundamental limitations in current fault injection testing. The project develops special compiler support to enable the injection of system-specific faults at a fine granularity with precise control. To efficiently explore the large fault injection space and expose bugs, this project designs new fault injection decision algorithms and machine learning methods. A new adaptive method further analyzes production execution traces to quickly reproduce fault-induced failures in offline environment. Bugs in production distributed systems have resulted in substantial financial losses to society. The new fault injection techniques developed in this project will help effectively catch a wide range of production-grade bugs in large distributed systems and improve the availability of cloud services. This project will closely engage with developers in the open-source community to improve the distributed systems code quality and testing practice.The software artifact this project develops will be open sourced and available at https://github.com/OrderLab. The project results, including paper publications, technical reports, and presentations will be made available for free download and be maintained for at least five years beyond the completion of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

生产分布式系统的故障成本很高。尽管在测试分布式系统上进行了广泛的努力，但即使使用适当的输入进行了测试，许多错误仍然很难在测试中找到。这是因为这些错误是由生产环境中独特的故障事件触发的。已经提出了故障注入以模拟测试期间的故障，目的是捕获此类错误。 但是，现有解决方案将目标系统视为黑匣子，并且仅使用随机选择注入简单的故障。 生产故障通常是由于需要在谨慎时间触发时需要复杂的系统特定故障的错误引起的。该项目采用一种整体方法来解决当前故障注入测试中的基本局限性。 该项目开发了特殊的编译器支持，以使系统特定的故障以精确的控制良好。为了有效地探索大型故障注入空间并暴露了错误，该项目设计了新的故障注入决策算法和机器学习方法。一种新的自适应方法进一步分析了生产执行跟踪，以快速再现故障引起的离线环境中的故障​​。生产分布式系统中的错误已导致社会造成了巨大的财务损失。该项目中开发的新的故障注入技术将有助于有效地在大型分布式系统中捕获广泛的生产级错误，并提高云服务的可用性。该项目将与开源社区中的开发人员紧密互动，以改善分布式系统代码质量和测试实践。该项目开发的软件将是开源的，并在https://github.com/orderlab上获得。该项目的结果，包括纸质出版物，技术报告和演示文稿，将免费下载，并在项目完成之后至少维护五年。该奖项反映了NSF的法定任务，并认为值得通过基金会的知识分子优点和更广泛的审查标准通过评估来进行评估。