CNS Core: Small: Testing and detecting software upgrade failures in data-intensive distributed systems

CNS 核心：小型：测试和检测数据密集型分布式系统中的软件升级故障

• 批准号: 2300562
• 负责人: Yongle Zhang
• 金额: $ 60万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2300562&HistoricalAwards=false
• 关键词: CNS; Core; Small; Testing; detecting

In the current big data era, Internet services are often built on top of data-intensive distributed systems. Such distributed systems have to go through frequent software upgrade as vendors need to add new features, improve performance, and deploy patches. With the rise of continuous deployment in the industry, the frequency of distributed system software upgrade could reach thousands of deployments in a single day in a major Internet company. Unfortunately, distributed systems could experience upgrade failures – failures happen during software upgrade. These failures often have large-scale impact as upgrade is performed on the entire system. They are typically mitigated in the production environment with canary deployment, which slowly rollout updates from a small scale to the entire cluster and downgrade if a failure is encountered. However, canary deployment easily takes hours and creates a dilemma between safe and fast upgrade. In addition, many upgrade failures have persistent impact and cannot be easily resolved by downgrading. Despite the severe consequence of upgrade failures and challenges faced by production mitigation techniques, there are no existing testing and program analysis techniques that focus on testing and analyzing the distributed system upgrade procedure systematically. This work proposes to develop such techniques optimized to detect upgrade failures in early stages through exploring the effectiveness of unique properties of the distributed system software upgrade procedure. Data-intensive distributed systems deployed in public or private clouds are nowadays a cornerstone of many critical computing systems. The proposed techniques should dramatically improve the reliability of data-intensive distributed systems during upgrade and, consequently, reduce service disruptions and improve the availability of cloud systems. In addition, improved reliability of the upgrade procedure will lead to more timely feedbacks about new features in production, which is critical for developers’ productivity and the quality of the resulting software.In this project we plan to (1) implement differential testing between two standard distributed system upgrade procedures – full-stop upgrade and rolling upgrade, (2) explore utilizing source code difference between versions to design differential test oracles, feedback metrics, and input mutation strategies, that are specially tuned to trigger and detect upgrade failures, (3) design static program analysis guided by source code difference to detect data format incompatibilities between versions, and (4) validate the testing and detection techniques proposed through direct experimentation on real-world data-intensive distributed systems. The proposed fault localization and static analysis techniques will reduce the valuable time and effort that developers spend on root cause diagnosis, which is extremely challenging for bugs in distributed systems. All products of the project will be open sourced to ensure a widespread impact.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在当前的大数据时代，互联网服务往往建立在数据密集型分布式系统之上。这样的分布式系统必须经历频繁的软件升级，因为供应商需要添加新功能、提高性能和部署补丁。随着行业持续部署的兴起，在一家大型互联网公司，分布式系统软件升级的频率可以在一天内达到数千次部署。不幸的是，分布式系统可能会遇到升级故障--故障发生在软件升级过程中。当在整个系统上执行升级时，这些故障通常会产生大规模影响。在使用金丝雀部署的生产环境中，这些问题通常会得到缓解，金丝雀部署会缓慢地从小范围向整个集群推出更新，并在遇到故障时降级。然而，金丝雀部署很容易花费数小时，并在安全和快速升级之间制造两难境地。此外，许多升级失败具有持续性影响，不能通过降级轻松解决。尽管升级失败带来了严重的后果和生产缓解技术面临的挑战，但现有的测试和程序分析技术还没有集中于系统地测试和分析分布式系统升级过程。这项工作建议通过探索分布式系统软件升级过程独特属性的有效性来开发这种优化的技术，以便在早期阶段检测升级失败。部署在公共云或私有云中的数据密集型分布式系统如今是许多关键计算系统的基石。建议的技术应能在升级过程中显著提高数据密集型分布式系统的可靠性，从而减少服务中断并提高云系统的可用性。此外，升级过程的可靠性的提高将导致对生产中新功能的更及时的反馈，这对开发人员的工作效率和最终软件的质量至关重要。在本项目中，我们计划(1)在两个标准的分布式系统升级过程--完全停止升级和滚动升级--之间进行差异测试，(2)探索利用版本之间的源代码差异来设计差异测试预言、反馈度量和输入突变策略，这些策略经过专门调整来触发和检测升级失败，(3)设计以源代码差异为导向的静态程序分析，以检测版本之间的数据格式不兼容，以及(4)通过在真实数据密集型分布式系统上的直接实验来验证所提出的测试和检测技术。提出的故障定位和静态分析技术将减少开发人员在根本原因诊断上花费的宝贵时间和精力，而根本原因诊断对于分布式系统中的错误来说是极具挑战性的。该项目的所有产品都将是开源的，以确保广泛的影响。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。