CNS Core: Small: Testing and detecting software upgrade failures in data-intensive distributed systems

在当前的大数据时代，互联网服务往往建立在数据密集型分布式系统之上。这样的分布式系统必须经历频繁的软件升级，因为供应商需要添加新功能、提高性能和部署补丁。随着持续部署在业界的兴起，在一家大型互联网公司，分布式系统软件升级的频率可以达到一天数千次的部署。不幸的是，分布式系统可能会经历升级失败——在软件升级期间发生的失败。当对整个系统执行升级时，这些故障通常会产生大规模的影响。它们通常在生产环境中通过金丝雀部署得到缓解，金丝雀部署将从小规模的更新缓慢地推出到整个集群，并在遇到故障时降级。然而，金丝雀部署很容易花费数小时，并且在安全和快速升级之间造成两难。此外，许多升级失败具有持久的影响，无法通过降级轻松解决。尽管升级失败的严重后果和生产缓解技术面临的挑战，但目前还没有针对分布式系统升级过程进行系统测试和分析的测试和程序分析技术。本工作建议通过探索分布式系统软件升级过程的独特属性的有效性，开发这种优化的技术，以便在早期阶段检测升级失败。部署在公共云或私有云中的数据密集型分布式系统如今是许多关键计算系统的基石。所提出的技术将在升级过程中显著提高数据密集型分布式系统的可靠性，从而减少服务中断并提高云系统的可用性。此外，升级过程的可靠性的提高将导致对生产中的新特性的更及时的反馈，这对开发人员的生产力和最终软件的质量至关重要。在这个项目中，我们计划(1)实现两个标准的分布式系统升级过程之间的差异测试——完全停止升级和滚动升级；(2)探索利用版本之间的源代码差异来设计差异测试oracle、反馈指标和输入突变策略，这些策略专门用于触发和检测升级失败。(3)设计以源代码差异为指导的静态程序分析，检测版本之间的数据格式不兼容；(4)通过在真实数据密集型分布式系统上的直接实验，验证所提出的测试和检测技术。所提出的故障定位和静态分析技术将减少开发人员花费在根本原因诊断上的宝贵时间和精力，这对于分布式系统中的错误是极具挑战性的。该项目的所有产品都将开源，以确保广泛的影响。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In the current big data era, Internet services are often built on top of data-intensive distributed systems. Such distributed systems have to go through frequent software upgrade as vendors need to add new features, improve performance, and deploy patches. With the rise of continuous deployment in the industry, the frequency of distributed system software upgrade could reach thousands of deployments in a single day in a major Internet company. Unfortunately, distributed systems could experience upgrade failures – failures happen during software upgrade. These failures often have large-scale impact as upgrade is performed on the entire system. They are typically mitigated in the production environment with canary deployment, which slowly rollout updates from a small scale to the entire cluster and downgrade if a failure is encountered. However, canary deployment easily takes hours and creates a dilemma between safe and fast upgrade. In addition, many upgrade failures have persistent impact and cannot be easily resolved by downgrading. Despite the severe consequence of upgrade failures and challenges faced by production mitigation techniques, there are no existing testing and program analysis techniques that focus on testing and analyzing the distributed system upgrade procedure systematically. This work proposes to develop such techniques optimized to detect upgrade failures in early stages through exploring the effectiveness of unique properties of the distributed system software upgrade procedure. Data-intensive distributed systems deployed in public or private clouds are nowadays a cornerstone of many critical computing systems. The proposed techniques should dramatically improve the reliability of data-intensive distributed systems during upgrade and, consequently, reduce service disruptions and improve the availability of cloud systems. In addition, improved reliability of the upgrade procedure will lead to more timely feedbacks about new features in production, which is critical for developers’ productivity and the quality of the resulting software.In this project we plan to (1) implement differential testing between two standard distributed system upgrade procedures – full-stop upgrade and rolling upgrade, (2) explore utilizing source code difference between versions to design differential test oracles, feedback metrics, and input mutation strategies, that are specially tuned to trigger and detect upgrade failures, (3) design static program analysis guided by source code difference to detect data format incompatibilities between versions, and (4) validate the testing and detection techniques proposed through direct experimentation on real-world data-intensive distributed systems. The proposed fault localization and static analysis techniques will reduce the valuable time and effort that developers spend on root cause diagnosis, which is extremely challenging for bugs in distributed systems. All products of the project will be open sourced to ensure a widespread impact.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.