Fail through the Cracks: Cross-System Interaction Failures in Modern Cloud Systems
Fail through the Cracks: Cross-System Interaction Failures in Modern Cloud Systems
复制标题
DOI：
10.1145/3552326.3587448
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Lilia Tang;Chaitanya Bhandari;Yongle Zhang;Anna Karanika;Shuyang Ji;Indranil Gupta;Tianyi Xu
中科院分区：
文献类型：
--
作者：
Lilia Tang;Chaitanya Bhandari;Yongle Zhang;Anna Karanika;Shuyang Ji;Indranil Gupta;Tianyi Xu
现代云系统是独立和交互（子）系统的编排，每个系统专门用于重要服务（例如，数据处理、存储、资源管理等）。因此，云系统的可靠性不仅受到每个系统的可靠性的影响，而且还受到这些系统之间的相互作用的影响。我们观察到，最近云系统的许多生产事件都是通过跨系统边界的交互来表现的。然而，有一个缺乏系统的理解，这种新兴的故障模式，我们称之为跨系统交互故障（或CSI故障）。这阻碍了更好的设计、集成实践和新工具的开发。在本文中，我们讨论了跨系统交互失败的基础上分析（1）11 CSI故障引起的云事件的谷歌，Azure和AWS，和（2）120 CSI故障的情况下，7个广泛的共同部署的开源系统。我们专注于理解相互作用的系统之间的差异作为CSI故障的根本原因-CSI故障不能通过孤立地分析一个系统来理解。本文提请注意这种新兴的故障模式，提供了一个全面的理解CSI故障模式，并讨论了潜在的缓解方法。我们提倡跨系统测试和验证，并通过交叉测试Spark-Hive数据平面和暴露15个新的差异来展示其潜力。
Modern cloud systems are orchestrations of independent and interacting (sub-)systems, each specializing in important services (e.g., data processing, storage, resource management, etc.). Hence, cloud system reliability is affected not only by the reliability of each individual system, but also by the interplay between these systems. We observe that many recent production incidents of cloud systems are manifested through interactions across the system boundaries. However, there is a lack of systematic understanding of this emerging mode of failures, which we term as cross-system interaction failures (or CSI failures). This hinders the development of better design, integration practices, and new tooling. In this paper, we discuss cross-system interaction failures based on analyses of (1) 11 CSI-failure-induced cloud incidents of Google, Azure, and AWS, and (2) 120 CSI failure cases of seven widely co-deployed open-source systems. We focus on understanding discrepancies between interacting systems as the root causes of CSI failures---CSI failures cannot be understood by analyzing one single system in isolation. This paper draws attention to this emerging failure mode, provides a comprehensive understanding of CSI failure patterns, and discusses potential approaches for mitigation. We advocate for cross-system testing and verification and demonstrate its potential by cross-testing the Spark-Hive data plane and exposing 15 new discrepancies.