Automatic Reliability Testing For Cluster Management Controllers
Automatic Reliability Testing For Cluster Management Controllers
复制标题
DOI：
--
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
Xudong Sun;Wenqing Luo;Jiawei Tyler Gu;Aishwarya Ganesan;Ramnatthan Alagappan;Michael Gasch;Lalith Suresh;Tianyin Xu
中科院分区：
文献类型：
--
作者：
Xudong Sun;Wenqing Luo;Jiawei Tyler Gu;Aishwarya Ganesan;Ramnatthan Alagappan;Michael Gasch;Lalith Suresh;Tianyin Xu
像博格、Omega和Kubernetes这样的现代集群管理器依赖于状态协调原则来实现高度的弹性和可扩展性。在这些系统中，所有的集群管理逻辑都嵌入在一个松散耦合的微服务集合中，称为控制器。每个控制器独立地观察当前集群状态并发出校正动作以将集群收敛到期望状态。然而，整个系统复杂的分布式特性使得很难构建可靠和正确的控制器-我们发现控制器面临着无数的可靠性问题，这些问题会导致严重的后果，如数据丢失，安全漏洞和资源泄漏。我们介绍了Sieve，这是第一个用于集群管理控制器的自动可靠性测试工具。Sieve通过系统地和广泛地干扰控制器对当前集群状态的视图，以其期望容忍的方式，将控制器驱动到其潜在的错误角落。然后，它比较了集群状态的演化与无扰动，以检测安全性和活性问题。Sieve的设计是由状态协调系统中的一个基本机会提供动力的-这些系统基于控制器和集群状态之间以状态为中心的接口;这些接口是高度透明的，从而实现全自动可靠性测试。到目前为止，Sieve已经在10个流行的控制器中有效地发现了46个严重的安全性和活性错误（35个确认和22个修复），误报率为3.5%。
Modern cluster managers like Borg, Omega and Kubernetes rely on the state-reconciliation principle to be highly resilient and extensible. In these systems, all cluster-management logic is embedded in a loosely coupled collection of microservices called controllers . Each controller independently observes the current cluster state and issues corrective actions to converge the cluster to a desired state. However, the complex distributed nature of the overall system makes it hard to build reliable and correct controllers – we find that controllers face myriad reliability issues that lead to severe consequences like data loss, security vulnerabilities, and resource leaks. We present Sieve, the first automatic reliability-testing tool for cluster-management controllers. Sieve drives controllers to their potentially buggy corners by systematically and extensively perturbing the controller’s view of the current cluster state in ways it is expected to tolerate. It then compares the cluster state’s evolution with and without perturbations to detect safety and liveness issues. Sieve’s design is powered by a fundamental opportunity in state-reconciliation systems – these systems are based on state-centric interfaces between the controllers and the cluster state; such interfaces are highly transparent and thereby enable fully-automated reliability testing. To date, Sieve has efficiently found 46 serious safety and liveness bugs (35 confirmed and 22 fixed) in ten popular controllers with a low false-positive rate of 3.5%.