Kill Two Birds with One Stone: Auto-tuning RocksDB for High Bandwidth and Low Latency
Kill Two Birds with One Stone: Auto-tuning RocksDB for High Bandwidth and Low Latency
复制标题
DOI：
10.1109/icdcs47774.2020.00113
复制
发表时间：
2020-11
期刊：
影响因子：
--
通讯作者：
Yichen Jia;Feng Chen
中科院分区：
文献类型：
--
作者：
Yichen Jia;Feng Chen
基于日志结构化合并（Log-Structured Merge，LSM）树的键值存储被广泛部署在数据中心中。由于其复杂的内部结构，适当地配置现代键值数据存储系统，它可以有50多个参数与各种硬件和系统设置，是一项极具挑战性的任务。目前，该行业仍然严重依赖传统的、基于经验的手动调优方法来进行性能调优。许多人只是采用默认设置开箱即用，没有任何更改。自动调优，作为一种自适应的解决方案，因此非常有吸引力，以实现最佳或接近最佳的性能在现实世界的deployment.In本文中，我们定量研究和比较五种优化方法的自动调优性能的LSM-树的键值存储。为了评估自动调优过程，我们已经进行了一系列详尽的实验RocksDB，一个代表性的LSM树数据存储。我们在6个月内收集了超过12，000个实验记录，在不同的硬件设置上有大约2，000个6参数的软件配置。通过对5种典型算法在吞吐量、第99百分位尾延迟、收敛时间、实时系统吞吐量、迭代过程等方面的比较，发现多目标优化（MOO）方法能够在多个目标之间取得较好的平衡，满足了键值服务的独特需求。用户能够提供的服务质量（QoS）要求越具体，这些算法能够实现的性能就越好。我们还发现，并发线程数和写缓冲区大小是决定不同硬件和工作负载的吞吐量和第99百分位数尾部延迟的两个最具影响力的参数。最后，我们提供了系统级的自动调整结果的解释，并讨论了相关的系统设计师和从业人员的影响。我们希望这项工作将铺平道路走向实用，高速的自动调优解决方案的键值数据存储系统。
Log-Structured Merge (LSM) tree based key-value stores are widely deployed in data centers. Due to its complex internal structures, appropriately configuring a modern key-value data store system, which can have more than 50 parameters with various hardware and system settings, is a highly challenging task. Currently, the industry still heavily relies on a traditional, experience-based, hand-tuning approach for performance tuning. Many simply adopt the default setting out of the box with no changes. Auto-tuning, as a self-adaptive solution, is thus highly appealing for achieving optimal or near-optimal performance in real-world deployment.In this paper, we quantitatively study and compare five optimization methods for auto-tuning the performance of LSM-tree based key-value stores. In order to evaluate the auto-tuning processes, we have conducted an exhaustive set of experiments over RocksDB, a representative LSM-tree data store. We have collected over 12,000 experimental records in 6 months, with about 2,000 software configurations of 6 parameters on different hardware setups. We have compared five representative algorithms, in terms of throughput, the 99th percentile tail latency, convergence time, real-time system throughput, and the iteration process, etc. We find that multi-objective optimization (MOO) methods can achieve a good balance among multiple targets, which satisfies the unique needs of key-value services. The more specific Quality of Service (QoS) requirements users can provide, the better performance these algorithms can achieve. We also find that the number of concurrent threads and the write buffer size are the two most impactful parameters determining the throughput and the 99th percentile tail latency across different hardware and workloads. Finally, we provide system-level explanations for the auto-tuning results and also discuss the associated implications for system designers and practitioners. We hope this work will pave the way towards a practical, high-speed auto-tuning solution for key-value data store systems.