III: Small: Automatic Database Management System Tuning Through Large-scale Machine Learning

III：小型：通过大规模机器学习自动调整数据库管理系统

• 批准号: 1423210
• 负责人: Andrew Pavlo
• 金额: $ 49.97万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-08-01 至 2018-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1423210&HistoricalAwards=false
• 关键词: III; Small; Automatic; Database; Management

The ability to collect, process, and analyze large amounts of data is paramount for being able to extrapolate new knowledge in business, scientific, and medical applications. Database management systems (DBMSs) are the critical component of modern "Big Data" applications because they are the central repository for all of this information. But tuning a DBMS to perform well is historically a difficult task because they have hundreds of configuration "knobs" that control everything in the system, such as the amount of memory to use and how often data is written. Getting these settings wrong will prevent the system from answering questions about data in a reasonable amount of time or even cause it to lose data. Many organizations resort to hiring experts to configure these knobs, but this is prohibitively expensive. Personnel cost is estimated to be almost 50% of the total ownership cost of a DBMS, and many administrators spend nearly a quarter of their time on these tuning activities. Furthermore, as databases grow in both size and complexity, optimizing a DBMS to meet the needs of new applications has surpassed the abilities of even the best human experts. Thus, the goal of this proposal is to develop the foundation and corresponding practical techniques for the automatic configuration of DBMSs by using machine learning on large-scale collections of historical performance data. Our approach will differ from previous work in that we seek to reduce the amount of time that is needed to train the algorithms that tune the DBMS for each application by relying on knowledge gained from previous tuning efforts. The results from this work will allow anyone to deploy a DBMS that is able to handle large amounts of data and more complex workloads without any expertise in database administration.Achieving good performance in a database management system (DBMS) is non-trivial because they are complex systems with many tunable options that control nearly all aspects of their runtime operation. Getting this tuning right is critical for modern high-volume and high-throughput workloads, as the performance gains can be significant. As such, many organizations resort to hiring an expensive database administrator to manually tune their DBMS. But the size and complexity of databases have now surpassed the abilities of even the best human experts. Hence, we plan to develop automatic techniques for tuning and optimizing DBMS configurations for a broad class of application workloads. We will explore the foundations of using machine learning to scale DBMSs for larger data sets, thereby removing a major impediment in deriving the full benefits of data-driven decision making applications. The crux of our approach is to map an arbitrary application's workload to features of one or more canonical benchmarks that best represents the workload's properties, and then to collect performance data from the DBMS using that benchmark. This data is then used to train models that will allow us to identify the dependencies between knobs and their effects on the DBMS. From this, the models will select a near-optimal knob setting for the application. This differs from earlier work that focused on optimizing a single DBMS installation in isolation and are unable to leverage knowledge gained from previous tuning efforts. Our approach will not require the user to generate a large sample data set of (potentially expensive) experiments to derive the proper configuration.For further information see project web site at: http://oltpbenchmark.com

收集，处理和分析大量数据的能力对于能够推断业务，科学和医学应用方面的新知识至关重要。数据库管理系统（DBMS）是现代“大数据”应用程序的关键组成部分，因为它们是所有这些信息的中央存储库。 但是，将DBM调整为良好是一项艰巨的任务，因为它们具有数百种控制系统中所有内容的配置“旋钮”，例如要使用的内存量和编写数据的频率。将这些设置误解将阻止系统在合理的时间内回答有关数据的问题，甚至导致其丢失数据。许多组织诉诸雇用专家以配置这些旋钮，但这非常昂贵。人员成本估计占DBM的总拥有成本的近50％，许多管理员将近四分之一的时间用于这些调整活动。此外，随着数据库的规模和复杂性的增长，优化DBM以满足新应用程序的需求，甚至超出了最佳人类专家的能力。因此，该提案的目的是通过在历史绩效数据的大规模集合中使用机器学习来开发DBMS自动配置的基础和相应的实用技术。我们的方法与以前的工作有所不同，因为我们试图减少训练通过从以前的调整工作中获得的知识来训练每个应用程序调整DBM的算法所需的时间。这项工作的结果将使任何人都可以部署能够处理大量数据和更复杂的工作负载的DBM，而无需在数据库管理方面任何专业知识。在数据库管理系统（DBMS）中获得良好的性能是非平地的，因为它们是具有许多可调选项的复杂系统，这些系统几乎可以控制运行时运行的所有方面。正确进行调整对于现代的大量和高通量工作量至关重要，因为性能增长可能很大。因此，许多组织求助于雇用昂贵的数据库管理员手动调整其DBMS。但是，数据库的规模和复杂性现在已经超过了最好的人类专家的能力。因此，我们计划开发用于调整和优化大量应用程序工作负载的DBMS配置的自动技术。我们将探索使用机器学习来扩展DBMS的基础，以消除在得出数据驱动决策应用程序的全部好处的主要障碍中。我们方法的关键是将任意应用程序的工作负载映射到最能代表工作负载属性的一个或多个规范基准的功能，然后使用该基准从DBMS收集性能数据。然后将这些数据用于训练模型，以使我们能够识别旋钮及其对DBM的影响之间的依赖性。由此，模型将为应用程序选择一个近乎最佳的旋钮设置。这与较早的工作不同，该工作重点是孤立地优化单个DBMS安装，并且无法利用以前的调整工作中获得的知识。我们的方法将不需要用户生成大量的示例数据集（潜在昂贵）实验以得出适当的配置。有关更多信息，请参见项目网站：http：//oltpbenchmark.com