CAREER: Self-Driving Database Management Systems

职业：自动驾驶数据库管理系统

• 批准号: 1846158
• 负责人: Andrew Pavlo
• 金额: $ 49.41万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-02-15 至 2024-01-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1846158&HistoricalAwards=false
• 关键词: CAREER; Self; Driving; Database; Management

Over the last four decades, both researchers and vendors have built advisory tools to assist human administrators in database management system (DBMS) tuning and physical design. All of this previous work, however, is incomplete because they require humans to make the final decisions about any changes and they are reactionary measures that fix problems after they occur. What is needed for a truly "self-driving" DBMS is a new software architecture that is explicitly designed for autonomous operation. With this, the DBMS will remove the need for humans to oversee and maintain the software. It also enables new optimizations that are important for modern high-performance DBMSs, but which are not possible today because the complexity of managing these systems has surpassed the abilities of human experts. Such a system will remove the human capital impediments of deploying databases and allow organizations in all facets of society (e.g., business, science, government) to more easily derive the benefits of data-driven decision-making applications. The techniques developed as part of this research are also applicable to other problem domains where autonomous operation could improve a software system's performance and efficiency, including both larger systems (e.g., distributed DBMSs, data centers) and smaller devices (e.g., mobile devices, IoT sensors).This project investigates techniques for self-driving DBMSs that combines state-of-the-art methods from database systems, machine learning (ML), and control theory. Achieving autonomous operation in a DBMS is now possible due to algorithmic advancements in ML, as well as improvements to storage and computation hardware. What makes this different than earlier attempts is that all aspects of the system are controlled by an integrated planning component that not only optimizes the system for the current workload but also predicts future workload trends before they occur so that the system can prepare itself accordingly. This work will produce on-line methods for discovering relevant optimization actions based on these workload forecast models, thereby enabling the planning component to converge to a better configuration with less training data. In addition to this, this research will study how to deploy these actions without hindering the DBMS's performance (e.g., downtime due to restarts) or causing incorrect behavior. The outcome will be a set of first principles for efficient self-driving DBMS architectures that can deploy these modifications and provide the necessary feedback for their integrated models. Such principles are timely in identifying issues that inhibit automation of existing systems and influencing the design of future DBMS architectures.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在过去的40年里，研究人员和供应商都构建了咨询工具，以帮助人类管理员进行数据库管理系统(DBMS)调优和物理设计。然而，所有这些之前的工作都是不完整的，因为它们需要人类对任何变化做出最终决定，而且它们是在问题发生后修复问题的反动措施。一个真正的“自动驾驶”数据库管理系统所需要的是一个明确为自主操作而设计的新软件体系结构。有了这一点，数据库管理系统将不再需要人类来监督和维护软件。它还实现了新的优化，这些优化对现代高性能DBMS很重要，但在今天是不可能的，因为管理这些系统的复杂性已经超出了人类专家的能力。这一系统将消除部署数据库的人力资本障碍，使社会各方面的组织(例如，商业、科学、政府)能够更容易地从数据驱动的决策应用程序中获益。作为本研究的一部分，开发的技术也适用于自主操作可以提高软件系统的性能和效率的其他问题领域，包括较大的系统(例如，分布式DBMS、数据中心)和较小的设备(例如，移动设备、物联网传感器)。本项目研究结合数据库系统、机器学习(ML)和控制理论的最先进方法的自动驾驶DBMS技术。由于ML算法的进步以及存储和计算硬件的改进，在DBMS中实现自主操作现在是可能的。与以前的尝试不同的是，系统的所有方面都由一个综合规划组件控制，该组件不仅针对当前的工作量优化系统，而且在未来的工作量趋势发生之前对其进行预测，以便系统能够相应地做好准备。这项工作将产生基于这些工作量预测模型发现相关优化行动的在线方法，从而使规划部分能够以更少的训练数据收敛到更好的配置。除此之外，本研究还将研究如何在不影响DBMS性能的情况下部署这些操作(例如，由于重新启动导致的停机)或导致不正确的行为。其结果将是一套有效的自动驾驶DBMS架构的基本原则，这些架构可以部署这些修改并为其集成模型提供必要的反馈。这些原则能够及时发现阻碍现有系统自动化的问题，并影响未来DBMS架构的设计。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

MB2: Decomposed Behavior Modeling for Self-Driving Database Management Systems

• DOI: 10.1145/3448016.3457276
• 发表时间: 2021-06
• 期刊: Proceedings of the 2021 International Conference on Management of Data
• 作者: Lin Ma;William Zhang;Jie Jiao;Wuwen Wang;Matthew Butrovich;Wan Shen Lim;Prashanth Menon;Andrew Pavlo

Make Your Database System Dream of Electric Sheep: Towards Self-Driving Operation

• DOI: 10.14778/3476311.3476411
• 发表时间: 2021-07
• 期刊: Proc. VLDB Endow.
• 作者: Andrew Pavlo;Matthew Butrovich;Lin Ma;Prashanth Menon;Wan Shen Lim;Dana Van Aken;William Zhang

External vs. Internal: An Essay on Machine Learning Agents for Autonomous Database Management Systems

• DOI: 10.37745/ejcsit.2013/vol10n52431
• 发表时间: 2022-05
• 期刊: IEEE Data Eng. Bull.
• 作者: Andrew Pavlo;Matthew Butrovich;Ananya Joshi;Lin Ma;Prashanth Menon;Dana Van Aken;Lisa Lee;R. Salakhutdinov

Mainlining databases: supporting fast transactional workloads on universal columnar data file formats

主线数据库：支持通用列式数据文件格式的快速事务工作负载

• DOI: 10.14778/3436905.3436913
• 发表时间: 2020
• 期刊: Proceedings of the VLDB Endowment
• 影响因子: 2.5
• 作者: Li, Tianyu;Butrovich, Matthew;Ngom, Amadou;Lim, Wan Shen;McKinney, Wes;Pavlo, Andrew
• 通讯作者: Pavlo, Andrew

Everything is a Transaction: Unifying Logical Concurrency Control and Physical Data Structure Maintenance in Database Management Systems

• 发表时间: 2021
• 作者: Ling Zhang;Matthew Butrovich;Tianyu Li;Yash Nannapanei;Andrew Pavlo;J. Rollinson;Huanchen Zhang;Ambarish Balakumar;Daniel Biales;Ziqi Dong;Emmanuel Eppinger;Jordi Gonzàlez;Wan Shen Lim;Jianqiao Liu;Prashanth Menon;Soumil Mukherjee;Tanuj Nayak;Amadou Latyr Ngom;Jeff Niu;D. Patra;P. Raj;Stephanie Wang;Wuwen Wang;Yao-Tin Yu;William Zhang