Machine learning based optimization for database workloads

该项目旨在研究数据库工作负载的机器学习和自优化算法。重点是能够通过工作负载分类、性能预测和资源规划和执行实现运行时自适应重新配置的技术。当前业界采用的最先进的方法是基于为访问计划中的每种消费查询操作符（例如排序、散列连接等）实现的许多静态内存模型。每个模型根据操作符类型（例如基数、列数、列宽度和各种其他统计数据）提供不同的输入，并计算操作符将使用的内存的估计值。虽然这些模型是在理解运行时行为的基础上构建的，但它们是静态模型，不够精确，也不能进行自调优和自优化。因此，当操作符的运行时实现或操作环境发生变化时，模型就会过时。与现有方法相比，该项目着眼于广泛的深度学习预测模型和自适应前瞻性优化，以实现准确、经济高效的运行时决策。该项目的第一个目标是开发和测试用于数据库工作负载估计的新机器学习模型。第二个互补目标是开发新的基于模型的内存、计算和存储分配的自优化算法。这项研究将产生知识和创新，从而产生(i)更好的系统利用率和更低的成本，以及（ii）更好的系统性能和稳定性。与此同时，该项目将培养新一代的研究人员和开发人员，接受机器学习、数据库系统和云计算方面的培训。该项目的合作伙伴是IBM加拿大实验室和IBM高级研究中心。

This project aims to investigate machine learning and self-optimization algorithms for database workloads. The focus is on techniques that can enable run-time adaptive reconfiguration through workload classification, performance prediction and resource planning and execution. Current state-of-the-art approaches adopted in the industry reside on a number of static memory models implemented for each sort of consuming query operator in the access plan (e.g. sort, hash join, etc.). Each model is provided with different inputs depending on the operator type (e.g. cardinality, number of columns, column widths, and various other statistics) and computes an estimate of the memory that would be used by the operator. While these models were built with an understanding of runtime behavior, they are static models, not sufficiently accurate and not enabled for self-tuning and self-optimization. As a result, when runtime implementation of an operator or the operating environment changes, then the models become outdated. In contrast to existing approaches, this project looks at a broad spectrum of deep learning models for prediction and at adaptive look-ahead optimization to account for accurate, cost-effective runtime decisions. The first goal of the project is to develop and test new machine learning models for database workload estimation. The second complementary goal is to develop new model based self-optimization algorithms for memory, computing and storage allocation. The research will generate knowledge and innovation that yield (i) better system utilization and less cost and (ii) better system performance and stability. At the same time, the project will prepare a new generation of researchers and developers, trained in machine learning, database systems and cloud computing.The partner in the project is IBM Canada Lab and IBM Centre for Advanced Studies.