III: Small: Non-Invasive Real-Time Analytics in Database Systems using Holistic Query Compilation

III：小型：使用整体查询编译在数据库系统中进行非侵入式实时分析

• 批准号: 1718582
• 负责人: Andrew Pavlo
• 金额: $ 49.98万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-08-01 至 2020-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1718582&HistoricalAwards=false
• 关键词: III; Small; Non; Invasive; Real

There are two major trends in modern data processing applications that make them distinct from applications in previous decades. The first is that they are noted for their continuously changing data sets. This could come from transactions updating the database or from upstream sources. The second is that they want to analyze the latest obtained data as quickly as possible. Data has immense value as soon as it is created, but that value diminishes over time. Therefore, it is imperative that the queries access the newest data generated in order for their results to have the most impact. The ability to ask complex questions about data as soon as it enters in the database is useful in many application domains, including real-time monitoring systems (e.g., is an incoming packet from a potential attacker?) and financial services (e.g., is this new credit card purchase fraudulent?). But current systems contain architecture remnants of legacy database management systems (DBMSs) that prevent them from taking advantage of newer hardware support for parallel optimizations. This limits the types of queries that an application executes on a DBMS that targets data as soon as it arrives. In turn, this adds additional cost to deploying a database application in terms of both hardware and administration overhead. Thus, the goal of this project is to investigate using query compilation to allow non-invasive analytical operations that are more complex than what is practical in today's DBMSs. Such query compilation techniques are beneficial to a wide array of data processing systems. The results of this study will allow organizations to deploy DBMSs that are able to handle applications with larger data sets and more complex workloads with fewer resources (e.g., hardware, personnel, energy).Modern data-intensive applications seek to obtain new insights in real-time by analyzing a combination of historical data sets alongside recently collected data. To support such workloads, database management systems (DBMSs) need to support complex analytical queries over diverse data sets. The ever decreasing cost of DRAM is allowing a greater number of these applications to be memory-resident. As such, in-memory DBMSs will be used for most analytical and machine learning applications in the future. But there are remnants of how legacy disk-oriented DBMSs process queries that still exist in newer in-memory DBMSs that inhibit the kind of high-performance query execution over large data sets that this project targets. Thus, the goal of this project is to overcome this barrier through a new holistic approach to query compilation that integrates it comprehensively throughout the DBMS, and which builds upon (and adapts) recent advances in "just-in-time" (JIT) compilation technology and heterogeneous hardware resources. Using compilation to optimize many different aspects of the DBMS's architecture is important to support future "Big Data" applications that need to ingest large amounts of new data while simultaneously executing complex analytical workloads in near real-time.

现代数据处理应用程序有两个主要趋势，使其与前几十年的应用程序不同。首先，它们以其不断变化的数据集而闻名。这可能来自更新数据库的事务或来自上游源。第二，他们希望尽快分析最新获得的数据。数据一经创建就具有巨大的价值，但随着时间的推移，这种价值会逐渐减少。因此，查询必须访问生成的最新数据，以使其结果产生最大的影响。在数据一进入数据库就询问关于数据的复杂问题的能力在许多应用领域中是有用的，包括实时监控系统（例如，是来自潜在攻击者的传入数据包？）和金融服务（例如，这种新的信用卡购买是欺诈吗？）。但是当前的系统包含遗留数据库管理系统（DBMS）的架构残余，这阻止了它们利用新的硬件支持进行并行优化。这限制了应用程序在DBMS上执行的查询类型，DBMS在数据到达时就将其作为目标。反过来，这在硬件和管理开销方面增加了部署数据库应用程序的额外成本。因此，这个项目的目标是研究使用查询编译来允许非侵入性的分析操作，这些操作比今天的DBMS中的实际操作更复杂。这样的查询编译技术对于广泛的数据处理系统是有益的。这项研究的结果将使组织能够部署能够用更少的资源处理具有更大数据集和更复杂工作负载的应用程序的DBMS（例如，现代数据密集型应用寻求通过分析历史数据集以及最近收集的数据的组合来实时获得新的见解。为了支持这样的工作负载，数据库管理系统（DBMS）需要支持对不同数据集的复杂分析查询。DRAM成本的不断降低使得更多的应用程序可以驻留在内存中。因此，内存中的DBMS将在未来用于大多数分析和机器学习应用程序。但是，遗留的面向磁盘的DBMS处理查询的方式仍然存在于较新的内存中DBMS中，这些查询抑制了该项目针对的大型数据集的高性能查询执行。因此，这个项目的目标是克服这个障碍，通过一个新的整体的方法来查询编译，它全面集成在整个DBMS，并建立在（和适应）的最新进展“即时”（JIT）编译技术和异构硬件资源。使用编译来优化DBMS架构的许多不同方面对于支持未来的“大数据”应用程序非常重要，这些应用程序需要摄取大量新数据，同时近乎实时地执行复杂的分析工作负载。

项目成果

Relaxed Operator Fusion for In-Memory Databases: Making Compilation, Vectorization, and Prefetching Work Together At Last

• DOI: 10.14778/3151113.3151114
• 发表时间: 2017-09
• 期刊: Proc. VLDB Endow.
• 作者: Prashanth Menon;Andrew Pavlo;T. Mowry

Permutable compiled queries: dynamically adapting compiled queries without recompiling

可改变的编译查询：动态调整编译查询而无需重新编译

• DOI: 10.14778/3425879.3425882
• 发表时间: 2020
• 期刊: Proceedings of the VLDB Endowment
• 影响因子: 2.5
• 作者: Menon, Prashanth;Ngom, Amadou;Ma, Lin;Mowry, Todd C.;Pavlo, Andrew
• 通讯作者: Pavlo, Andrew