III: Small: Non-Invasive Real-Time Analytics in Database Systems using Holistic Query Compilation

III：小型：使用整体查询编译在数据库系统中进行非侵入式实时分析

基本信息

批准号：
1718582
负责人：
Andrew Pavlo
金额：
$ 49.98万
依托单位：
Carnegie-Mellon University
依托单位国家：
美国
项目类别：
Continuing Grant
财政年份：
2017
资助国家：
美国
起止时间：
2017-08-01 至 2020-07-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1718582&HistoricalAwards=false
关键词：
III Small Non Invasive Real

项目摘要

There are two major trends in modern data processing applications that make them distinct from applications in previous decades. The first is that they are noted for their continuously changing data sets. This could come from transactions updating the database or from upstream sources. The second is that they want to analyze the latest obtained data as quickly as possible. Data has immense value as soon as it is created, but that value diminishes over time. Therefore, it is imperative that the queries access the newest data generated in order for their results to have the most impact. The ability to ask complex questions about data as soon as it enters in the database is useful in many application domains, including real-time monitoring systems (e.g., is an incoming packet from a potential attacker?) and financial services (e.g., is this new credit card purchase fraudulent?). But current systems contain architecture remnants of legacy database management systems (DBMSs) that prevent them from taking advantage of newer hardware support for parallel optimizations. This limits the types of queries that an application executes on a DBMS that targets data as soon as it arrives. In turn, this adds additional cost to deploying a database application in terms of both hardware and administration overhead. Thus, the goal of this project is to investigate using query compilation to allow non-invasive analytical operations that are more complex than what is practical in today's DBMSs. Such query compilation techniques are beneficial to a wide array of data processing systems. The results of this study will allow organizations to deploy DBMSs that are able to handle applications with larger data sets and more complex workloads with fewer resources (e.g., hardware, personnel, energy).Modern data-intensive applications seek to obtain new insights in real-time by analyzing a combination of historical data sets alongside recently collected data. To support such workloads, database management systems (DBMSs) need to support complex analytical queries over diverse data sets. The ever decreasing cost of DRAM is allowing a greater number of these applications to be memory-resident. As such, in-memory DBMSs will be used for most analytical and machine learning applications in the future. But there are remnants of how legacy disk-oriented DBMSs process queries that still exist in newer in-memory DBMSs that inhibit the kind of high-performance query execution over large data sets that this project targets. Thus, the goal of this project is to overcome this barrier through a new holistic approach to query compilation that integrates it comprehensively throughout the DBMS, and which builds upon (and adapts) recent advances in "just-in-time" (JIT) compilation technology and heterogeneous hardware resources. Using compilation to optimize many different aspects of the DBMS's architecture is important to support future "Big Data" applications that need to ingest large amounts of new data while simultaneously executing complex analytical workloads in near real-time.

现代数据处理应用程序中有两个主要趋势，使它们与过去几十年的应用不同。首先是他们因其不断变化的数据集而被指出。这可能来自更新数据库或上游来源的交易。第二个是他们想尽快分析最新获得的数据。数据创建后立即具有巨大的值，但是随着时间的流逝，该值的值会降低。因此，必须查询访问生成的最新数据，以便其结果产生最大的影响。在数据库中输入数据的复杂问题的能力在许多应用程序域中很有用，包括实时监控系统（例如，来自潜在攻击者的传入数据包）和金融服务（例如，这是新的信用卡购买欺诈性吗？）。但是当前的系统包含旧数据库管理系统（DBMS）的体系结构残留物，该系统阻止了它们利用并行优化的较新的硬件支持。这限制了应用程序在DBMS上执行数据后立即执行的查询类型。反过来，这增加了根据硬件和管理开销部署数据库应用程序的额外成本。因此，该项目的目的是使用查询汇编进行调查，以允许非侵入性分析操作比当今DBMS中实用更为复杂。这种查询汇编技术对各种数据处理系统有益。这项研究的结果将使组织能够部署能够处理具有较大数据集的应用程序和更复杂的资源（例如，硬件，人员，能源）的更复杂的工作负载。现代数据密集型应用程序寻求通过分析最近收集到的数据集的组合来实时获得新的见解。为了支持此类工作负载，数据库管理系统（DBMS）需要支持各种数据集的复杂分析查询。 DRAM的成本不断下降，使这些应用程序中有更多的记忆居住。因此，将来将用于大多数分析和机器学习应用程序中的内存DBMS。但是，有一些残余的内容是如何在较新的内存DBMS中仍然存在的面向遗产的DBMS处理查询，这些查询抑制了该项目目标的大型数据集，这些内存中的DBMS中仍存在着抑制高性能查询执行的类型。因此，该项目的目的是通过一种新的整体方法来克服这一障碍，以在整个DBMS中全面整合它，并基于“正义”（JIT）汇编技术和异构硬件资源的最新进展（并适应）最新的进步。使用汇编来优化DBMS体系结构的许多不同方面，对于支持未来的“大数据”应用程序很重要，这些应用程序需要摄入大量新数据，同时近乎实时执行复杂的分析工作负载。