III: Small: Non-Invasive Real-Time Analytics in Database Systems using Holistic Query Compilation

现代数据处理应用程序有两个主要趋势，使其有别于过去几十年的应用程序。首先，它们以其不断变化的数据集而闻名。这可能来自更新数据库的事务，也可能来自上游来源。第二，他们希望尽快分析最新获得的数据。数据一经创建就具有巨大的价值，但随着时间的推移，这种价值会逐渐减少。因此，查询必须访问生成的最新数据，才能使其结果产生最大影响。在数据进入数据库时立即询问有关数据的复杂问题的能力在许多应用程序领域都很有用，包括实时监控系统(例如，传入的包是否来自潜在攻击者？)和金融服务(例如，这种新的信用卡购买是否具有欺诈性？)。但当前的系统包含遗留数据库管理系统(DBMS)的体系结构残留物，使它们无法利用较新的硬件支持进行并行优化。这限制了应用程序在数据到达时立即以数据为目标的DBMS上执行的查询类型。反过来，这在硬件和管理开销方面增加了部署数据库应用程序的额外成本。因此，该项目的目标是研究使用查询编译来实现非侵入性的分析操作，这些操作比今天的DBMS中的实际操作更复杂。这样的查询编译技术对广泛的数据处理系统是有益的。这项研究的结果将使组织能够部署能够以更少的资源(例如硬件、人员、能源)处理具有更大数据集和更复杂工作负载的应用程序的DBMS。现代数据密集型应用程序寻求通过分析历史数据集和最近收集的数据的组合来实时获得新的见解。为了支持这样的工作负载，数据库管理系统(DBMS)需要支持对不同数据集的复杂分析查询。随着DRAM成本的不断降低，越来越多的此类应用程序可以驻留在内存中。因此，内存中的DBMS将用于未来大多数分析和机器学习应用程序。但是，遗留的面向磁盘的DBMS处理查询的方式仍然存在于较新的内存中DBMS中，这阻碍了本项目针对大型数据集的高性能查询执行。因此，本项目的目标是通过一种新的整体查询编译方法克服这一障碍，这种方法在整个数据库管理系统中全面整合，并建立(并适应)“即时”编译技术和各种硬件资源的最新进展。使用编译来优化DBMS体系结构的许多不同方面对于支持未来的“大数据”应用程序非常重要，这些应用程序需要摄取大量新数据，同时近乎实时地执行复杂的分析工作负载。

There are two major trends in modern data processing applications that make them distinct from applications in previous decades. The first is that they are noted for their continuously changing data sets. This could come from transactions updating the database or from upstream sources. The second is that they want to analyze the latest obtained data as quickly as possible. Data has immense value as soon as it is created, but that value diminishes over time. Therefore, it is imperative that the queries access the newest data generated in order for their results to have the most impact. The ability to ask complex questions about data as soon as it enters in the database is useful in many application domains, including real-time monitoring systems (e.g., is an incoming packet from a potential attacker?) and financial services (e.g., is this new credit card purchase fraudulent?). But current systems contain architecture remnants of legacy database management systems (DBMSs) that prevent them from taking advantage of newer hardware support for parallel optimizations. This limits the types of queries that an application executes on a DBMS that targets data as soon as it arrives. In turn, this adds additional cost to deploying a database application in terms of both hardware and administration overhead. Thus, the goal of this project is to investigate using query compilation to allow non-invasive analytical operations that are more complex than what is practical in today's DBMSs. Such query compilation techniques are beneficial to a wide array of data processing systems. The results of this study will allow organizations to deploy DBMSs that are able to handle applications with larger data sets and more complex workloads with fewer resources (e.g., hardware, personnel, energy).Modern data-intensive applications seek to obtain new insights in real-time by analyzing a combination of historical data sets alongside recently collected data. To support such workloads, database management systems (DBMSs) need to support complex analytical queries over diverse data sets. The ever decreasing cost of DRAM is allowing a greater number of these applications to be memory-resident. As such, in-memory DBMSs will be used for most analytical and machine learning applications in the future. But there are remnants of how legacy disk-oriented DBMSs process queries that still exist in newer in-memory DBMSs that inhibit the kind of high-performance query execution over large data sets that this project targets. Thus, the goal of this project is to overcome this barrier through a new holistic approach to query compilation that integrates it comprehensively throughout the DBMS, and which builds upon (and adapts) recent advances in "just-in-time" (JIT) compilation technology and heterogeneous hardware resources. Using compilation to optimize many different aspects of the DBMS's architecture is important to support future "Big Data" applications that need to ingest large amounts of new data while simultaneously executing complex analytical workloads in near real-time.