III: Small: Bringing database query optimization to data intensive applications

III：小型：将数据库查询优化引入数据密集型应用程序

• 批准号: 2008295
• 负责人: Kenneth Ross
• 金额: $ 50万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2008295&HistoricalAwards=false
• 关键词: III; Small; Bringing; database; query

Modern database management systems (DBMSs) employ sophisticated query optimization techniques that enable the generation of efficient execution plans for queries over very large data sets. A variety of other applications also process large data sets, but cannot leverage database-style query optimization for their code. In this project, an open-source programming language compiler will be enhanced with database-style query optimization capabilities. Data-intensive parts of ordinary programs will be executed in chunks using different execution plans. Based on feedback from earlier chunks, alternative plans might be used for later chunks. The compiler could be used for a variety of data-intensive applications, allowing all of them to benefit from this class of performance optimizations.Existing query optimization techniques for in-memory processing are limited in several ways: (a) they are not extensively used outside relational database management systems; (b) they are limited to a handful of relational operators, and do not cover access patterns or dynamically-defined functions found in other data-analysis scenarios; (c) they treat the underlying compiler as a black-box, with unpredictable performance depending on which compiler is used with which compiler settings; (d) they often bake-in design choices that may be appropriate for usage within a particular DBMS, but not for more general cases. This project directly addresses these challenges by optimizing data-analysis style queries expressed as tight loops in a conventional imperative programming language. This project will extend an open-source compiler (GraalVM/Truffle) with both known and novel optimization techniques that will automatically be applied whenever the compiler identifies that a loop is time-consuming. Integration into the compiler allows many applications to efficiently process large data sets. The system will support dynamic queries involving user-defined functions and arbitrary access patterns. Database-style and compiler optimizations will co-exist in one system, eliminating some of the mismatches that happen when the compiler is used as a black-box by a DBMS. The system will tune a variety of run-time execution parameters automatically, with minimal guidance from the programmer. The extended compiler will be validated by developing an interactive data exploration application that will allow users to dynamically specify and analyze a variety of large in-memory datasets.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代数据库管理系统（dbms）采用复杂的查询优化技术，能够为非常大的数据集上的查询生成高效的执行计划。其他各种应用程序也处理大型数据集，但不能对其代码利用数据库样式的查询优化。在这个项目中，开源编程语言编译器将增强数据库风格的查询优化功能。普通程序的数据密集型部分将使用不同的执行计划在块中执行。根据来自早期数据块的反馈，可以为以后的数据块使用替代计划。该编译器可用于各种数据密集型应用程序，使所有这些应用程序都能从这类性能优化中受益。现有的内存处理查询优化技术在以下几个方面受到限制：(a)它们在关系数据库管理系统之外没有广泛使用；(b)它们仅限于少数关系运算符，不包括在其他数据分析场景中发现的访问模式或动态定义函数；(c)他们将底层编译器视为黑盒，其性能取决于使用哪种编译器和哪种编译器设置；(d)它们经常在设计选择中加入可能适合在特定DBMS中使用的选项，但不适合更一般的情况。这个项目通过优化在传统命令式编程语言中表达为紧密循环的数据分析风格查询，直接解决了这些挑战。该项目将扩展开源编译器（GraalVM/Truffle），使用已知的和新颖的优化技术，当编译器识别出循环耗时时，将自动应用这些优化技术。集成到编译器中允许许多应用程序有效地处理大型数据集。系统将支持涉及用户定义函数和任意访问模式的动态查询。数据库风格和编译器优化将在一个系统中共存，消除了当编译器被DBMS用作黑盒时发生的一些不匹配。系统将自动调整各种运行时执行参数，而程序员的指导最少。扩展的编译器将通过开发交互式数据探索应用程序来验证，该应用程序将允许用户动态指定和分析各种大型内存数据集。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Adaptive Code Generation for Data-Intensive Analytics

• DOI: 10.14778/3447689.3447697
• 发表时间: 2021-02
• 期刊: Proc. VLDB Endow.
• 作者: Wangda Zhang;Junyoung Kim;K. A. Ross;Eric Sedlar;Lukas Stadler

DIEL: Interactive Visualization Beyond the Here and Now

DIEL：超越此时此地的交互式可视化

• 发表时间: 2021
• 期刊: IEEE Vis
• 作者: Wu, Yifan;Chang, Remco;Hellerstein, Joseph;Satyanarayan, Arvind;Wu, Eugene
• 通讯作者: Wu, Eugene