CAREER: Query Compilation Techniques for Complex Analytics on Enterprise Clusters

职业：企业集群上复杂分析的查询编译技术

• 批准号: 1453171
• 负责人: Tim Kraska
• 金额: $ 55万
• 依托单位: Brown University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-06-01 至 2023-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1453171&HistoricalAwards=false
• 关键词: CAREER; Query; Compilation; Techniques; Complex

Big data and the evolving field of Data Science are fundamentally shifting the meaning of analytics. Highly complex computations have come to define the typical workload, with jobs ranging from machine learning to large-scale visualization. However, there is a fundamental discrepancy between the availability of analytical tools for big Internet companies and those for non-tech enterprises. Current analytics frameworks, like Spark or Hadoop, are designed to meet the needs of giant Internet companies; that is, they are built to process petabytes of data in cloud deployments consisting of thousands of cheap commodity machines. Yet non-tech companies like banks and retailers - or even the typical data scientist - seldom operate deployments of that size, instead preferring smaller clusters, aka Enterprise clusters, with more reliable hardware. In fact, recent industry surveys reported that the median Hadoop cluster was fewer than 10 nodes, and over 65% of users operate clusters smaller than 50 nodes. Targeting complex analytics workloads on smaller clusters, however, fundamentally changes the way we should design analytics tools. Most current systems focus on the major challenges associated with large cloud deployments, where network and disk I/O are the primary bottleneck and failures are common, where the next generation of analytics frameworks should optimize specifically for the computation bottleneck. As part of this project, the PIs will systematically design a new analytical open-source engine, called Tupleware, build specifically for the infrastructure of non-tech companies. Tupleware will make complex analytics more accessible and push the boundaries of what computations are possible.Specifically, the PIs will design, implement and evaluate various program synthesis, i.e., query compilation techniques, for complex analytics on enterprise clusters with fast interconnects and considerable available memory. Existing query compilation techniques focus on SQL and are not designed for workloads where UDFs and iterations dominate the computation, nor do they target distributed setups; all issues the PIs will address in this proposal. Furthermore, the PIs aim to combine high-level query optimization with compiler technology to holistically optimize complex analytical workflows by considering statistics about the data (e.g., the selectivity of predicates) with low-level statistics about the UDFs (e.g., the number of used registers). Finally, all the results will be integrated into the Tupleware system and thus, made accessible for a broader range of users.For further information see the project web site at: http://tupleware.cs.brown.edu/

大数据和数据科学不断发展的领域从根本上转移了分析的含义。高度复杂的计算已经定义了典型的工作量，从机器学习到大规模可视化，作业不等。但是，大型互联网公司的分析工具的可用性与非技术企业的分析工具之间存在根本差异。当前的分析框架（如Spark或Hadoop）旨在满足巨型互联网公司的需求；也就是说，它们是为了处理由数千台廉价商品机器组成的云部署中的数据之前。然而，像银行和零售商这样的非技术公司（甚至是典型的数据科学家）很少经营这种尺寸的部署，而是更喜欢较小的群集（又名企业集群），并具有更可靠的硬件。实际上，最近的行业调查报告说，Hadoop集群中位数少于10个节点，超过65％的用户操作簇小于50个节点。但是，针对较小群集的复杂分析工作负载从根本上改变了我们应该设计分析工具的方式。当前的大多数系统都集中在与大型云部署相关的主要挑战上，其中网络和磁盘I/O是主要的瓶颈，而失败很常见，下一代分析框架应专门为计算瓶颈优化。作为该项目的一部分，PIS将系统地设计一种名为Tupleware的新分析开源引擎，专门为非技术公司的基础架构建立。 Tupleware将使复杂的分析更容易访问，并突破哪些计算的界限。特别是，PIS将设计，实施和评估各种程序合成，即查询编译技术，以便在企业集群中使用快速互连和可用的可用内存，以对企业集群进行复杂的分析。现有的查询汇编技术集中于SQL，并且不是为UDF和迭代占主导地位的工作负载而设计的，也不针对分布式设置。 PI在此提案中将解决的所有问题。此外，PIS旨在通过考虑有关数据（例如谓词的选择性）与UDFS的低级统计数据（例如，使用的登记册的数量），将高级查询优化与编译器技术相结合，以整体优化复杂的分析工作流程。最后，所有结果都将集成到Tupleware系统中，因此可以为更广泛的用户访问。有关更多信息，请参见项目网站：http：//tupleware.cs.brown.edu/