BIGDATA: Mid-Scale: DCM: Collaborative Research: Eliminating the Data Ingestion Bottleneck in Big Data Applications

BIGDATA：中型：DCM：协作研究：消除大数据应用中的数据摄取瓶颈

• 批准号: 1247726
• 负责人: Michael Bender
• 金额: $ 80万
• 依托单位: SUNY at Stony Brook
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2013
• 资助国家: 美国
• 起止时间: 2013-02-01 至 2019-01-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1247726&HistoricalAwards=false
• 关键词: BIGDATA; Mid; Scale; DCM; Collaborative

Big-data practice suggests that there is a tradeoff between the speed of data ingestion, the ability to answer queries quickly (e.g., via indexing), and the freshness of data. This perceived tradeoff lies, for example, at the heart of the historic division between OLTP (online transaction processing) and OLAP (online analytical processing). In an OLTP database, data gets ingested quickly and the data available for querying is fresh, but analytical queries run prohibitively slowly. In an OLAP data warehouse, data is buffered for off-line indexing so that analytical queries run quickly, but by the time the data gets indexed, it is stale. This tradeoff has manifestations in the design of all types of storage systems. For example, some file-systems are optimized for reads and others for writes, but workloads generally involve a mixture of reads and writes. In this project the PIs show that this is not a fundamental tradeoff, but rather a tradeoff imposed by the choice of data structure. The PIs use write-optimized structures, an alternative to traditional indexing methodologies, to build storage systems in which this tradeoff is significantly mitigated or alleviated altogether. The performance promise of such indexing schemes follows from the PIs previous work establishing that write-optimized data structures can speed up both inserts and queries. This project addresses the remaining obstacles in the deployment of write-optimized indexes within big-data file-systems and databases. Big data imposes a new set of constraints on any storage system, and the PIs will show how write-optimized indexing can yield order-of-magnitude performance improvements at scale. In particular, this project will show that such techniques are not only applicable today but that they will scale with hardware trends, including the widespread adoption of solid-state disks (SSDs).

大数据实践表明，数据摄取速度、快速回答查询的能力（例如通过索引）和数据的新鲜度之间存在权衡。例如，这种感知到的权衡是 OLTP（在线事务处理）和 OLAP（在线分析处理）之间历史性划分的核心。在 OLTP 数据库中，数据获取速度很快，并且可用于查询的数据是最新的，但分析查询的运行速度却非常慢。在 OLAP 数据仓库中，数据被缓冲以用于离线索引，以便分析查询快速运行，但当数据被索引时，它已经过时了。这种权衡体现在所有类型存储系统的设计中。例如，某些文件系统针对读取进行了优化，另一些则针对写入进行了优化，但工作负载通常涉及读取和写入的混合。在这个项目中，PI 表明这不是一个基本的权衡，而是数据结构选择所强加的权衡。 PI 使用写优化结构（传统索引方法的替代方案）来构建存储系统，在该系统中，这种权衡得到显着缓解或完全缓解。此类索引方案的性能承诺源自 PI 之前的工作，即确定写入优化的数据结构可以加速插入和查询。该项目解决了在大数据文件系统和数据库中部署写入优化索引的剩余障碍。大数据对任何存储系统都施加了一系列新的约束，PI 将展示写入优化索引如何能够大规模地实现数量级的性能改进。特别是，该项目将表明此类技术不仅适用于当今，而且将随着硬件趋势而扩展，包括固态磁盘 (SSD) 的广泛采用。