CSR-DMSS, SM: Incremental Web-scale Data Processing

CSR-DMSS、SM：增量网络规模数据处理

• 批准号: 0834784
• 负责人: Kenneth Yocum
• 金额: $ 10万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2008
• 资助国家: 美国
• 起止时间: 2008-09-01 至 2009-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0834784&HistoricalAwards=false
• 关键词: CSR; DMSS; SM; Incremental; Web

This project seeks to develop incremental processing abstractions andtechnologies to address the approaching bottleneck in processingunstructured web-scale data. Government, medical, financial, and web-basedservices increasingly depend on the ability to rapidly sift through huge,evolving data sets. These data-intensive applications perform complexmulti-step computations over successive generations of data inflows (e.g.,weekly web crawls or nightly telescope dumps). Current approaches toprocessing unstructured data have driven the development of massivelyparallel "ad-hoc" data processing systems, such as MapReduce. However,they process data in a snap-shot fashion, forcing massive re-computationswhen even a small amount of new data arrives.The core of the project consists of a cluster-based incremental dataprocessing system that overcomes these limitations. A key component is adataflow programming model that combines massive parallelism and flexible,incremental computations. An incremental processing controllerorchestrates multiple backend data processing tasks, ensuring reliable,consistent operation in the event of node failures. The project seeks toshed light on the fundamental challenges and benefits of incrementalprocessing for ad-hoc data by using both industrial and e-scienceapplications. For example, through cooperation with Yahoo! Research, theproject will vet existing prototypes on real-world web-indexing dataflowsand large data sets. While in the short term the project provides aplatform for such highly skilled operators, the long-term goal is tosignificantly advance the methods and abstractions that the scientificcommunity and commercial world use to tackle processing vast, dynamic datasets.

该项目旨在开发增量处理抽象和技术，以解决处理非结构化网络规模数据的瓶颈问题。 政府、医疗、金融和基于网络的服务越来越依赖于快速筛选庞大、不断变化的数据集的能力。 这些数据密集型应用程序在连续几代数据流入上执行复杂的多步计算（例如，每周网络抓取或每晚望远镜转储）。 当前处理非结构化数据的方法已经推动了并行“ad-hoc”数据处理系统的发展，例如MapReduce。 然而，它们以快照的方式处理数据，即使是少量的新数据到达时，也会迫使大量的重新计算。该项目的核心是一个基于集群的增量数据处理系统，它克服了这些限制。 一个关键组件是adataflow编程模型，它结合了大量的并行性和灵活的增量计算。 增量处理队列可协调多个后端数据处理任务，确保在发生节点故障时可靠、一致地运行。 该项目旨在阐明通过使用工业和电子科学应用程序对特定数据进行增量处理的基本挑战和好处。 例如，通过与Yahoo！研究，该项目将审查现有的原型在现实世界中的网络索引数据库和大型数据集。 虽然在短期内，该项目为这些高技能的操作人员提供了一个平台，但长期目标是显著推进科学界和商业界用于处理大量动态数据集的方法和抽象。