CSR-DMSS, SM: Incremental Web-scale Data Processing

点击翻译按钮获取中文摘要

This project seeks to develop incremental processing abstractions andtechnologies to address the approaching bottleneck in processingunstructured web-scale data. Government, medical, financial, and web-basedservices increasingly depend on the ability to rapidly sift through huge,evolving data sets. These data-intensive applications perform complexmulti-step computations over successive generations of data inflows (e.g.,weekly web crawls or nightly telescope dumps). Current approaches toprocessing unstructured data have driven the development of massivelyparallel "ad-hoc" data processing systems, such as MapReduce. However,they process data in a snap-shot fashion, forcing massive re-computationswhen even a small amount of new data arrives.The core of the project consists of a cluster-based incremental dataprocessing system that overcomes these limitations. A key component is adataflow programming model that combines massive parallelism and flexible,incremental computations. An incremental processing controllerorchestrates multiple backend data processing tasks, ensuring reliable,consistent operation in the event of node failures. The project seeks toshed light on the fundamental challenges and benefits of incrementalprocessing for ad-hoc data by using both industrial and e-scienceapplications. For example, through cooperation with Yahoo! Research, theproject will vet existing prototypes on real-world web-indexing dataflowsand large data sets. While in the short term the project provides aplatform for such highly skilled operators, the long-term goal is tosignificantly advance the methods and abstractions that the scientificcommunity and commercial world use to tackle processing vast, dynamic datasets.