CSR: Small: Scalable Fine-Grain Lineage for Debugging Data-Intensive Workflows

点击翻译按钮获取中文摘要

This project addresses the next major impediment to the continued adoptionof "big-data" analytics---the management of their life cycle, whichincludes debugging, tuning, and auditing. Today, data-intensive analyticsare improving operations across multiple industries, translating terabytesof raw data into useful data analysis. Taking advantage of big data willbe necessary to sustain competitive advantages for areas ranging from powergeneration, to retail, oil exploration, manufacturing, various scientificdisciplines, and national security. However, the extreme scalability ofthese data processing architectures hides inefficiencies and obfuscatesperformance analysis, creating both obvious and hidden costs to theiradoption. Tuning and debugging large data-intensive workflows is currentlya black art that mostly consists of tedious manual analysis.The research seeks to dramatically alter how data scientists design anddebug their analytics to sidestep this authoring and deployment bottleneck.In particular, the PI's are developing scalable, efficient architecturesfor capturing fine-grain data lineage, information that tracks the use ofdata through the analytic pipeline, from a range of data-intensive scalablecomputing (DISC) systems. Such lineage serves as a basis for discoveringinefficiencies and suggesting optimizations via step-wise debugging, faulttracing, anomaly detection, and lineage-driven data cleaning and datamining. The development and open-source release of such lineage-captureand analysis platforms promises to dramatically accelerate the adoption ofbig-data analytics.