CSR: Small: Scalable Fine-Grain Lineage for Debugging Data-Intensive Workflows

CSR：小型：用于调试数据密集型工作流程的可扩展细粒度谱系

• 批准号: 1219220
• 负责人: Alin Deutsch
• 金额: $ 45万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-09-01 至 2018-03-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1219220&HistoricalAwards=false
• 关键词: CSR; Small; Scalable; Fine; Grain

This project addresses the next major impediment to the continued adoptionof "big-data" analytics---the management of their life cycle, whichincludes debugging, tuning, and auditing. Today, data-intensive analyticsare improving operations across multiple industries, translating terabytesof raw data into useful data analysis. Taking advantage of big data willbe necessary to sustain competitive advantages for areas ranging from powergeneration, to retail, oil exploration, manufacturing, various scientificdisciplines, and national security. However, the extreme scalability ofthese data processing architectures hides inefficiencies and obfuscatesperformance analysis, creating both obvious and hidden costs to theiradoption. Tuning and debugging large data-intensive workflows is currentlya black art that mostly consists of tedious manual analysis.The research seeks to dramatically alter how data scientists design anddebug their analytics to sidestep this authoring and deployment bottleneck.In particular, the PI's are developing scalable, efficient architecturesfor capturing fine-grain data lineage, information that tracks the use ofdata through the analytic pipeline, from a range of data-intensive scalablecomputing (DISC) systems. Such lineage serves as a basis for discoveringinefficiencies and suggesting optimizations via step-wise debugging, faulttracing, anomaly detection, and lineage-driven data cleaning and datamining. The development and open-source release of such lineage-captureand analysis platforms promises to dramatically accelerate the adoption ofbig-data analytics.

该项目解决了继续采用“大数据”分析的下一个主要障碍——对其生命周期的管理，包括调试、调优和审计。如今，数据密集型分析正在改善多个行业的运营，将tb级的原始数据转化为有用的数据分析。从发电、零售、石油勘探、制造业、各种科学学科到国家安全等领域，利用大数据将是保持竞争优势的必要条件。然而，这些数据处理架构的极端可扩展性隐藏了效率低下和混淆性能分析，为采用它们创造了明显和隐藏的成本。目前，调优和调试大型数据密集型工作流是一项繁琐的手工分析工作。该研究试图极大地改变数据科学家如何设计和调试他们的分析，以避开这个创作和部署瓶颈。特别是，PI正在开发可扩展的高效架构，用于捕获细粒度数据谱系，通过分析管道跟踪数据使用的信息，来自一系列数据密集型可扩展计算（DISC）系统。这种沿袭可以作为发现效率低下和通过逐步调试、故障跟踪、异常检测以及沿袭驱动的数据清理和数据挖掘提出优化建议的基础。这种谱系捕获和分析平台的开发和开源发布有望极大地加速大数据分析的采用。