CSR: Small: Scalable Fine-Grain Lineage for Debugging Data-Intensive Workflows

这个项目解决了继续采用“大数据”分析的下一个主要障碍-它们的生命周期管理，包括调试、调优和审计。 如今，数据密集型分析正在改善多个行业的运营，将TB级的原始数据转化为有用的数据分析。 利用大数据将是保持从发电到零售、石油勘探、制造业、各种科学学科和国家安全等领域的竞争优势所必需的。 然而，这些数据处理架构的极端可扩展性隐藏了效率低下和混淆性能分析，为他们的irad选项带来了明显和隐藏的成本。 调整和调试大型数据密集型工作流目前是一种黑艺术，主要由繁琐的手动分析组成。该研究旨在大幅改变数据科学家设计和调试其分析的方式，以避开这种创作和部署瓶颈。特别是，PI正在开发可扩展的高效架构，用于捕获细粒度数据沿袭，通过分析管道跟踪数据使用的信息，从一系列数据密集型可扩展计算（DISC）系统。 这样的谱系作为通过逐步调试、故障跟踪、异常检测和谱系驱动的数据清理和数据挖掘来消除低效率和建议优化的基础。 开发和开源发布这样的血统捕获和分析平台有望大大加快大数据分析的采用。

This project addresses the next major impediment to the continued adoptionof "big-data" analytics---the management of their life cycle, whichincludes debugging, tuning, and auditing. Today, data-intensive analyticsare improving operations across multiple industries, translating terabytesof raw data into useful data analysis. Taking advantage of big data willbe necessary to sustain competitive advantages for areas ranging from powergeneration, to retail, oil exploration, manufacturing, various scientificdisciplines, and national security. However, the extreme scalability ofthese data processing architectures hides inefficiencies and obfuscatesperformance analysis, creating both obvious and hidden costs to theiradoption. Tuning and debugging large data-intensive workflows is currentlya black art that mostly consists of tedious manual analysis.The research seeks to dramatically alter how data scientists design anddebug their analytics to sidestep this authoring and deployment bottleneck.In particular, the PI's are developing scalable, efficient architecturesfor capturing fine-grain data lineage, information that tracks the use ofdata through the analytic pipeline, from a range of data-intensive scalablecomputing (DISC) systems. Such lineage serves as a basis for discoveringinefficiencies and suggesting optimizations via step-wise debugging, faulttracing, anomaly detection, and lineage-driven data cleaning and datamining. The development and open-source release of such lineage-captureand analysis platforms promises to dramatically accelerate the adoption ofbig-data analytics.