I-Corps: Interactive and Automated Debugging for Big Data Analytics

点击翻译按钮获取中文摘要

The broader impact of this I-Corps project is to investigate the challenges that data scientists face in debugging big data analytics today and to investigate the commercial potential of research work on interactive and automated debugging of big data analytics. Big data analytics is increasingly important in the 21st century, where our daily lives leave behind a detailed digital record. Decision-makers of all kinds, from companies to government agencies, would like to base their actions on data. If successful, this project will offer a unique opportunity to discover software development tooling needs for big data systems and to identify innovative software tooling products that sit across the software stack from the user-facing API all the way down to the systems infrastructure.This I-Corps project builds on early research work on real-time debugging primitives and tool-assisted fault-localization services for big data processing applications written in modern data intensive scalable computing (DISC) systems like Apache Spark. Designing debugging primitives for DISC requires re-thinking the traditional step-through debugging primitives as provided by tools such as gdb. For example, a breakpoint feature that simply pauses the entire computation would waste large amounts of computational resources and prevent correct tasks from completing, reducing overall throughput. Further, requiring the user to inspect the millions of intermediate records produced during execution is clearly infeasible. When a failure or incorrect result is generated (e.g., outlier), pinpointing the root cause is extremely time-consuming and expensive due to massive scale of data. In short, the intellectual merit of this I-Corps project is to investigate how users can benefit from expressive debugging primitives and automated fault localization services when they must leverage for data science and big data analytics capabilities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.