I-Corps: Interactive and Automated Debugging for Big Data Analytics

该 I-Corps 项目的更广泛影响是调查数据科学家当今在调试大数据分析时面临的挑战，并调查大数据分析交互式和自动调试研究工作的商业潜力。大数据分析在 21 世纪变得越来越重要，我们的日常生活留下了详细的数字记录。从公司到政府机构的各种决策者都希望根据数据采取行动。如果成功，该项目将提供一个独特的机会来发现大数据系统的软件开发工具需求，并确定创新的软件工具产品，这些产品横跨软件堆栈，从面向用户的 API 一直到系统基础设施。这个 I-Corps 项目建立在实时调试原语和工具辅助故障定位服务的早期研究工作的基础上，这些研究工作适用于用现代数据密集型可扩展计算 (DISC) 系统（如 Apache Spark）编写的大数据处理应用程序。为 DISC 设计调试原语需要重新思考 gdb 等工具提供的传统逐步调试原语。例如，简单地暂停整个计算的断点功能会浪费大量计算资源并阻止正确的任务完成，从而降低总体吞吐量。此外，要求用户检查执行期间产生的数百万条中间记录显然是不可行的。当生成失败或不正确的结果（例如异常值）时，由于数据规模庞大，查明根本原因非常耗时且昂贵。简而言之，这个 I-Corps 项目的智力价值在于研究用户在必须利用数据科学和大数据分析功能时如何从表达性调试原语和自动故障定位服务中受益。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

The broader impact of this I-Corps project is to investigate the challenges that data scientists face in debugging big data analytics today and to investigate the commercial potential of research work on interactive and automated debugging of big data analytics. Big data analytics is increasingly important in the 21st century, where our daily lives leave behind a detailed digital record. Decision-makers of all kinds, from companies to government agencies, would like to base their actions on data. If successful, this project will offer a unique opportunity to discover software development tooling needs for big data systems and to identify innovative software tooling products that sit across the software stack from the user-facing API all the way down to the systems infrastructure.This I-Corps project builds on early research work on real-time debugging primitives and tool-assisted fault-localization services for big data processing applications written in modern data intensive scalable computing (DISC) systems like Apache Spark. Designing debugging primitives for DISC requires re-thinking the traditional step-through debugging primitives as provided by tools such as gdb. For example, a breakpoint feature that simply pauses the entire computation would waste large amounts of computational resources and prevent correct tasks from completing, reducing overall throughput. Further, requiring the user to inspect the millions of intermediate records produced during execution is clearly infeasible. When a failure or incorrect result is generated (e.g., outlier), pinpointing the root cause is extremely time-consuming and expensive due to massive scale of data. In short, the intellectual merit of this I-Corps project is to investigate how users can benefit from expressive debugging primitives and automated fault localization services when they must leverage for data science and big data analytics capabilities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.