SHF: Medium: Interactive Debegging for Big Data Analytics

点击翻译按钮获取中文摘要

An abundance of data in science, engineering, national security, and health care has led to the emerging field of big data analytics. To process massive quantities of data, developers leverage data-intensive scalable computing (DISC) systems in the cloud, such as Google's MapReduce, Apache Hadoop, and Apache Spark. While DISC systems help to address the scalability challenges of big data analytics, they also introduce an enormous challenge for data scientists in understanding and resolving errors. This project addresses the severe lack of debugging support in DISC systems today, which makes it difficult for data scientists to understand their applications, determine the causes of identified errors, and ensure that such errors are properly repaired. The research provides two kinds of debugging support for big data processing programs in modern DISC systems like Apache Spark: new interactive, real-time debugging primitives for large-scale distributed processing and tool-assisted fault-localization services for big data. Technical approaches include a new data provenance technique for providing fine-grained visibility into large-scale distributed data processing and runtime optimizations for iterative development and debugging workloads. Tool-assisted fault localization services leverage these underlying provenance and optimization techniques to pinpoint and characterize the root causes of errors efficiently. Big data analytics is increasingly important in the 21st century, where daily lives leave behind a detailed digital record and decision-makers of all kinds, from companies to government agencies, would like to base their actions on data. The research contributes to improving productivity and correctness of big data applications, which is crucial for many disciplines that distill terabytes of low-value data into high-value insights.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.