SHF: Medium: Interactive Debegging for Big Data Analytics

SHF：中：大数据分析的交互式调试

• 批准号: 1764077
• 负责人: Miryung Kim
• 金额: $ 90万
• 依托单位: University of California-Los Angeles
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-07-01 至 2024-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1764077&HistoricalAwards=false
• 关键词: SHF; Medium; Interactive; Debegging; Big

An abundance of data in science, engineering, national security, and health care has led to the emerging field of big data analytics. To process massive quantities of data, developers leverage data-intensive scalable computing (DISC) systems in the cloud, such as Google's MapReduce, Apache Hadoop, and Apache Spark. While DISC systems help to address the scalability challenges of big data analytics, they also introduce an enormous challenge for data scientists in understanding and resolving errors. This project addresses the severe lack of debugging support in DISC systems today, which makes it difficult for data scientists to understand their applications, determine the causes of identified errors, and ensure that such errors are properly repaired. The research provides two kinds of debugging support for big data processing programs in modern DISC systems like Apache Spark: new interactive, real-time debugging primitives for large-scale distributed processing and tool-assisted fault-localization services for big data. Technical approaches include a new data provenance technique for providing fine-grained visibility into large-scale distributed data processing and runtime optimizations for iterative development and debugging workloads. Tool-assisted fault localization services leverage these underlying provenance and optimization techniques to pinpoint and characterize the root causes of errors efficiently. Big data analytics is increasingly important in the 21st century, where daily lives leave behind a detailed digital record and decision-makers of all kinds, from companies to government agencies, would like to base their actions on data. The research contributes to improving productivity and correctness of big data applications, which is crucial for many disciplines that distill terabytes of low-value data into high-value insights.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

科学，工程，国家安全和医疗保健方面的大量数据导致了大数据分析的新兴领域。 为了处理大量数据，开发人员利用云中的数据密集型可扩展计算系统（例如Google的MapReduce，Apache Hadoop和Apache Spark）。尽管光盘系统有助于应对大数据分析的可伸缩性挑战，但它们还为数据科学家在理解和解决错误时引入了巨大的挑战。该项目解决了当今光盘系统中严重缺乏调试支持，这使得数据科学家难以理解其应用，确定识别错误的原因，并确保正确修复此类错误。 该研究为现代光盘系统中的大数据处理程序提供了两种调试支持，例如Apache Spark：新的交互式，实时调试原始图，用于大规模分布式处理和用于大数据的工具辅助的故障 - 平局服务。技术方法包括一种新的数据出处技术，用于为迭代开发和调试工作负载提供大规模的分布式数据处理和运行时优化。工具辅助的故障本地化服务利用这些基本的出处和优化技术来确定和表征错误的根本原因。大数据分析在21世纪越来越重要，在21世纪，从公司到政府机构，日常生活留下了详细的数字记录和各种决策者，希望将其行动基于数据。这项研究有助于提高大数据应用的生产力和正确性，这对于许多学科至关重要，这些学科将低价值数据的Terabytes提炼为高价值见解。该奖项反映了NSF的法定任务，并且通过使用该基金会的知识功能和广泛的影响来评估CRITERIA CRITERIA CRITERIA。

项目成果

An Empirical Study of Common Challenges in Developing Deep Learning Applications

• DOI: 10.1109/issre.2019.00020
• 发表时间: 2019-10
• 期刊: 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE)
• 作者: Tianyi Zhang;Cuiyun Gao;Lei Ma;Michael R. Lyu;Miryung Kim

Software Engineering for Data Analytics

数据分析软件工程

• DOI: 10.1109/ms.2020.2985775
• 发表时间: 2020
• 期刊: IEEE Software
• 影响因子: 3.3
• 作者: Kim, Miryung

Sibylvariant Transformations for Robust Text Classification

用于稳健文本分类的 Sibylvariant 变换

• DOI: 10.18653/v1/2022.findings-acl.140
• 发表时间: 2022
• 期刊: Findings of the Association for Computational Linguistics: ACL 2022
• 作者: Harel-Canada, Fabrice;Gulzar, Muhammad Ali;Peng, Nanyun;Kim, Miryung
• 通讯作者: Kim, Miryung

Canvas: Isolated and Adaptive Swapping for Multi-Applications on Remote Memory

• DOI: 10.48550/arxiv.2203.09615
• 发表时间: 2022-03
• 作者: Chenxi Wang;Yifan Qiao;Haoran Ma;Shiafun Liu;Yiying Zhang;Wenguang Chen;R. Netravali;Miryung Kim;Guoqing Harry Xu

Data Scientists in Software Teams: State of the Art and Challenges

• DOI: 10.1109/tse.2017.2754374
• 发表时间: 2018-11-01
• 期刊: IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
• 影响因子: 7.4
• 作者: Kim, Miryung;Zimmermann, Thomas;Begel, Andrew
• 通讯作者: Begel, Andrew