EAGER: A Python Program Analysis Infrastructure to Facilitate Better Data Processing

Python是第三大最流行的编程语言，仅次于C和Java，也是机器学习和数据科学中使用最广泛的语言。Python中的应用程序与其他语言中的应用程序一样容易出现人为错误，或者由于Python的动态特性，可能更容易出现人为错误。因此，分析、测试、验证和优化Python应用程序的工具是迫切需要的。这些工具对于Python来说是滞后的，或者根本不存在。根本原因是缺乏支持构建实用和有效工具的基础设施，这需要解决Python的动态特性，例如动态类型、动态代码加载/执行以及对以其他语言实现的外部库函数的普遍调用。该项目旨在通过开发两个示例工具来探索构建Python程序分析基础设施的可行性，这两个示例工具依赖于一组通用的基础设施功能，包括仪器、静态分析和符号分析功能。这两个示例工具是用于机器学习应用程序的数据来源跟踪工具和用于检测数据格式不一致的bug查找工具，这是数据处理中最主要的bug类型。溯源工具将演示静态分析和程序插装的重要性，bug查找工具将演示符号分析的重要性。这两个工具都将说明高级工具可以给数据科学家带来的巨大好处。此外，它们将说明上述功能不能简单地从其他语言（如C和Java）的现有基础结构移植过来。该基础设施将满足对Python的全面工具构建支持的迫切需求。许多尖端的协同研究将在CISE研究社区中实现，以服务于数据应用程序程序员、数据科学家甚至最终用户。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Python is the third most popular programming language, after C and Java,  and the most widely used language in Machine Learning and Data Science. Applications in Python are prone to human errors as much as those in other languages, or maybe more so due to the dynamic nature of Python. Therefore, tools to analyze, test, verify, and optimize Python applications are in a pressing need. Such tools are lagging or non-existent for Python. The root cause is the lack of infrastructure to support building practical and effective tools, which entails addressing the dynamic features of Python, such as dynamic typing, dynamic code loading/execution, and pervasive invocations to external library functions implemented in other languages. This project aims to explore the feasibility of building a Python program analysis infrastructure by developing two sample tools that rely upon a common set of infrastructural capabilities including the instrumentation, static analysis and symbolic analysis capabilities. The two sample tools are a data provenance tracking tool for machine learning applications and a bug finding tool to detect data format inconsistencies, which are the most dominant type of bugs in data processing. The provenance tool will demonstrate the importance of static analysis and program instrumentation, and the bug finding tool will demonstrate the importance of symbolic analysis. Both tools will illustrate the great benefits that can be brought to data scientists by advanced tools. In addition, they will illustrate that the aforementioned capabilities cannot be simply ported from existing infrastructures for other languages such as C and Java. The infrastructure will meet the pressing need of comprehensive tool building support for Python. A lot of cutting-edge synergistic research will be enabled across the CISE research community to serve data application programmers, data scientists and even end users.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.