III: Large: Collaborative Research: Analysis Engineering for Robust End-to-End Data Science

从导致科学“发现”被撤回的糟糕统计实践，到颠覆高风险分析的低级电子表格错误，数据分析的失败可能会带来灾难性的后果。在过去十年中，数据科学实践的快速增长导致了大量的协作努力，以开发新的数据处理，机器学习和分析工具，将更先进的数据分析交到更广泛的从业者手中，从学生到科学家再到设计师。数据科学最主要的工具是代码，可以从现有库中应用尖端算法。然而，随着数据科学的民主化降低了使用先进方法的门槛，在合理的统计实践下安全地使用这些工具仍然像以往一样困难。为了促进更强大的数据科学，本项目研究了编写程序的数据科学家用于分析工程的模型和工具。重点是用代码执行数据分析的完整端到端过程：分析人员将数据转化为数据的迭代，通常是探索性的步骤。该项目将有助于分析工作的见解和特征，捕获和分析数据科学活动的新方法，并开发用于创作和验证分析的新编程工具和可视化方法。如果成功，该项目将增强人们进行和评估数据分析的能力，促进更可靠的结果，并缩小新手和专家分析师之间的差距。项目的发现和工具将被纳入教育工作，包括课堂教学和教程，并作为开源软件集成到流行的分析环境中（例如，Jupyter）。数据分析是科学研究的一项核心活动，但往往以散漫的方式进行。这个项目将整个分析过程作为我们研究的中心现象。该项目将采用混合方法来研究和描述常见的分析实践和缺陷，包括数据分析师的直接观察，计算笔记本的大规模分析，以及分析编程环境（如JupyterLab）的仪器仪表。该项目将提供用于指定和保护分析的新方法，包括特定于领域的语言和程序合成方法，以指导用户优选下一步。它还将探索“多元宇宙”工作流程，以管理和评估分析决策的多样性。将开发调试和测试工具的类似物来标记问题并执行错误分析，同时捕获和可视化分析来源以帮助再现性、验证和协作审查。这项工作将通过对照研究、课堂使用和大规模现场使用的开源部署进行评估。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

From poor statistical practices leading to retractions of scientific "discoveries" to low-level spreadsheet errors subverting high-stakes analyses, failures of data analysis can have catastrophic consequences. The rapid growth of data science practice in the last decade has led to large collaborative efforts to develop new data processing, machine learning, and analytics tools that put more advanced data analysis into the hands of a wider audience of practitioners, from students to scientists to designers. The most dominant tool for data science is code, where cutting-edge algorithms can be applied from an existing libraries. However, as this democratization of data science has lowered the barrier to using advanced methods, safely using these tools under sound statistical practice remains as difficult as ever. To facilitate more robust data science, this project investigates models and tools for analysis engineering by data scientists who write programs. The focus is on the complete end-to-end process of data analysis performed with code: the iterative, and often exploratory, steps that analysts go through to turn data into This project will contribute insights and characterizations of analytic work, novel methods for capturing and analyzing data science activities, and develop new programming tools and visualization methods for authoring and validating analyses. If successful, this project will augment people's ability to conduct and assess data analyses, promoting more robust results and reducing the gap between novice and expert analysts. The findings and tools from the project will be incorporated into educational efforts, including classroom teaching and tutorials and available as open source software integrated into popular analytical environments (e.g., Jupyter).Data analysis is a central activity to scientific research, yet is too often conducted in an undisciplined fashion. This project treats the entire analytic process as our central phenomenon of study. The project will employ mixed methods to study and characterize common analysis practices and pitfalls, including direct observations of data analysts, large-scale analysis of computational notebooks, and instrumentation of analytic programming environments like JupyterLab. The project will contribute new methods for specifying and safeguarding analyses, including domain-specific languages and program synthesis methods to guide users to preferred next steps. It will also explore "multiverse" workflows to manage and assess a diversity of analysis decisions. Analogues of debugging and testing tools will be developed to flag problems and perform error analysis, while the capture and visualization of analytic provenance to aid reproducibility, verification, and collaborative review. The work will be evaluated through controlled studies, classroom use, and open-source deployment for wide-scale field use.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.