III: Large: Collaborative Research: Analysis Engineering for Robust End-to-End Data Science

从糟糕的统计实践导致科学“发现”的撤回，到低级电子表格错误颠覆高风险的分析，数据分析的失败可能会产生灾难性的后果。在过去十年中，数据科学实践的快速增长导致了大规模的合作努力，以开发新的数据处理、机器学习和分析工具，将更高级的数据分析掌握在更广泛的从业者手中，从学生到科学家再到设计师。数据科学最主要的工具是代码，在代码中可以从现有的库中应用尖端算法。然而，由于数据科学的民主化降低了使用先进方法的障碍，在合理的统计实践下安全使用这些工具仍然一如既往地困难。为了促进更强大的数据科学，该项目研究了由编写程序的数据科学家进行分析工程的模型和工具。重点是使用代码执行的完整的端到端数据分析过程：分析师将数据转化为此项目所经历的迭代且通常是探索性的步骤将有助于分析工作的洞察和特征、捕获和分析数据科学活动的新方法，以及开发用于创作和验证分析的新编程工具和可视化方法。如果成功，这个项目将增强人们进行和评估数据分析的能力，促进更有力的结果，并缩小新手和专家分析师之间的差距。该项目的成果和工具将纳入教育工作，包括课堂教学和教程，并作为开放源码软件整合到流行的分析环境中(例如木星)。数据分析是科学研究的一项中心活动，但往往以无纪律的方式进行。这个项目将整个分析过程视为我们研究的中心现象。该项目将使用混合方法来研究和描述常见的分析做法和陷阱，包括对数据分析员的直接观察、对计算笔记本的大规模分析以及对JupyterLab等分析编程环境的工具。该项目将提供指定和保护分析的新方法，包括特定领域的语言和程序合成方法，以指导用户首选的下一步步骤。它还将探索“多元宇宙”工作流程，以管理和评估各种分析决策。将开发调试和测试工具的模拟工具，以标记问题和进行错误分析，同时捕获和可视化分析来源，以帮助重复性、核查和协作审查。这项工作将通过受控研究、课堂使用和大规模现场使用的开源部署进行评估。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

From poor statistical practices leading to retractions of scientific "discoveries" to low-level spreadsheet errors subverting high-stakes analyses, failures of data analysis can have catastrophic consequences. The rapid growth of data science practice in the last decade has led to large collaborative efforts to develop new data processing, machine learning, and analytics tools that put more advanced data analysis into the hands of a wider audience of practitioners, from students to scientists to designers. The most dominant tool for data science is code, where cutting-edge algorithms can be applied from an existing libraries. However, as this democratization of data science has lowered the barrier to using advanced methods, safely using these tools under sound statistical practice remains as difficult as ever. To facilitate more robust data science, this project investigates models and tools for analysis engineering by data scientists who write programs. The focus is on the complete end-to-end process of data analysis performed with code: the iterative, and often exploratory, steps that analysts go through to turn data into This project will contribute insights and characterizations of analytic work, novel methods for capturing and analyzing data science activities, and develop new programming tools and visualization methods for authoring and validating analyses. If successful, this project will augment people's ability to conduct and assess data analyses, promoting more robust results and reducing the gap between novice and expert analysts. The findings and tools from the project will be incorporated into educational efforts, including classroom teaching and tutorials and available as open source software integrated into popular analytical environments (e.g., Jupyter).Data analysis is a central activity to scientific research, yet is too often conducted in an undisciplined fashion. This project treats the entire analytic process as our central phenomenon of study. The project will employ mixed methods to study and characterize common analysis practices and pitfalls, including direct observations of data analysts, large-scale analysis of computational notebooks, and instrumentation of analytic programming environments like JupyterLab. The project will contribute new methods for specifying and safeguarding analyses, including domain-specific languages and program synthesis methods to guide users to preferred next steps. It will also explore "multiverse" workflows to manage and assess a diversity of analysis decisions. Analogues of debugging and testing tools will be developed to flag problems and perform error analysis, while the capture and visualization of analytic provenance to aid reproducibility, verification, and collaborative review. The work will be evaluated through controlled studies, classroom use, and open-source deployment for wide-scale field use.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.