HCC: Medium: Improving data visualization and analysis tools to support reasoning about analysis assumptions

使用统计数据对数据进行建模通常需要对数据所代表的内容做出假设，包括基础模式和错误的性质。例如，一位对财富和年龄之间的关系感兴趣的分析师可能会从她的数据中删除极低或极高的报告年龄作为异常值，并选择使用线性模型对其余数据进行建模，这意味着年龄会以某种恒定因素增加财富。她的结果可能意味着，与对应该包括哪些数据做出不同决定的分析相比，关于财富和年龄如何相关的结论可能会有很大不同。对于分析师来说，解释这种敏感性的一种方法是，在给定数据集和他们想要使用数据回答的问题的情况下，报告许多合理分析的结果。不幸的是，现有的数据分析和可视化工具几乎不支持对这种“多元宇宙分析”的推理。它们对做出不同选择的多个模型和可视化的比较提供了有限的支持，在帮助分析人员推理和表达这些选择方面提供的支持更少。此外，几乎没有已知的方法可以有效地传达特定分析结果中的不确定性和与该分析中所作假设有关的不确定性。这个项目的目标是改进多元宇宙分析：更好地了解分析师目前是如何思考多元宇宙分析的，并确定帮助分析师使用多元宇宙分析的需求、机会和方法。该项目通过创建新的方法和工具来帮助分析人员定义、推理和表达他们可以分析数据的多种替代方法，从而解决了表达与分析选择相关的难以量化的不确定性方面的这些挑战。研究将侧重于分析师使用的两种常见工具：便于绘制和比较数据的可视化分析软件，以及允许更无缝地整合代码和叙述性评论的计算笔记本。项目团队将开发新的用户界面和编程库，以获取分析师的知识，以及新的视觉表示和交互技术，分析师可以通过这些技术在不同的模型或分析路径之间进行比较。该项目还将产生新的软件基础设施，使在现有工具和工作流程中进行和评估多种分析成为可能。此外，该团队将开发更好地交流多元宇宙分析的方法：使多元宇宙分析报告可共享的方法，包含分析代码和数字以及叙事上下文的互动文档，以及描述合理分析的不同表示如何影响读者理解的实证结果。这些研究活动将以与真实世界的分析师进行的形成性研究的结果为指导，这些研究将解决分析师在定义和推理他们本可以采取的替代模型或分析步骤方面面临的困难的现有知识差距。所有研究结果和计算工具将免费和公开提供。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Using statistics to model data often requires making assumptions about what the data represent, including the nature of underlying patterns and errors. For example, an analyst interested in the relationship between wealth and age might remove extremely low or high reported ages from her data as outliers, and choose to model the remaining data using a linear model, which implies that age increases wealth by some constant factor. Her results could imply substantially different conclusions about how wealth and age relate compared to an analysis that made different decisions about which data to include. One way for analysts to account for such sensitivity is by reporting the results of many reasonable analyses given a dataset and questions they want to answer using the data. Unfortunately, existing data analysis and visualization tools offer little support for reasoning about such a “multiverse analysis.” They provide limited support for comparison of multiple models and visualizations that make different choices, and even less support in helping analysts reason about and express those choices. Further, there are few known ways to effectively convey both uncertainty in the results of a given analysis and uncertainty related to the assumptions made in that analysis. This project’s goal is to improve multiverse analysis: to better understand how analysts currently think about multiverse analysis, and to identify needs, opportunities, and approaches to help analysts use multiverse analyses. This project addresses these challenges to expressing hard-to-quantify uncertainty related to analysis choices by creating new methods and tools to help analysts define, reason about, and express multiple alternative ways they could analyze their data. The research will focus on two common types of tools analysts use: visual analysis software that makes it easy to plot and compare data, and computational notebooks that allow for more seamless integration of code and narrative commentary. The project team will develop new user interfaces and programming libraries to elicit analysts’ knowledge, as well as new visual representations and interaction techniques by which an analyst can compare between alternative models or analysis paths. The project will also produce novel software infrastructure to make conducting and evaluating multiple analyses feasible within existing tools and workflows. Further, the team will develop ways to better communicate multiverse analyses: ways to make multiverse analysis reports shareable, interactive documents that contain both the analysis code and figures as well as narrative context, and empirical results describing how different representations of plausible analyses impact readers’ understanding. These research activities will be guided by the results of formative studies with real-world analysts that will address gaps in existing knowledge about the difficulties analysts face in defining and reasoning about alternative models or analysis steps they could have taken. All study results and computational tools will be made freely and publicly available.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.