CAREER: Understanding visual reasoning for visual communication

本项目旨在促进对视觉推理在视觉交流中的理解。视觉推理使人们能够将视觉输入转化为抽象的概念。例如，要使用天气图解释哪些县将获得更多降雪量，就必须弄清楚天气图上的哪些颜色表示降雪量。人们对视觉特征应该如何映射到可视化中的概念有期望，即使映射有明确的标签，他们也更难解释违反这些期望的可视化。然而，这些期望的性质及其在视觉推理中的作用还没有得到很好的理解，因此信息可视化的设计往往是无原则的和特别的。通过更好地了解视觉推理是如何工作的，就有可能设计出适合其优势的可视化设计，并优化视觉交流。研究人员将通过研究人们如何在可视化中从颜色推断含义来解决这个问题。这项研究可以转化为制作用于设计可视化的在线工具，这将改善STEM教育，提高公众识字能力和对科学技术的参与。该教育计划将使用视觉交流，通过虚拟现实(VR)以及伴随着色彩和可视化的实践体验，使大学本科生和中学生更容易接触和参与科学。调查人员还将通过在PI、研究生、本科生实习生和中学女生中进行指导，支持扩大女性在STEM中的参与。这项拟议的研究将解决视觉推理如何实现视觉交流的基本问题。在目标1中，研究人员将研究人们如何通过环境统计学习将感知特征与新概念联系起来。他们还将研究由于认知表征中的范畴结构，习得的联想如何延伸到知觉输入之外。在目标2中，该团队将研究如何根据图像统计数据自动估计颜色与概念的关联。他们将构建和评估新的模型，这些模型结合了基于视觉输入和颜色类别认知表征的预测因素。在构建这些模型的过程中，该团队将开发新的方法来量化连续空间中的分级类别成员资格。通过对人类判断进行建模，有可能对这些判断是如何做出的发展出新的见解。在目标3中，该团队将调查人们如何使用颜色概念关联来解释可视化，这一过程被称为分配推理。赋值推理使人们能够推断可视化中颜色和概念之间的最佳映射，但人们对这一过程的工作原理知之甚少。建议使用决策研究中通常使用的累加器模型来理解它，这将在决策领域和信息可视化领域之间建立新的联系。这些结果将增加人们如何整合来自多个相互冲突的来源的信息的知识，并提供关于如何优化设计语义可解释的可视化的新知识。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to advance the understanding of visual reasoning for visual communication. Visual reasoning enables people to translate visual input to abstract concepts. For example, to interpret which counties will receive more snowfall using a weather map, it is necessary to figure out which colors on the weather map indicate which amounts of snowfall. People have expectations about how visual features should map to concepts in visualizations, and it is harder for them to interpret visualizations that violate those expectations, even if mappings are clearly labeled. However, the nature of those expectations and their role in visual reasoning is not well-understood, so the design of information visualizations is often unprincipled and ad-hoc. With a better understanding of how visual reasoning works, it will be possible to design visualizations that fit its strengths and optimize visual communication. The investigators will address this problem by studying how people infer meaning from color in visualizations. This research can be translated to producing online tools for designing visualizations, which will improve STEM education and increase public literacy and engagement with science and technology. The education plan will use visual communication to make science more accessible and engaging through virtual reality (VR) and accompanying hands-on experiences with color and visualization, for both college undergraduates and middle-school students. The investigators will also support broadening participation of females in STEM through mentoring among the PI, graduate student, undergraduate intern, and middle school girls. The proposed research will address fundamental questions about how visual reasoning enables visual communication. In Objective 1, the investigators will study how people learn to associate perceptual features with novel concepts through environmental statistics. They will also examine how learned associations extend beyond perceptual input due to categorical structure in cognitive representations. In Objective 2, the team will study how to automatically estimate color–concept associations from image statistics. They will construct and evaluate new models that incorporate predictors based on visual input and cognitive representations of color categories. In the process of constructing these models, the team will develop new methods for quantifying graded category membership in a continuous space. By modeling human judgments, there is potential to develop new insights into how those judgments are made. In Objective 3, the team will investigate how people use color–concept associations to interpret visualizations, in a process called assignment inference. Assignment inference enables people to infer optimal mappings between colors and concepts in visualizations, but little is known about how this process works. It is proposed that it can be understood using accumulator models typically used in decision making research, which will forge new connections between the fields of decision making and information visualization. The results will increase knowledge of how people integrate information from multiple, conflicting sources and provide new knowledge about how to optimally design semantically interpretable visualizations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.