CGV: Small: Toward Objective, In-Situ, and Generalizable Evaluation of Visual Analytics by Integrating Brain Imaging with Cognitive Factors Analysis

由于许多原因，评估复杂的交互式可视化分析系统是具有挑战性的。使用可视化的探索性使得可视化任务的各个组成部分的定量测量不可行。潜在用户的范围和他们不同的目标使得标准化的指标过于受限。并且环境条件会影响实验结果，使得比较和概括单独评估的结果变得困难。尽管已经提出了许多评估可视化和HCI的方法，但很少有方法可以很容易地应用于客观地评估现实环境中的可视化分析系统。在这个项目中，PI将通过开发一种方法来评估复杂的交互式可视化分析系统，该系统使用客观的测量方法，可以在现场执行，以产生可重复的、可推广的结果，从而解决这一公开的挑战。他的方法是利用功能性近红外光谱（fNIRS）和认知因素测量整合无创脑成像。PI认为，这将允许他通过在更深层次上解释用户的认知过程来解决视觉分析评估中的问题。近年来，fNIRS等轻量、无创的脑成像技术已经变得更加成熟和可靠。fNIRS易于设置，对运动具有很强的稳健性，并且在研究中被证明在使用可视化时有效地确定用户的认知负荷、偏好和感知。认知因素，如控制点、空间可视化能力和感知速度，最近被证明与用户与可视化交互的能力相关，并且可以推广到预测具有不同认知概况的用户的行为模式。这两种方法都旨在更好地了解用户的认知状态和能力。在这项研究中，认知因素测量将提供关于用户稳定不变的底层基线信息（“特征”），而fNIRS在与可视化交互时提供用户当前认知状态的即时实时反馈（“状态”）。在实践中，如果不考虑个人用户的特征，就很难概括fNIRS提供的信号（例如，右撇子和左撇子受试者产生非常不同的大脑信号）。通过结合“特征”和“状态”的信息，评估结果可以基于认知概况推广到更大的人群。PI和他的团队已经进行了两个初步实验来证明这种方法的可行性。他们用近红外光谱（fNIRS）复制了克利夫兰和麦吉尔的经典实验，并成功区分了使用柱状图和饼状图时参与者的大脑信号。在另一个实验中，他们成功地将参与者的控制点与他们使用不同视觉隐喻的分层视觉化能力联系起来。更广泛的影响：这项工作的结果将对可视化领域产生直接和长期的影响。在短期内，项目结果将为测量复杂的、交互式的可视化分析系统的有效性提供一个健壮和可靠的评估机制。最终，这项研究的发现将有助于打开人类认知的“黑盒子”，并阐明与可视化交互如何帮助用户获得洞察力。这样的理解可能反过来导致实现基于洞察力的评价和基于人类认知加工的新的可视化理论的出现。最后，通过将实时fNIRS集成到视觉分析系统中，有效的自适应混合主动视觉分析系统可以离现实更近一步。

Evaluating complex, interactive visual analytics systems is challenging for many reasons. The exploratory nature of using visualization makes quantitative measurements of individual components of the visualization task infeasible. The range of potential users and their differing goals render standardized metrics too restrictive. And environmental conditions can influence experimental outcomes, making it difficult to compare and generalize the results of separate evaluations. Although numerous methods have been proposed for evaluating visualizations and HCI, few can be readily applied to objectively evaluate visual analytics systems in real-world settings. In this project the PI will address this open challenge by developing a method to evaluate complex, interactive visual analytics systems using objective measures that can be performed in-situ to yield reproducible, generalizable results. His approach is to integrate noninvasive brain imaging using functional near-infrared spectroscopy (fNIRS) and cognitive factors measurements. The PI argues this will allow him to address issues in visual analytics evaluation by explaining the user's cognitive processes at a deeper level. Lightweight, noninvasive brain imaging techniques such as fNIRS have become more mature and reliable in recent years. fNIRS is easy to set up, robust to movement, and has been demonstrated in studies to be effective in determining a user's cognitive load, preferences, and perception when using a visualization. Cognitive factors such as locus of control, spatial visualization ability, and perceptual speed have recently been shown to correlate with a user's ability to interact with a visualization, and can be generalized to predict the behavioral patterns of users with different cognitive profiles. Both approaches aim to better understand the user's cognitive state and abilities. In this research cognitive factors measurements will provide low-level, baseline information about the user which is stable and unchanging ("traits"), while fNIRS provides immediate, real-time feedback on the user's current cognitive state when interacting with a visualization ("states"). In practice, without accounting for the individual user's traits, it is difficult to generalize the signals provided by fNIRS (for example, right- and left-handed subjects produce very different brain signals). By combining information on both "traits" and "states," evaluation results can generalize to a larger population based on cognitive profiles. The PI and his team have conducted two preliminary experiments that demonstrate the feasibility of the approach. They have replicated the classic experiment by Cleveland and McGill using fNIRS, and successfully distinguished participants' brain signals when using bar charts versus pie charts. In another experiment, they successfully correlated participants' locus of control with their ability to use hierarchical visualizations with different visual metaphors.Broader Impacts: The results of this work will have both immediate and long term impact on the field of visualization. In the short term, project outcomes will provide a robust and reliable evaluation mechanism for measuring the effectiveness of complex, interactive visual analytics systems. Ultimately, the findings of this research will help open the human-cognition "black box" and illuminate how interacting with a visualization helps a user gain insight. Such an understanding may in turn lead to the realization of insight-based evaluation and the emergence of a new visualization theory that is based on human cognitive processing. Finally, by integrating real-time fNIRS into visual analytics systems, effective adaptive mixed-initiative visual analytics systems can become one step closer to reality.