CGV: Small: Toward Objective, In-Situ, and Generalizable Evaluation of Visual Analytics by Integrating Brain Imaging with Cognitive Factors Analysis

点击翻译按钮获取中文摘要

Evaluating complex, interactive visual analytics systems is challenging for many reasons. The exploratory nature of using visualization makes quantitative measurements of individual components of the visualization task infeasible. The range of potential users and their differing goals render standardized metrics too restrictive. And environmental conditions can influence experimental outcomes, making it difficult to compare and generalize the results of separate evaluations. Although numerous methods have been proposed for evaluating visualizations and HCI, few can be readily applied to objectively evaluate visual analytics systems in real-world settings. In this project the PI will address this open challenge by developing a method to evaluate complex, interactive visual analytics systems using objective measures that can be performed in-situ to yield reproducible, generalizable results. His approach is to integrate noninvasive brain imaging using functional near-infrared spectroscopy (fNIRS) and cognitive factors measurements. The PI argues this will allow him to address issues in visual analytics evaluation by explaining the user's cognitive processes at a deeper level. Lightweight, noninvasive brain imaging techniques such as fNIRS have become more mature and reliable in recent years. fNIRS is easy to set up, robust to movement, and has been demonstrated in studies to be effective in determining a user's cognitive load, preferences, and perception when using a visualization. Cognitive factors such as locus of control, spatial visualization ability, and perceptual speed have recently been shown to correlate with a user's ability to interact with a visualization, and can be generalized to predict the behavioral patterns of users with different cognitive profiles. Both approaches aim to better understand the user's cognitive state and abilities. In this research cognitive factors measurements will provide low-level, baseline information about the user which is stable and unchanging ("traits"), while fNIRS provides immediate, real-time feedback on the user's current cognitive state when interacting with a visualization ("states"). In practice, without accounting for the individual user's traits, it is difficult to generalize the signals provided by fNIRS (for example, right- and left-handed subjects produce very different brain signals). By combining information on both "traits" and "states," evaluation results can generalize to a larger population based on cognitive profiles. The PI and his team have conducted two preliminary experiments that demonstrate the feasibility of the approach. They have replicated the classic experiment by Cleveland and McGill using fNIRS, and successfully distinguished participants' brain signals when using bar charts versus pie charts. In another experiment, they successfully correlated participants' locus of control with their ability to use hierarchical visualizations with different visual metaphors.Broader Impacts: The results of this work will have both immediate and long term impact on the field of visualization. In the short term, project outcomes will provide a robust and reliable evaluation mechanism for measuring the effectiveness of complex, interactive visual analytics systems. Ultimately, the findings of this research will help open the human-cognition "black box" and illuminate how interacting with a visualization helps a user gain insight. Such an understanding may in turn lead to the realization of insight-based evaluation and the emergence of a new visualization theory that is based on human cognitive processing. Finally, by integrating real-time fNIRS into visual analytics systems, effective adaptive mixed-initiative visual analytics systems can become one step closer to reality.