Establishing Usability Evaluation Method and Developing the Evaluation System based on Multimodal Information

本研究探讨了多模态数据在可用性评估中的应用，包括主观、生理、眼动追踪和物理数据，旨在改进现有的评估技术。我们主要有三个猜想：猜想A：生理测量可以用来客观地衡量用户体验。猜想B：结合眼动和物理数据可以更深入地了解用户行为，从而有助于评估任务表现，了解个体差异。猜想C：对经验的生理测量将与对经验的主观评价相对应，但它们并不总是一致的。我们提出了一系列的三个实验，通过将生理、物理和眼动追踪数据纳入传统方法来改进当前的HCI评估技术。研究结果表明，这些多模态数据可以有力地改进评估技术：(1)生理测量可以在定量和更客观地评估用户体验方面发挥作用，特别是在涉及主观评分和视频分析时。我们发现有证据表明，在游戏环境中，不同的压力水平会产生不同的生理反应。这些结果为猜想a提供了支持。(2)我们认为眼动是认知过程的反映，而肢体动作是认知过程的结果；因此，结合眼和手的运动可以提供更多关于用户行为和交互过程的信息。(3)研究了传统可用性指标与新可用性数据源（眼动追踪和生理数据）之间的关系。生理数据可以反映在评估用户体验的主观报告中。这些结果强调主观评价技术和生理评价技术应该同时使用，因为它们可能提供不同的用户体验信息。这些结果为我们的C. Less猜想提供了支持

This research explores the use of the multimodal data in evaluating usability, including subjective, physiological, eye tracking and physical data, with the aim of improving current evaluation techniques. We have three main conjectures: Conjecture A: Physiological measures can be used to objectively measure user experience. Conjecture B: Combining eye movement and physical data can get closer insights into user behavior, thereby helping to evaluate task performance and understand individual difference. Conjecture C: Physiological measures of experience will correspond to subjective ratings of the experience, but they will not be always consistent.We have presented a series of three experiments to improve current HCI evaluation techniques by incorporating physiological, physical and eye tracking data into traditional methods. The study results showed these multimodal data can be powerful for improving evaluation techniques:(1) Physiological measures can be powerful for quantitatively an … More d objectively evaluating user experience, especially when used in concern with subjective ratings and video analysis. We find evidence that there are different physiological responses across different stress levels in a game environment. These results provide support for Conjecture A.(2) We argue that that eye movements are a reflection of cognitive process, while physical actions are the result of such process ; thus, combining eye and hand movements can provide more information about user behavior and interaction process. These results support the Conjecture B.(3) We also investigate the relationships between traditional usability index and new usability data sources (eye tracking and physiological data). Physiological data can be mirrored in the subjective reports of assessing user experience. These results emphasize that both subjective and physiological evaluation techniques should be used together because they may provide different information of user experience. These results provide support for our Conjecture C. Less