CAREER: Computational Model of Perceived Color and Appearance in Augmented Reality

增强现实（AR）允许观看者将数字虚拟对象可视化到其真实世界环境中。与显示虚拟内容但阻挡真实的世界的视图的虚拟现实不同，AR实现了诸如远程人与其他人坐在真实的会议室中的虚拟表示的混合。AR应用的其他示例包括：（i）科学学生可视化通过电路中的电线移动的无形电流，或可视化现实世界环境中的磁性或大气电流等看不见的力量;（ii）医生在检查或腹腔镜手术期间覆盖内部器官的X射线视图。该项目旨在回答一个复杂而重要的问题：人类视觉系统如何感知虚拟AR内容和真实的世界的混合？研究人员将利用测试视觉反应的实验数据，建立一个AR系统视觉特征感知模型，包括颜色、亮度、深度和图形质量。该模型将模拟对不同照明环境的视觉适应，例如室内与室外，以及对反射和与环境的其他相互作用的补偿。该项目将改进AR系统设计，包括响应算法，对环境变化做出反应，并预测用户的视觉感知。它还将使人们对人类视觉系统有更好的科学了解，这有可能改善除AR之外的许多其他视觉界面。将该项目与教育相结合，研究人员将为大学的科学、数学和艺术课程开发和测试AR学习模块，从而改善学习效果。通过提高对AR视觉感知的理解，该项目将有助于实现视觉上更准确、更舒适、反应更灵敏的AR显示系统。该项目旨在构建AR系统中视觉外观的鲁棒计算模型，该模型考虑了视觉适应、认知解释以及虚拟显示内容与环境中真实的对象和照明之间的光学相互作用。视觉实验将采用心理物理缩放，通过调整进行颜色匹配，以及恒定刺激任务，以了解AR虚拟前景和现实世界背景的亮度，颜色，对比度，复杂性和深度的影响。透明AR环境中的颜色和材料外观的模型将建立在工作假设上，即感知的颜色是前景和背景的非物理添加，其权重取决于层的认知折扣。其他实验将测量在时间变化的AR观看环境中的视觉亮度和色彩适应，例如，要求观察者在不同的照明情况下和不同的适应时间将显示的AR和真实世界刺激调整为非彩色。物理照明环境，包括光源和明亮物体的位置，强度和颜色，将通过相机和颜色传感器进行感测，作为响应显示算法的输入，利用AR中的视觉适应和外观的开发模型。响应式算法将确保稳健、可预测的颜色外观和显示。在项目期间，将开发几个结合研究结果的AR教育应用程序进行验证和测试。学生和教师的课堂评估将评估这些应用程序，以验证研究结果。AR将在研究小组的研究生色彩科学课程中被采用，作为展示适应和环绕效果的工具，并作为与正在进行的研究相关的实验的心理物理方法的实践环境。研究人员将通过出版物和专业组织分享他们的计算模型和实验结果，目标是在AR系统设计和AR应用中实现有意义的实现。此外，AR的影响和色彩科学的迷人主题将通过大学的开放日和学生和教师的招聘与社区分享。这个奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Augmented reality (AR) allows a viewer to visualize digital, virtual objects mixed into their real-world environment. Different from virtual reality, which displays virtual content but blocks out the view of the real world, AR enables mixtures such as a virtual representation of a remote person sitting in a real conference room with other people. Other examples of AR applications include: (i) science students visualizing the invisible flow of electricity moving through wires in their circuits or visualize unseen forces such as magnetism or atmospheric currents in their real-world environment; (ii) doctors overlaying an X-ray-like view of internal organs during exams or laparoscopic surgery. This project aims to answer a complex and important question: how does human visual system perceive the mix of virtual AR content and the real world? The researchers will build a model of perception of the visual characteristics of AR systems, including color, brightness, depth, and graphics quality, with data from experiments that test visual responses. The model will mimic visual adaptation to different lighting environments, such as indoors versus outdoors, as well as compensations for reflections and other interactions with the environment. The project will result in improved AR system designs, including responsive algorithms that react to changes in the environment and anticipate the visual perception of the user. It will also result in a better scientific understanding of the human visual system, which has the potential to improve many other visual interfaces in addition to AR. Integrating the project with education, the researchers will develop and test AR learning modules for university courses in science, math, and art, leading to improved learning. By improving the understanding of visual perception in AR, this project will help enable visually accurate, more comfortable, and more responsive AR display systems. These have the potential to enhance the visual sense in applications such as education, medicine, and transportation.This project aims to build a robust computational model of visual appearance in AR systems that takes into account visual adaptation, cognitive interpretations, and the optical interaction between virtually displayed content and real objects and illumination in the environment. Visual experiments will employ psychophysical scaling, color matching via adjustment, and constant stimuli tasks to understand the influence of the luminance, color, contrast, complexity, and depth of both the AR virtual foreground and the real-world background. A model of color and material appearance in transparent AR environments will build on the working hypothesis that the perceived color is a non-physical addition of foreground and background whose weightings depend on cognitive discounting of the layers. Additional experiments will measure visual luminance and chromatic adaptation in temporally changing AR viewing environments, for instance, asking observers to adjust displayed AR and real-world stimuli to be achromatic in different lighting situations and at different adaptation times. The physical lighting environment, including the location, intensity, and color of light sources and bright objects, will be sensed with cameras and color sensors as input to responsive display algorithms utilizing the developed model of visual adaptation and appearance in AR. The responsive algorithms will ensure robust, predictable color appearance and display. During the project, several AR education applications incorporating the research results will be developed for validation and testing. Classroom assessments by students and faculty will evaluate these applications for validating the research results. AR will be adopted in the research group’s graduate color science courses as a tool for demonstrating adaptation and surround effects and as an environment for practicing psychophysical methods with experiments related to the ongoing research. The researchers will share their computational model and experimental findings via publication and professional organizations, with the goal of meaningful implementation both in AR system design and AR applications. Further, the impact of AR and the fascinating topic of color science will be shared with the community through the University’s open houses and the recruiting of students and faculty.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.