CAREER: Computational Model of Perceived Color and Appearance in Augmented Reality

增强现实(AR)允许观看者可视化混合到他们的真实世界环境中的数字、虚拟对象。与虚拟现实不同，虚拟现实显示虚拟内容，但遮挡了现实世界的视线，AR实现了混合，例如远程人员与其他人坐在真实会议室中的虚拟表示。AR应用的其他例子包括：(I)理科学生在他们的电路中可视化看不见的电流通过电线流动，或者在他们的真实世界环境中可视化看不见的力，如磁场或大气电流；(Ii)医生在考试或腹腔镜手术中叠加内部器官的X光图像。该项目旨在回答一个复杂而重要的问题：人类视觉系统如何感知虚拟AR内容和现实世界的混合？研究人员将利用测试视觉反应的实验数据，建立AR系统视觉特性的感知模型，包括颜色、亮度、深度和图形质量。该模型将模拟对不同照明环境的视觉适应，如室内和室外，以及对反射和与环境的其他交互的补偿。该项目将改进AR系统设计，包括对环境变化做出反应并预测用户视觉感知的响应算法。它还将导致对人类视觉系统的更好的科学理解，这有可能改善除AR之外的许多其他视觉界面。将该项目与教育相结合，研究人员将为大学的科学、数学和艺术课程开发和测试AR学习模块，从而改善学习。通过提高对AR中视觉感知的理解，该项目将有助于实现视觉上的准确、更舒适和更灵敏的AR显示系统。这个项目的目的是在AR系统中建立一个稳健的视觉外观计算模型，该模型考虑了视觉适应、认知解释以及虚拟显示内容与环境中真实对象和光照之间的光学相互作用。视觉实验将使用心理物理比例、通过调整的颜色匹配和恒定刺激任务来了解AR虚拟前景和现实世界背景的亮度、颜色、对比度、复杂性和深度的影响。透明AR环境中的颜色和材料外观模型将建立在这样的工作假设之上，即感知的颜色是前景和背景的非物理相加，其权重取决于层的认知折扣。其他实验将测量在时间变化的AR观看环境中的视觉亮度和颜色适应，例如，要求观察者在不同的照明情况和不同的适应时间将显示的AR和真实世界的刺激调整为非彩色。物理照明环境，包括光源和明亮物体的位置、强度和颜色，将使用相机和颜色传感器作为响应显示算法的输入，利用开发的AR中的视觉适应和外观模型进行感知。响应性算法将确保稳健、可预测的颜色外观和显示。在项目期间，将开发几个包含研究成果的AR教育应用程序进行验证和测试。学生和教师的课堂评估将对这些应用程序进行评估，以验证研究结果。AR将在研究组的颜色科学研究生课程中被采用，作为演示适应和环绕效果的工具，并作为练习心理物理方法的环境，与正在进行的研究相关的实验。研究人员将通过出版物和专业组织分享他们的计算模型和实验结果，目标是在AR系统设计和AR应用中有意义的实施。此外，AR的影响和色彩科学的迷人主题将通过大学的开放参观和招收学生和教师与社区分享。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Augmented reality (AR) allows a viewer to visualize digital, virtual objects mixed into their real-world environment. Different from virtual reality, which displays virtual content but blocks out the view of the real world, AR enables mixtures such as a virtual representation of a remote person sitting in a real conference room with other people. Other examples of AR applications include: (i) science students visualizing the invisible flow of electricity moving through wires in their circuits or visualize unseen forces such as magnetism or atmospheric currents in their real-world environment; (ii) doctors overlaying an X-ray-like view of internal organs during exams or laparoscopic surgery. This project aims to answer a complex and important question: how does human visual system perceive the mix of virtual AR content and the real world? The researchers will build a model of perception of the visual characteristics of AR systems, including color, brightness, depth, and graphics quality, with data from experiments that test visual responses. The model will mimic visual adaptation to different lighting environments, such as indoors versus outdoors, as well as compensations for reflections and other interactions with the environment. The project will result in improved AR system designs, including responsive algorithms that react to changes in the environment and anticipate the visual perception of the user. It will also result in a better scientific understanding of the human visual system, which has the potential to improve many other visual interfaces in addition to AR. Integrating the project with education, the researchers will develop and test AR learning modules for university courses in science, math, and art, leading to improved learning. By improving the understanding of visual perception in AR, this project will help enable visually accurate, more comfortable, and more responsive AR display systems. These have the potential to enhance the visual sense in applications such as education, medicine, and transportation.This project aims to build a robust computational model of visual appearance in AR systems that takes into account visual adaptation, cognitive interpretations, and the optical interaction between virtually displayed content and real objects and illumination in the environment. Visual experiments will employ psychophysical scaling, color matching via adjustment, and constant stimuli tasks to understand the influence of the luminance, color, contrast, complexity, and depth of both the AR virtual foreground and the real-world background. A model of color and material appearance in transparent AR environments will build on the working hypothesis that the perceived color is a non-physical addition of foreground and background whose weightings depend on cognitive discounting of the layers. Additional experiments will measure visual luminance and chromatic adaptation in temporally changing AR viewing environments, for instance, asking observers to adjust displayed AR and real-world stimuli to be achromatic in different lighting situations and at different adaptation times. The physical lighting environment, including the location, intensity, and color of light sources and bright objects, will be sensed with cameras and color sensors as input to responsive display algorithms utilizing the developed model of visual adaptation and appearance in AR. The responsive algorithms will ensure robust, predictable color appearance and display. During the project, several AR education applications incorporating the research results will be developed for validation and testing. Classroom assessments by students and faculty will evaluate these applications for validating the research results. AR will be adopted in the research group’s graduate color science courses as a tool for demonstrating adaptation and surround effects and as an environment for practicing psychophysical methods with experiments related to the ongoing research. The researchers will share their computational model and experimental findings via publication and professional organizations, with the goal of meaningful implementation both in AR system design and AR applications. Further, the impact of AR and the fascinating topic of color science will be shared with the community through the University’s open houses and the recruiting of students and faculty.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.