Collaborative Research: HCC: Medium: Deep Learning-Based Tracking of Eyes and Lens Shape from Purkinje Images for Holographic Augmented Reality Glasses

合作研究：HCC：媒介：基于深度学习的浦肯野图像眼睛和晶状体形状跟踪，用于全息增强现实眼镜

• 批准号: 2107454
• 负责人: Henry Fuchs
• 金额: $ 97.5万
• 依托单位: University of North Carolina at Chapel Hill
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2107454&HistoricalAwards=false
• 关键词: Collaborative; Research; HCC; Medium; Deep

This project seeks to develop head-worn Augmented Reality (AR) systems that look and feel like ordinary prescription eyeglasses, and can be worn comfortably all day, with a field of view that matches the wide field of view of today's eyewear. Such future AR glasses will enable vast new capabilities for individuals and groups, integrating computer assistance as 3D enhancements within the user’s surroundings. For example, wearing such AR glasses, an individual will see around them remote individuals as naturally as they now see and interact with nearby real individuals. Virtual personal assistants such as Alexa and Siri may become 3D-embodied within these AR glasses and situationally aware, guiding the wearer around a new airport, or coaching the user in customized physical exercise. This project aims to advance two crucial, synergistic parts of such AR glasses: 1) the see-through display itself and 2) the 3D eye-tracking subsystem. The see-through display needs to be both very compact and have a wide field of view. To achieve these display requirements, the project uses true holographic image generation, and improves the algorithms that generate these holograms by a) concentrating higher image quality in the direction and distance of the user's current gaze, and b) algorithmically steering the "eye box" (the precise location where the eye needs to be to observe the image) to the current location of the eye's pupil opening. In current holographic displays, this viewing eye box is typically less than 1 cubic millimeter, far too small for a practical head-worn system. Therefore, a practical system may need both a precise eye tracking system that locates the pupil opening and a display system that algorithmically steers the holographic image to be viewable at that precise location. The 3D eye tracking system also seeks to determine the direction of the user's gaze, and the distance of the point of gaze from the eye (whether near or far), so that the display system can optimize the generated holographic image for the precise focus of attention. The proposed AR display can render images at variable focal lengths, so it could be used for people with visual accommodation issues, thereby allowing them to participate in AR-supported education and training programs. The device could also have other possible uses in medical (such as better understanding of the human visual system) and training fields. The two branches of this project, the holographic display, and the 3D eye tracker, are closely linked and each improved by the other. The 3D eye tracker utilizes an enriched set of signals and sensors (multiple cameras for each eye, and a multiplicity of infra-red (IR) LEDs), from which the system extracts the multiple tracking parameters in real time: the horizontal and vertical gaze angles, the distance accommodation, and the 3D position and size of the pupil's opening. The distance accommodation is extracted by analyzing Purkinje reflections of the IR LEDs from the multiple layers in the eye's cornea and lens. A neural network extracts the aforementioned 3D tracking results from the multiple sensors after being trained on a large body of ground truth data. The training data is generated from multiple human subjects who are exposed, instantaneously to known patterns on external displays at a range of distances and angles from the eye. Simultaneous to these instantaneous patterns, the subject is also shown images from the near-eye holographic image generator whose eye box location and size have been previously optically calibrated. One part of each pattern will be shown, instantaneously, on an external display and the other part, at the same instant, on the holographic display. The subject can only answer correctly a challenge question if they have observed both displays simultaneously. This can only occur if the eye is at a precise 3D location and also at a precise known gaze angle. The eye tracker will be further improved by integrated its training and calibration with the high precision (but very bulky) BinoScopic tracker at UC Berkeley, which tracks using precise maps of the user's retina. The holograhic image generator uses the real time data from the 3D eye tracker to generate holograms whose highest image quality is at the part of image that is currently on the viewer's fovea, and at the distance to which the user is currently accommodated. The image quality is improved by a trained neural network whose inputs are images from a camera placed, during training, at the position of the viewer's eye.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目旨在开发头戴式增强现实（AR）系统，其外观和感觉就像普通的处方眼镜，并且可以全天舒适地佩戴，其视野与当今眼镜的宽视野相匹配。这种未来的AR眼镜将为个人和团体提供大量新功能，将计算机辅助作为3D增强功能集成在用户周围环境中。例如，佩戴这种AR眼镜，个体将像他们现在看到的那样自然地看到他们周围的远程个体，并与附近的真实的个体进行交互。像Alexa和Siri这样的虚拟个人助理可以在这些AR眼镜中实现3D，并能够感知情况，引导佩戴者在新机场周围走动，或者指导用户进行定制的体育锻炼。该项目旨在推进这种AR眼镜的两个关键的协同部分：1）透视显示器本身和2）3D眼睛跟踪子系统。透视显示器需要非常紧凑并且具有宽视场。为了实现这些显示要求，该项目使用真正的全息图像生成，并通过以下方式改进生成这些全息图的算法：a）在用户当前注视的方向和距离上集中更高的图像质量，以及B）通过算法将“眼框”（眼睛需要观察图像的精确位置）转向眼睛瞳孔张开的当前位置。在当前的全息显示器中，该观看眼箱通常小于1立方毫米，对于实际的头戴式系统来说太小了。 因此，实际系统可能需要定位瞳孔开口的精确眼睛跟踪系统和在算法上操纵全息图像以在该精确位置处可见的显示系统。 3D眼睛跟踪系统还试图确定用户注视的方向以及注视点距眼睛的距离（无论是近还是远），使得显示系统可以优化所生成的全息图像以用于精确的注意力聚焦。拟议的AR显示器可以以可变焦距呈现图像，因此它可以用于有视觉调节问题的人，从而使他们能够参与AR支持的教育和培训计划。该设备还可以在医疗（例如更好地了解人类视觉系统）和培训领域中有其他可能的用途。该项目的两个分支，全息显示和3D眼动仪，紧密相连，彼此改进。3D眼睛跟踪器利用丰富的信号和传感器集合（每只眼睛的多个摄像机和多个红外（IR）LED），系统从中以真实的时间提取多个跟踪参数：水平和垂直注视角度、距离调节以及瞳孔开口的3D位置和大小。通过分析来自眼睛的角膜和透镜中的多个层的IR LED的浦肯野反射来提取距离调节。神经网络在对大量地面实况数据进行训练之后从多个传感器提取上述3D跟踪结果。训练数据是从多个人类受试者生成的，这些受试者在离眼睛一定距离和角度的范围内瞬时暴露于外部显示器上的已知图案。在这些瞬时图案的同时，还向受试者显示来自近眼全息图像生成器的图像，其眼箱位置和尺寸先前已经被光学校准。每个图案的一部分将在外部显示器上即时显示，而另一部分在同一时刻在全息显示器上显示。受试者只有在同时观察两个显示屏时才能正确回答质询问题。这只能在眼睛处于精确的3D位置并且还处于精确的已知注视角度时发生。 眼动仪将通过将其训练和校准与加州大学伯克利分校的高精度（但非常庞大）BinoScopic跟踪器相结合来进一步改进，该跟踪器使用用户视网膜的精确地图进行跟踪。全息图像生成器使用来自3D眼睛跟踪器的真实的时间数据来生成全息图，该全息图的最高图像质量是在当前在观看者的中央凹上的图像部分处以及在用户当前被容纳的距离处。图像质量通过一个训练过的神经网络来提高，该网络的输入是在训练过程中放置在观察者眼睛位置的摄像机的图像。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Geometry-Aware Eye Image-To-Image Translation

• DOI: 10.1145/3517031.3532524
• 发表时间: 2022-06
• 期刊: 2022 Symposium on Eye Tracking Research and Applications
• 作者: Conny Lu;Qian Zhang;K. Krishnakumar;Jixu Chen;H. Fuchs;S. Talathi;Kunlin Liu

Hogel-Free Holography

• DOI: 10.1145/3516428
• 发表时间: 2022-03
• 期刊: ACM Transactions on Graphics
• 影响因子: 6.2
• 作者: Praneeth Chakravarthula;Ethan Tseng;H. Fuchs;Felix Heide

Pupil-Aware Holography

• DOI: 10.1145/3550454.3555508
• 发表时间: 2022-03
• 期刊: ACM Transactions on Graphics (TOG)
• 作者: Praneeth Chakravarthula;Seung-Hwan Baek;Ethan Tseng;Andrew Maimone;Grace Kuo;Florian Schiffers;N. Matsuda;O. Cossairt;Douglas Lanman;Felix Heide

Joint neural phase retrieval and compression for energy- and computation-efficient holography on the edge

• DOI: 10.1145/3528223.3530070
• 发表时间: 2022-07
• 期刊: ACM Transactions on Graphics (TOG)
• 作者: Yujie Wang;Praneeth Chakravarthula;Qingyan Sun;Baoquan Chen

Stochastic Light Field Holography

• DOI: 10.1109/iccp56744.2023.10233716
• 发表时间: 2023-07
• 期刊: 2023 IEEE International Conference on Computational Photography (ICCP)
• 作者: Florian Schiffers;Praneeth Chakravarthula;N. Matsuda;Grace Kuo;Ethan Tseng;Douglas Lanman;Felix Heide-Felix-Hei