Collaborative Research: HCC: Medium: Deep Learning-Based Tracking of Eyes and Lens Shape from Purkinje Images for Holographic Augmented Reality Glasses

这个项目旨在开发头戴式增强现实（AR）系统，它的外观和感觉就像普通的处方眼镜，可以舒适地戴上一整天，视野与当今眼镜的宽视野相匹配。这种未来的AR眼镜将为个人和团体提供大量的新功能，将计算机辅助集成为用户周围环境中的3D增强功能。例如，戴上这种增强现实眼镜，一个人就会像现在一样自然地看到周围的远程个体，并与附近的真实个体互动。像Alexa和Siri这样的虚拟个人助理可能会在这些增强现实眼镜中实现3d化，并具有情境感知能力，引导佩戴者在新机场周围行走，或者指导用户进行定制的体育锻炼。该项目旨在推进这类AR眼镜的两个关键协同部分：1)透明显示器本身和2)3D眼球追踪子系统。透明显示器既要非常紧凑，又要有宽广的视野。为了实现这些显示要求，该项目使用了真正的全息图像生成，并改进了生成这些全息图的算法：a)将更高的图像质量集中在用户当前凝视的方向和距离上，b)通过算法将“眼盒”（眼睛需要观察图像的精确位置）转向眼睛瞳孔张开的当前位置。在目前的全息显示器中，这种观看眼盒通常小于1立方毫米，对于实际的头戴式系统来说太小了。因此，一个实用的系统可能既需要一个精确的眼动追踪系统来定位瞳孔开口，也需要一个显示系统来算法地引导全息图像在那个精确的位置上可见。3D眼动追踪系统还寻求确定用户凝视的方向，以及凝视点与眼睛的距离（无论是近还是远），以便显示系统可以优化生成的全息图像以精确关注焦点。拟议中的AR显示器可以以可变焦距呈现图像，因此它可以用于有视觉适应问题的人，从而使他们能够参与AR支持的教育和培训计划。该设备在医疗（如更好地了解人类视觉系统）和训练领域也有其他可能的用途。这个项目的两个分支，全息显示和3D眼动仪，紧密相连，相互完善。3D眼动仪利用了一组丰富的信号和传感器（每只眼睛有多个摄像头，以及多个红外（IR） led），系统从中实时提取多个跟踪参数：水平和垂直凝视角度、距离调节、瞳孔张开的3D位置和大小。通过分析红外led在眼睛角膜和晶状体的多层浦肯野反射来提取距离调节。神经网络在大量地面真实数据上进行训练后，从多个传感器中提取上述3D跟踪结果。训练数据是由多名受试者产生的，这些受试者在距离眼睛一定距离和角度的情况下，立即暴露在外部显示器上的已知模式下。在这些瞬时模式的同时，受试者还会看到来自近眼全息图像发生器的图像，这些图像发生器的眼箱位置和大小已经事先进行了光学校准。每个图案的一部分将立即显示在外部显示器上，而另一部分将同时显示在全息显示器上。受试者只有在同时观察到两种表现的情况下才能正确回答挑战问题。只有当眼睛处于一个精确的3D位置，并且处于一个精确的已知凝视角度时，这种情况才会发生。通过将眼动仪的训练和校准与加州大学伯克利分校（UC Berkeley）的高精度（但非常笨重）BinoScopic追踪器集成在一起，眼动仪将得到进一步改进，后者使用用户视网膜的精确地图进行追踪。全息图像生成器使用来自3D眼动仪的实时数据来生成全息图，其最高图像质量位于当前位于观看者中央凹的图像部分，以及用户当前所处的距离。图像质量通过训练后的神经网络得到改善，该神经网络的输入是在训练期间放置在观看者眼睛位置的摄像机的图像。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project seeks to develop head-worn Augmented Reality (AR) systems that look and feel like ordinary prescription eyeglasses, and can be worn comfortably all day, with a field of view that matches the wide field of view of today's eyewear. Such future AR glasses will enable vast new capabilities for individuals and groups, integrating computer assistance as 3D enhancements within the user’s surroundings. For example, wearing such AR glasses, an individual will see around them remote individuals as naturally as they now see and interact with nearby real individuals. Virtual personal assistants such as Alexa and Siri may become 3D-embodied within these AR glasses and situationally aware, guiding the wearer around a new airport, or coaching the user in customized physical exercise. This project aims to advance two crucial, synergistic parts of such AR glasses: 1) the see-through display itself and 2) the 3D eye-tracking subsystem. The see-through display needs to be both very compact and have a wide field of view. To achieve these display requirements, the project uses true holographic image generation, and improves the algorithms that generate these holograms by a) concentrating higher image quality in the direction and distance of the user's current gaze, and b) algorithmically steering the "eye box" (the precise location where the eye needs to be to observe the image) to the current location of the eye's pupil opening. In current holographic displays, this viewing eye box is typically less than 1 cubic millimeter, far too small for a practical head-worn system. Therefore, a practical system may need both a precise eye tracking system that locates the pupil opening and a display system that algorithmically steers the holographic image to be viewable at that precise location. The 3D eye tracking system also seeks to determine the direction of the user's gaze, and the distance of the point of gaze from the eye (whether near or far), so that the display system can optimize the generated holographic image for the precise focus of attention. The proposed AR display can render images at variable focal lengths, so it could be used for people with visual accommodation issues, thereby allowing them to participate in AR-supported education and training programs. The device could also have other possible uses in medical (such as better understanding of the human visual system) and training fields. The two branches of this project, the holographic display, and the 3D eye tracker, are closely linked and each improved by the other. The 3D eye tracker utilizes an enriched set of signals and sensors (multiple cameras for each eye, and a multiplicity of infra-red (IR) LEDs), from which the system extracts the multiple tracking parameters in real time: the horizontal and vertical gaze angles, the distance accommodation, and the 3D position and size of the pupil's opening. The distance accommodation is extracted by analyzing Purkinje reflections of the IR LEDs from the multiple layers in the eye's cornea and lens. A neural network extracts the aforementioned 3D tracking results from the multiple sensors after being trained on a large body of ground truth data. The training data is generated from multiple human subjects who are exposed, instantaneously to known patterns on external displays at a range of distances and angles from the eye. Simultaneous to these instantaneous patterns, the subject is also shown images from the near-eye holographic image generator whose eye box location and size have been previously optically calibrated. One part of each pattern will be shown, instantaneously, on an external display and the other part, at the same instant, on the holographic display. The subject can only answer correctly a challenge question if they have observed both displays simultaneously. This can only occur if the eye is at a precise 3D location and also at a precise known gaze angle. The eye tracker will be further improved by integrated its training and calibration with the high precision (but very bulky) BinoScopic tracker at UC Berkeley, which tracks using precise maps of the user's retina. The holograhic image generator uses the real time data from the 3D eye tracker to generate holograms whose highest image quality is at the part of image that is currently on the viewer's fovea, and at the distance to which the user is currently accommodated. The image quality is improved by a trained neural network whose inputs are images from a camera placed, during training, at the position of the viewer's eye.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.