Collaborative Research: HCC: Medium: Deep Learning-Based Tracking of Eyes and Lens Shape from Purkinje Images for Holographic Augmented Reality Glasses

合作研究：HCC：媒介：基于深度学习的浦肯野图像眼睛和晶状体形状跟踪，用于全息增强现实眼镜

• 批准号: 2107049
• 负责人: Jorge Otero-Millan
• 金额: $ 22.5万
• 依托单位: University of California-Berkeley
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2107049&HistoricalAwards=false
• 关键词: Collaborative; Research; HCC; Medium; Deep

This project seeks to develop head-worn Augmented Reality (AR) systems that look and feel like ordinary prescription eyeglasses, and can be worn comfortably all day, with a field of view that matches the wide field of view of today's eyewear. Such future AR glasses will enable vast new capabilities for individuals and groups, integrating computer assistance as 3D enhancements within the user’s surroundings. For example, wearing such AR glasses, an individual will see around them remote individuals as naturally as they now see and interact with nearby real individuals. Virtual personal assistants such as Alexa and Siri may become 3D-embodied within these AR glasses and situationally aware, guiding the wearer around a new airport, or coaching the user in customized physical exercise. This project aims to advance two crucial, synergistic parts of such AR glasses: 1) the see-through display itself and 2) the 3D eye-tracking subsystem. The see-through display needs to be both very compact and have a wide field of view. To achieve these display requirements, the project uses true holographic image generation, and improves the algorithms that generate these holograms by a) concentrating higher image quality in the direction and distance of the user's current gaze, and b) algorithmically steering the "eye box" (the precise location where the eye needs to be to observe the image) to the current location of the eye's pupil opening. In current holographic displays, this viewing eye box is typically less than 1 cubic millimeter, far too small for a practical head-worn system. Therefore, a practical system may need both a precise eye tracking system that locates the pupil opening and a display system that algorithmically steers the holographic image to be viewable at that precise location. The 3D eye tracking system also seeks to determine the direction of the user's gaze, and the distance of the point of gaze from the eye (whether near or far), so that the display system can optimize the generated holographic image for the precise focus of attention. The proposed AR display can render images at variable focal lengths, so it could be used for people with visual accommodation issues, thereby allowing them to participate in AR-supported education and training programs. The device could also have other possible uses in medical (such as better understanding of the human visual system) and training fields. The two branches of this project, the holographic display, and the 3D eye tracker, are closely linked and each improved by the other. The 3D eye tracker utilizes an enriched set of signals and sensors (multiple cameras for each eye, and a multiplicity of infra-red (IR) LEDs), from which the system extracts the multiple tracking parameters in real time: the horizontal and vertical gaze angles, the distance accommodation, and the 3D position and size of the pupil's opening. The distance accommodation is extracted by analyzing Purkinje reflections of the IR LEDs from the multiple layers in the eye's cornea and lens. A neural network extracts the aforementioned 3D tracking results from the multiple sensors after being trained on a large body of ground truth data. The training data is generated from multiple human subjects who are exposed, instantaneously to known patterns on external displays at a range of distances and angles from the eye. Simultaneous to these instantaneous patterns, the subject is also shown images from the near-eye holographic image generator whose eye box location and size have been previously optically calibrated. One part of each pattern will be shown, instantaneously, on an external display and the other part, at the same instant, on the holographic display. The subject can only answer correctly a challenge question if they have observed both displays simultaneously. This can only occur if the eye is at a precise 3D location and also at a precise known gaze angle. The eye tracker will be further improved by integrated its training and calibration with the high precision (but very bulky) BinoScopic tracker at UC Berkeley, which tracks using precise maps of the user's retina. The holograhic image generator uses the real time data from the 3D eye tracker to generate holograms whose highest image quality is at the part of image that is currently on the viewer's fovea, and at the distance to which the user is currently accommodated. The image quality is improved by a trained neural network whose inputs are images from a camera placed, during training, at the position of the viewer's eye.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目旨在开发看起来和感觉上都像普通处方眼镜的头戴式增强现实(AR)系统，并且可以整天舒适地佩戴，视野与当今眼镜的宽广视野相匹配。这种未来的AR眼镜将为个人和团体提供巨大的新功能，将计算机辅助作为3D增强功能集成到用户周围。例如，戴上这样的AR眼镜，个人将看到他们周围的遥远个人，就像他们现在看到的一样，并与附近的真实个人互动。Alexa和Siri等虚拟个人助理可以在这些AR眼镜中实现3D效果，并根据情况感知，引导佩戴者参观新机场，或指导用户进行定制的体育锻炼。该项目旨在推进这类AR眼镜的两个关键、协同部分：1)透明显示器本身和2)3D眼睛跟踪子系统。透视式显示器需要非常紧凑，并且有很宽的视野。为了实现这些显示要求，该项目使用真正的全息图像生成，并通过a)在用户当前凝视的方向和距离上集中更高的图像质量，以及b)算法地将“眼框”(眼睛需要观察图像的精确位置)引导到眼睛的瞳孔张开的当前位置来改进生成这些全息图的算法。在目前的全息显示器中，这种观察眼框通常不到1立方毫米，对于实用的头戴式系统来说太小了。因此，一个实用的系统可能既需要定位瞳孔开口的精确眼睛跟踪系统，也需要通过算法引导全息图像在该精确位置可见的显示系统。3D眼睛跟踪系统还试图确定用户的凝视方向，以及凝视点与眼睛的距离(无论是近还是远)，以便显示系统可以针对精确的注意力焦点优化生成的全息图像。建议的AR显示器可以在可变焦距下呈现图像，因此它可以用于有视觉调节问题的人，从而允许他们参与AR支持的教育和培训计划。该设备还可能在医学(例如更好地了解人类视觉系统)和培训领域有其他可能的用途。这个项目的两个分支--全息显示和3D眼球跟踪器--紧密相连，相互改进。3D眼睛跟踪器利用了丰富的信号和传感器集(每只眼睛有多个摄像头，以及多个红外线(IR)LED)，系统从这些传感器中实时提取多个跟踪参数：水平和垂直凝视角度、距离调节以及瞳孔开口的3D位置和大小。通过分析眼睛角膜和晶状体中多层红外LED的浦肯野反射来提取距离调节。神经网络在对大量地面真实数据进行训练后，从多个传感器中提取上述3D跟踪结果。训练数据是从多个人类受试者产生的，这些受试者在与眼睛的距离和角度范围内瞬间暴露在外部显示器上的已知图案中。在这些瞬时图案的同时，还向受试者展示了来自近眼全息图像生成器的图像，其眼盒位置和大小先前已经被光学校准。每种图案的一部分将瞬间显示在外部显示器上，而另一部分将同时显示在全息显示器上。只有当受试者同时观察到两个显示器时，他们才能正确回答挑战问题。只有当眼睛处于精确的3D位置，并且处于精确的已知凝视角度时，才会发生这种情况。通过将眼睛跟踪器的训练和校准与加州大学伯克利分校的高精度(但非常笨重)BinoScope跟踪器相结合，眼睛跟踪器将得到进一步改进。BinoScope跟踪器使用用户视网膜的精确地图进行跟踪。全息图像生成器使用来自3D眼睛跟踪器的实时数据来生成全息图，该全息图的最高图像质量位于当前在观察者的中心凹上的图像部分以及在用户当前适应的距离处。图像质量由经过训练的神经网络改进，其输入是在训练期间放置在观看者眼睛位置的摄像机的图像。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。