CHS: Small: Minimal-Latency Tracking and Display for Head-Worn Augmented Reality Systems

增强现实技术（AR）可以将计算机图像叠加到用户对周围环境的看法上，最自然的方式是通过头戴式显示器。几十年来，许多AR应用一直受到笨重的头戴设备和不完善的显示器的阻碍。随着接近眼镜外形的新型商用紧凑型显示器设计的出现，这种情况有望很快改变。我们希望这些设备和其他设备能够重新激发人们对增强现实及其应用的科学和商业兴趣。然而，AR系统仍然有一个致命的缺陷，那就是对于人类的视觉来说，它们的速度太慢了。这种缓慢或延迟隐藏在每个子系统中，从跟踪摄像机的捕获到深度渲染管道，到渲染帧缓冲区，再到显示控制器中的重新格式化。这种延迟会导致合成图像和真实图像之间的错误注册。换句话说，当你转头时，本该叠加在现实世界上的图像会在不应该移动的时候开始移动，然后才会回到它应该停留的地方。这损害了增强在许多应用中的效用，特别是对于高精度用途，如飞机维护或手术。拟议的增强现实解决方案，结合新兴的舒适的眼镜式头戴式显示器，将使计算机生成的视觉增强技术在广泛的应用中受益：远程呈现、医疗检查和程序、维护和导航。许多目前使用传统显示器进行可视化的应用程序将能够使用头戴式显示器，并在用户看到的任何地方获得自然手眼协调和增强图像的好处。该项目将被整合到北卡罗来纳大学教堂山分校的多个课程中，这将刺激学生探索渲染、跟踪、图像获取和重建、增强现实和远程呈现等新方向。研究产品将向更广泛的研究和开发社区展示实时3D视觉和3D图形的新方法-扫描线流处理，从相机捕获到显示更新-这大大降低了延迟，从而提高了真实图像和增强图像之间的保真度。该项目将利用廉价相机的一个特点（连续扫描，或“滚动快门”），到目前为止，这一直被视为这些设备的一个重大弱点，并将证明这实际上是如何提供更频繁的场景信息更新的重要资产。该项目将用统一的基于扫描线的方法取代每个子系统的经典逐帧处理，从而显著减少延迟。用户跟踪的头部姿势将在每条扫描线上更新，就在每条扫描线从用户头戴式显示器上的一组摄像头传输过来之后。为了匹配这种跟踪性能，该项目将通过直接控制最快的显示技术——数字微镜显示器（DMD）的像素，在扫描线片段中呈现增强图像。所有这些操作最终都将在移动设备中完成，与用户的眼镜式显示器进行无线通信。这样的移动操作应该能够在用户的视野上进行精确的增强，以实现广泛有用的增强现实应用。

Augmented Reality (AR) enables computer images to be superimposed onto a user's view of his or her surroundings, most naturally via a head-worn display. For decades, many AR applications have been held back by bulky head-gear and inadequate displays. This situation is expected to change soon with the arrival of new commercially-available compact display designs that approach the form factor of eyeglasses. We expect these and other devices to spark renewed scientific and commercial interest in AR and its applications. AR systems, however, still suffer from a fatal flaw in that they are too slow for human vision. This slowness, or latency, hides in every subsystem, from a tracking-camera's capture, to deep rendering pipelines, to rendering frame buffers, to reformatting in the display controller. This latency causes causes misregistration between the synthetic imagery and its real-world counterparts. In other words, as you turn your head, the image that is supposed to remain superimposed on the real-world will start to move when it should not, and only later go back to where it should have stayed. This compromises the utility of the augmentation for many applications, in particular for high-precision uses such as aircraft maintenance or surgery. The proposed AR solution, combined with emerging comfortable, eyeglass-style head-worn displays should enable a wide range of applications to benefit from computer-generated visual augmentation: telepresence, medical examinations & procedures, maintenance, and navigation. Many applications that today use conventional displays for visualization will be able to use head-worn displays and reap the benefits of natural hand-eye coordination with augmented imagery anywhere the user looks. The project will be integrated into multiple courses at the University of North Carolina at Chapel Hill, which will stimulate student exploration of new directions in rendering, tracking, image acquisition and reconstruction, augmented reality and telepresence. Research products will expose to the broader research and development community a new approach to real-time 3D vision and 3D graphics - scanline stream processing, from camera capture to display update - which yields dramatically lower latencies and thus higher-fidelity alignment between real and augmented imagery.The project will exploit a characteristic of inexpensive cameras (continuous scanout, or "rolling shutter") that until now has been seen as a significant weakness of these devices, and will demonstrate how this is in fact a significant asset for providing more frequent updates of scene information. The project will replace the classic frame-by-frame processing of each subsystem with a unified scanline-based approach that significantly reduces latency. The user's tracked head pose will be updated at every scanline, just after each scanline is streamed in from a cluster of cameras affixed to the user's head-worn display. To match this tracking performance, the project will render the augmented imagery in scan line fragments by directly controlling the pixels in the fastest available display technology, that of Digital Micro-mirror Displays (DMD). All of these operations will be aimed to be eventually performed within a mobile device, communicating wirelessly with the user's eyeglass-stye display. Such mobile operation should empower precise augmentation over the user's visual field for a wide range of useful AR applications.