CHS: Small: Minimal-Latency Tracking and Display for Head-Worn Augmented Reality Systems

增强现实（AR）使计算机图像能够叠加到用户对他或她周围环境的视图上，最自然的是通过头戴式显示器。几十年来，许多AR应用一直受到笨重的头盔和不足的显示器的阻碍。随着接近眼镜形状因子的新的商业上可获得的紧凑显示器设计的到来，这种情况预计将很快改变。我们希望这些设备和其他设备能够激发人们对AR及其应用的新的科学和商业兴趣。然而，AR系统仍然存在一个致命的缺陷，那就是它们对于人类视觉来说太慢了。这种缓慢或延迟隐藏在每个子系统中，从跟踪相机的捕获到深度渲染管道，到渲染帧缓冲区，再到显示控制器中的重新格式化。这种延迟导致合成图像与其真实世界对应物之间的配准不良。换句话说，当你转动你的头时，应该保持叠加在现实世界上的图像将在它不应该移动的时候开始移动，然后才回到它应该停留的地方。这损害了增强对于许多应用的效用，特别是对于高精度用途，例如飞机维护或手术。拟议的AR解决方案，结合新兴的舒适的眼镜式头戴式显示器，应该能够使广泛的应用受益于计算机生成的视觉增强：远程呈现，医疗检查程序，维护和导航。如今，许多使用传统显示器进行可视化的应用将能够使用头戴式显示器，并在用户看到的任何地方获得自然手眼协调的好处。该项目将被整合到位于查佩尔山的北卡罗来纳州大学的多门课程中，这将激发学生对渲染、跟踪、图像获取和重建、增强现实和远程呈现等新方向的探索。研究产品将向更广泛的研究和开发社区展示一种实时3D视觉和3D图形的新方法--从相机捕获到显示更新的扫描线流处理--这种方法可以大大降低延迟，从而在真实的图像和增强图像之间实现更高保真的对齐。（连续扫描输出，或“滚动快门”），这直到现在被视为这些设备的显著弱点，并且将展示这实际上如何是用于提供场景信息的更频繁更新的显著资产。该项目将用统一的基于扫描线的方法取代每个子系统的经典逐帧处理，从而显著降低延迟。用户的跟踪的头部姿势将在每条扫描线处更新，就在每条扫描线从附连到用户的头戴式显示器的相机集群流式传输之后。为了匹配这种跟踪性能，该项目将通过直接控制最快的显示技术（数字微镜显示器（DMD））中的像素，在扫描线片段中呈现增强图像。所有这些操作最终都将在一个移动终端内完成，与用户的眼镜式显示器进行无线通信。这样的移动的操作应该能够针对广泛的有用AR应用在用户的视野上进行精确增强。

Augmented Reality (AR) enables computer images to be superimposed onto a user's view of his or her surroundings, most naturally via a head-worn display. For decades, many AR applications have been held back by bulky head-gear and inadequate displays. This situation is expected to change soon with the arrival of new commercially-available compact display designs that approach the form factor of eyeglasses. We expect these and other devices to spark renewed scientific and commercial interest in AR and its applications. AR systems, however, still suffer from a fatal flaw in that they are too slow for human vision. This slowness, or latency, hides in every subsystem, from a tracking-camera's capture, to deep rendering pipelines, to rendering frame buffers, to reformatting in the display controller. This latency causes causes misregistration between the synthetic imagery and its real-world counterparts. In other words, as you turn your head, the image that is supposed to remain superimposed on the real-world will start to move when it should not, and only later go back to where it should have stayed. This compromises the utility of the augmentation for many applications, in particular for high-precision uses such as aircraft maintenance or surgery. The proposed AR solution, combined with emerging comfortable, eyeglass-style head-worn displays should enable a wide range of applications to benefit from computer-generated visual augmentation: telepresence, medical examinations & procedures, maintenance, and navigation. Many applications that today use conventional displays for visualization will be able to use head-worn displays and reap the benefits of natural hand-eye coordination with augmented imagery anywhere the user looks. The project will be integrated into multiple courses at the University of North Carolina at Chapel Hill, which will stimulate student exploration of new directions in rendering, tracking, image acquisition and reconstruction, augmented reality and telepresence. Research products will expose to the broader research and development community a new approach to real-time 3D vision and 3D graphics - scanline stream processing, from camera capture to display update - which yields dramatically lower latencies and thus higher-fidelity alignment between real and augmented imagery.The project will exploit a characteristic of inexpensive cameras (continuous scanout, or "rolling shutter") that until now has been seen as a significant weakness of these devices, and will demonstrate how this is in fact a significant asset for providing more frequent updates of scene information. The project will replace the classic frame-by-frame processing of each subsystem with a unified scanline-based approach that significantly reduces latency. The user's tracked head pose will be updated at every scanline, just after each scanline is streamed in from a cluster of cameras affixed to the user's head-worn display. To match this tracking performance, the project will render the augmented imagery in scan line fragments by directly controlling the pixels in the fastest available display technology, that of Digital Micro-mirror Displays (DMD). All of these operations will be aimed to be eventually performed within a mobile device, communicating wirelessly with the user's eyeglass-stye display. Such mobile operation should empower precise augmentation over the user's visual field for a wide range of useful AR applications.