CHS: Small: Minimal-Latency Tracking and Display for Head-Worn Augmented Reality Systems

增强现实(AR)使计算机图像能够叠加到用户对周围环境的看法上，最自然的是通过头戴式显示器。几十年来，由于头盔笨重和显示屏不足，许多AR应用程序一直受到阻碍。随着新的商用紧凑型显示器设计的到来，这种情况预计很快就会改变，这些设计接近眼镜的形状系数。我们预计，这些设备和其他设备将重新激发人们对AR及其应用的科学和商业兴趣。然而，增强现实系统仍然存在一个致命的缺陷，那就是它们对人类视觉来说太慢了。这种缓慢或延迟隐藏在每个子系统中，从跟踪摄像机的捕获到深度渲染管道，到渲染帧缓冲区，再到显示控制器中的重新格式化。这种延迟会导致合成图像与其真实世界对应图像之间的配准错误。换句话说，当你转过头时，原本应该叠加在现实世界上的图像会在不应该移动的时候开始移动，只是后来才会回到它应该停留的地方。这影响了增强在许多应用中的效用，特别是对于高精度的应用，如飞机维护或手术。拟议的AR解决方案，再加上新兴的舒适、眼镜式的头戴式显示器，应该会使广泛的应用程序受益于计算机生成的视觉增强：远程呈现、医疗检查和检查程序、维护和导航。今天使用传统显示器进行可视化的许多应用程序将能够使用头戴式显示器，并在用户观看的任何地方获得自然手眼协调和增强图像的好处。该项目将被整合到北卡罗来纳大学教堂山分校的多门课程中，这将刺激学生在渲染、跟踪、图像获取和重建、增强现实和远程呈现方面探索新方向。研究产品将向更广泛的研发社区展示一种实时3D视觉和3D图形的新方法-扫描线流处理，从相机捕获到显示更新-大大降低延迟，从而在真实图像和增强图像之间实现更高保真度的对齐。该项目将利用廉价相机(连续扫描输出或滚动快门)的一个特点，到目前为止，这一直被视为这些设备的一个重大弱点，并将展示这实际上是如何提供更频繁的场景信息更新的重要资产。该项目将用基于扫描线的统一方法取代每个子系统的传统逐帧处理，从而显著减少延迟。用户跟踪的头部姿势将在每条扫描线上更新，就在每条扫描线从固定在用户头上的显示器上的一组摄像头流传输进来之后。为了匹配这种跟踪性能，该项目将通过直接控制现有最快的显示技术--数字微镜显示器(DMD)中的像素，在扫描线片段中呈现增强的图像。所有这些操作的最终目标都是在移动设备中执行，与用户的眼镜式显示器进行无线通信。这样的移动操作应该能够在用户的视野内进行精确的增强，以实现广泛的有用AR应用。

Augmented Reality (AR) enables computer images to be superimposed onto a user's view of his or her surroundings, most naturally via a head-worn display. For decades, many AR applications have been held back by bulky head-gear and inadequate displays. This situation is expected to change soon with the arrival of new commercially-available compact display designs that approach the form factor of eyeglasses. We expect these and other devices to spark renewed scientific and commercial interest in AR and its applications. AR systems, however, still suffer from a fatal flaw in that they are too slow for human vision. This slowness, or latency, hides in every subsystem, from a tracking-camera's capture, to deep rendering pipelines, to rendering frame buffers, to reformatting in the display controller. This latency causes causes misregistration between the synthetic imagery and its real-world counterparts. In other words, as you turn your head, the image that is supposed to remain superimposed on the real-world will start to move when it should not, and only later go back to where it should have stayed. This compromises the utility of the augmentation for many applications, in particular for high-precision uses such as aircraft maintenance or surgery. The proposed AR solution, combined with emerging comfortable, eyeglass-style head-worn displays should enable a wide range of applications to benefit from computer-generated visual augmentation: telepresence, medical examinations & procedures, maintenance, and navigation. Many applications that today use conventional displays for visualization will be able to use head-worn displays and reap the benefits of natural hand-eye coordination with augmented imagery anywhere the user looks. The project will be integrated into multiple courses at the University of North Carolina at Chapel Hill, which will stimulate student exploration of new directions in rendering, tracking, image acquisition and reconstruction, augmented reality and telepresence. Research products will expose to the broader research and development community a new approach to real-time 3D vision and 3D graphics - scanline stream processing, from camera capture to display update - which yields dramatically lower latencies and thus higher-fidelity alignment between real and augmented imagery.The project will exploit a characteristic of inexpensive cameras (continuous scanout, or "rolling shutter") that until now has been seen as a significant weakness of these devices, and will demonstrate how this is in fact a significant asset for providing more frequent updates of scene information. The project will replace the classic frame-by-frame processing of each subsystem with a unified scanline-based approach that significantly reduces latency. The user's tracked head pose will be updated at every scanline, just after each scanline is streamed in from a cluster of cameras affixed to the user's head-worn display. To match this tracking performance, the project will render the augmented imagery in scan line fragments by directly controlling the pixels in the fastest available display technology, that of Digital Micro-mirror Displays (DMD). All of these operations will be aimed to be eventually performed within a mobile device, communicating wirelessly with the user's eyeglass-stye display. Such mobile operation should empower precise augmentation over the user's visual field for a wide range of useful AR applications.