CHS: Small: Adaptive rendering and display for emerging immersive experiences

在电视和显示器上呈现信息的基本架构几乎没有改变：信息仍然以像素的形式发送，逐行扫描成一系列称为帧的图像。与此同时，一类越来越重要的沉浸式应用正在出现：社交丰富的视频会议支持非语言交流；现场学习让远程学生使用增强现实技术在情境中观看课程，比如在花园里学习生物学；远程手术使医生能够远距离提供医疗服务。不幸的是，传统的显示架构无法提供这些应用所需的关键功能，包括真人大小、零延迟的远程窗口；以及多人观看、不戴眼镜的体验。该项目研究了可能支持这些功能的变革性、自适应显示架构。通过使用API（应用程序编程接口）设计抽象的“理想显示”，研究团队将能够探索支持新兴应用的各种底层显示架构。在这个过程中，它将召开研讨会，建立多学科研究关系，以实现这些目标，并开始解决妇女和少数族裔在计算机领域的缺乏，并在大学的计算机图形学课程上做出特别努力。为了重新激发显示领域的创新，该项目将专注于五种非传统的自适应显示技术：无框渲染、选择性更新、本地存储和处理、观看者的知识和高级表示。无框渲染用代表不同时刻的像素构成图像。这在减少延误方面特别有效。选择性更新替换图像的一部分，而不是全部，这在图像太大而无法完全更改时非常有用。本地存储和处理允许将旧样本存储在显示器上，然后及时与新样本结合，以更少的延迟和带宽生成图像。像梯度和边缘这样的高级原语通过简洁地描述局部像素分布来减少带宽。最后，对观看者的了解允许显示器只在需要的地方和时间呈现信息，例如通过使用眼动追踪。该项目将应用这些技术实现三个目标。首先，为理想的感知显示定义一个开放的API，使所有的视觉感觉都饱和。这项工作将推动未来的研究，通过识别服务不足的感知灵敏度，澄清实际显示中的工程权衡，并增加这些显示的市场。第二个和第三个目标是在混合显示- fpga（现场可编程门阵列）系统上对理想感知显示的两个自适应近似进行原型化，以测试和通知我们的API。无框、注视点显示将减少周边的细节，以减少延迟，特别是对于大型显示器。焦点光场显示将根据视角变化颜色，以支持裸眼立体和多用户观看。该项目将对这两种原型进行评估，并将它们与更传统的显示器进行感性比较。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The fundamental architecture for rendering information on TVs and monitors has changed very little: the information is still sent as pixels, scanned row by row into a sequence of images called frames. At the same time, an increasingly important class of immersive applications is emerging: socially rich videoconferencing supports the nonverbal communication; in-situ learning has remote students using augmented reality to see lessons in context, such as biology in the garden; teleoperation allows doctors to deliver healthcare at distance. Unfortunately, traditional display architecture cannot deliver crucial features these applications require, including life-size, zero-delay remote windows; and multi-viewer, glasses-free experiences. This project investigates transformative, adaptive display architectures that might support these features. By designing an abstract “ideal display” using an API (Application Programming Interface), the research team will enable exploration of a diverse range of underlying display architectures for supporting emerging applications. In this process, it will convene workshops to build the multidisciplinary research relationships needed to meet these goals and begin addressing the dearth of women and minorities in computing with a special effort in the university's computer graphics class. To rekindle innovation in displays, the project will focus on five non-traditional, adaptive display techniques: frameless rendering, selective updates, local storage and processing, knowledge of the viewer, and high-level representations. Frameless rendering forms images with pixels representing different moments in time. This is particularly effective at reducing delay. Selective updates replace part of the image, rather than all of it, useful when images are too large for a full change. Local storage and processing allow older samples to be stored on the display, then combined with new samples just in time to make imagery with less delay and bandwidth. Higher level primitives like gradients and edges reduce bandwidth by succinctly describing local pixel distributions. Finally, knowledge of the viewer allows displays to render information only where and when it is needed, for example by using eye tracking. The project will apply these techniques toward three goals. First, to define an open API for the ideal perceptual display, which saturates all of the visual sense. This effort will drive future research by identifying underserved perceptual sensitivities, clarifying engineering tradeoffs in actual displays, and increasing the market for those displays. The second and third goals are to prototype two adaptive approximations of the ideal perceptual display on hybrid display-FPGA (Field Programmable Gate Arrays) systems, to test and inform our API. A frameless, foveated display will reduce detail in the periphery to reduce delay, particularly for large displays. A foveated light field display will vary color by view angle to support glasses-free stereo and multi-user viewing. The project will evaluate both prototypes with perceptual comparisons of them to more traditional displays.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.