CHS: Small: Adaptive rendering and display for emerging immersive experiences

在电视和显示器上呈现信息的基本架构几乎没有改变：信息仍然以像素的形式发送，逐行扫描成一系列称为帧的图像。与此同时，一类越来越重要的沉浸式应用正在出现：社交丰富的视频会议支持非语言交流；现场学习让远程学生使用增强现实在情景中观看课程，例如花园中的生物；远程操作允许医生远程提供医疗保健。不幸的是，传统的显示架构无法提供这些应用程序所需的关键功能，包括真人大小、零延迟的远程窗口以及多人观看、免戴眼镜的体验。该项目研究可能支持这些功能的变革性、自适应显示体系结构。通过使用API(应用程序编程接口)设计抽象的“理想显示”，研究团队将能够探索各种支持新兴应用的底层显示架构。在这一过程中，它将召开研讨会，建立实现这些目标所需的多学科研究关系，并开始在大学的计算机图形课程中特别努力，解决计算机领域女性和少数族裔的匮乏问题。为了重新激发显示器的创新，该项目将专注于五种非传统的自适应显示技术：无框渲染、选择性更新、本地存储和处理、观众的知识以及高级表示。无边框渲染使用表示不同时刻的像素来形成图像。这在减少延误方面特别有效。选择性更新将替换图像的一部分，而不是全部，这在图像太大而无法完全更改时很有用。本地存储和处理允许将较旧的样本存储在显示器上，然后及时与新样本组合，以更少的延迟和带宽生成图像。更高级别的基元，如渐变和边缘，通过简洁地描述局部像素分布来减少带宽。最后，了解观察者的信息允许显示器仅在需要的地方和时间呈现信息，例如通过使用眼睛跟踪。该项目将把这些技术应用于三个目标。首先，为理想的感知显示定义一个开放的API，它饱和了所有的视觉感觉。这一努力将通过识别未得到充分利用的感知敏感度、澄清实际显示器中的工程权衡并扩大这些显示器的市场来推动未来的研究。第二个和第三个目标是在混合显示--现场可编程门阵列(现场可编程门阵列)系统上原型两种理想感知显示的自适应近似，以测试和告知我们的API。无边框的凹槽显示器将减少外围的细节以减少延迟，特别是对于大型显示器。凹陷光场显示器将根据视角改变颜色，以支持免眼镜立体声和多用户观看。该项目将对这两个原型进行评估，并将它们与更传统的显示器进行感性比较。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The fundamental architecture for rendering information on TVs and monitors has changed very little: the information is still sent as pixels, scanned row by row into a sequence of images called frames. At the same time, an increasingly important class of immersive applications is emerging: socially rich videoconferencing supports the nonverbal communication; in-situ learning has remote students using augmented reality to see lessons in context, such as biology in the garden; teleoperation allows doctors to deliver healthcare at distance. Unfortunately, traditional display architecture cannot deliver crucial features these applications require, including life-size, zero-delay remote windows; and multi-viewer, glasses-free experiences. This project investigates transformative, adaptive display architectures that might support these features. By designing an abstract “ideal display” using an API (Application Programming Interface), the research team will enable exploration of a diverse range of underlying display architectures for supporting emerging applications. In this process, it will convene workshops to build the multidisciplinary research relationships needed to meet these goals and begin addressing the dearth of women and minorities in computing with a special effort in the university's computer graphics class. To rekindle innovation in displays, the project will focus on five non-traditional, adaptive display techniques: frameless rendering, selective updates, local storage and processing, knowledge of the viewer, and high-level representations. Frameless rendering forms images with pixels representing different moments in time. This is particularly effective at reducing delay. Selective updates replace part of the image, rather than all of it, useful when images are too large for a full change. Local storage and processing allow older samples to be stored on the display, then combined with new samples just in time to make imagery with less delay and bandwidth. Higher level primitives like gradients and edges reduce bandwidth by succinctly describing local pixel distributions. Finally, knowledge of the viewer allows displays to render information only where and when it is needed, for example by using eye tracking. The project will apply these techniques toward three goals. First, to define an open API for the ideal perceptual display, which saturates all of the visual sense. This effort will drive future research by identifying underserved perceptual sensitivities, clarifying engineering tradeoffs in actual displays, and increasing the market for those displays. The second and third goals are to prototype two adaptive approximations of the ideal perceptual display on hybrid display-FPGA (Field Programmable Gate Arrays) systems, to test and inform our API. A frameless, foveated display will reduce detail in the periphery to reduce delay, particularly for large displays. A foveated light field display will vary color by view angle to support glasses-free stereo and multi-user viewing. The project will evaluate both prototypes with perceptual comparisons of them to more traditional displays.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.