Graphics Pipelines for Next Generation Mixed Reality Systems

在过去的二十年里，显示技术不断进步，例如更高的分辨率，更快的帧速率和高动态范围的颜色。在同一时期，图形处理单元（GPU或图形卡）已经变得更快，功能更广泛。然而，我们认为，目前的实现传统的图形流水线，这是基于theresation和z缓冲，是不适合新兴的显示器。特别是，近眼显示器的增强现实和虚拟现实的类型提供了新的挑战。由于它们的使用环境非常不同，甚至不清楚什么属性是理想的。显示器应该是多焦点还是可变焦点？显示器更新速度需要多快才能让增强现实系统变得强大？我们如何利用人类视觉系统的特性来更有效地渲染？传统的计算机图形管道强调以显示尺寸和速率生成完整图像，其中显示可能是1920 x1080，60 Hz或更高。对于近眼显示器来说，很明显这样的流水线是不合适的：已经有一个消费者HMD要求在90 Hz时达到2480 x1080，大约是普通桌面显示器所需带宽的两倍。在这些速率下，仍然会有视觉敏锐度和延迟的问题（例如，存在固有的11 ms显示延迟）。未来的HMD被吹捧为8 K像素的分辨率，但这将需要显着增加图形计算和功耗。这不仅是昂贵的，它可能会限制设备的形状因素，从而限制用户对该技术的接受。目前在显示技术方面正在进行非常令人兴奋的工作。例如，Varjo（一个项目合作伙伴）正在建造一个HMD，它是一个跟踪眼睛的移动显示器插入物; Facebook已经建造了一个演示显示器，它使用焦面在同一图像中生成多个焦深。相比之下，图形管道大多被认为是给定的。最近，提议者，沿着一小群该领域的同事，已经开始挑战现状。我们并不打算放弃显卡中高度优化的计算单元，而是研究其中的框架，以便更容易地利用这些框架。我们相信，通过重新制定图形管道并关注近眼显示器的特殊需求，我们可以从根本上降低GPU所需的功耗，从而使近眼显示器更加可用。我们将重点关注三个相关的挑战，我们称之为延迟，冗余和带宽挑战。首先，我们将针对极低的延迟显示。我们将开发实现>1000 fps视觉输出的系统，延迟低于1 ms，并研究这些如何影响视觉反应。其次，我们将探索更强的解耦基于帧的渲染从显示。我们注意到，在近眼显示器中，大多数像素都被浪费了，因此我们的目标是减少冗余的新的空间和时间算法。第三，为了更普遍地利用冗余，我们需要使用它来减少显卡和显示器之间的带宽。从表面光场的概念中获得灵感，我们的环境场概念将渲染到缓冲区，这些缓冲区预计可在10- 100 ms内重新渲染给用户。总之，我们认为当前的图形流水线及其在GPU中的相关实现不适合驱动近眼显示器。我们希望近眼显示器成本低、能效高、用户接受度高。为了实现这一目标，我们提出了可以更有效地使用GPU功能的新租赁。我们的目标是减少冗余计算，通过图形管道实现更低的延迟和更低的带宽要求。

In the past twenty years display technology has moved on considerablywith improvements such as higher resolution, faster frame rates andhigh-dynamic range colour. In the same period graphics processingunits (GPUs or graphics cards) have become significantly faster withbroader functionality. However, we argue that current implementationsof the traditional graphics pipeline, which is based on therasterisation and z-buffering, are unsuited to emerging displays.In particular, the types of near-eye displays used for augmentedreality and virtual reality provide new challenges. Since theircontexts of use are very different, it is not even clear whatproperties are ideal. Should the displays be multi-focal orvari-focal? How fast do display update rates need to be to supportregistered augmented reality systems? How do we exploit the propertiesof the human visual system to render more efficiently?The traditional computer graphics pipeline puts emphasis on generatingfull images at the display size and rate, where that display might be1920x1080 at 60Hz or higher. For a near-eye display, it is clear thatsuch a pipeline is nowhere near suitable: already one consumer HMD isdemanding 2480x1080 at 90Hz, roughly double the bandwidth that a commondesktop display requires. At these rates, there will still be problemswith visual acuity and latency (e.g. there is an inherent 11ms displaylag). Future HMDs are touted with resolutions of 8K pixels across, butthis will require a very significant increase in graphics compute andthus power consumption. Not only is this expensive, it could limit theform factor of the device and thus user-acceptance of the technology.There is very exciting work going on in display technologies at themoment. For example, Varjo (a project partner) are building a HMD thathas a moving display insert that tracks the eye; Facebook have built ademonstration display that uses a focal surface to generate multiplefocal depths in the same image.In comparison, the graphics pipeline is taken mostly as a given.Recently, the proposers, along with a small group of colleagues in thefield, have started to the challenge the status quo. We don't proposeto ditch the highly optimised compute units in graphics cards, butrather to study frameworks within they can be exploited more readily.We believe that by reformulating the graphics pipeline and payingattention to the very specific needs of near-eye displays, we canradically reduce the power required from GPUs, and thus make near-eyedisplay more usable.We will focus on three connected challenges that we have labelled thelatency, redundancy and bandwidth challenges. First, we will targetextremely low latency displays. We will develop systems that achieve>1000 fps visual output, with latency under 1ms and study how theseimpact visual response. Second, we will explore stronger decoupling offrame-based rendering from display. We note that in near-eye displaysmost pixels are wasted, and thus we target novel spatial and temporalalgorithms that reduce redundancy. Third, to exploit redundancy moregenerally, we need to use it to reduce bandwidth between graphics cardand display. Taking inspiration from the concept of surface lightfields, our concept of ambient fields will render to buffers that areexpected to be valid for re-rendering to the user for 10-100ms. In summary, we believe that the current graphics pipeline and itsassociated implementation in GPUs is unsuited to drive near-eyedisplays. We want near-eye displays to be low-cost, power efficientand highly acceptable to users. To achieve this, we propose newalgorithms that can use GPU capabilities more effectively. We targetreducing redundant compute to enable lower latency and lower bandwidthrequirements through the graphics pipeline.