Graphics Pipelines for Next Generation Mixed Reality Systems

点击翻译按钮获取中文摘要

In the past twenty years display technology has moved on considerablywith improvements such as higher resolution, faster frame rates andhigh-dynamic range colour. In the same period graphics processingunits (GPUs or graphics cards) have become significantly faster withbroader functionality. However, we argue that current implementationsof the traditional graphics pipeline, which is based on therasterisation and z-buffering, are unsuited to emerging displays.In particular, the types of near-eye displays used for augmentedreality and virtual reality provide new challenges. Since theircontexts of use are very different, it is not even clear whatproperties are ideal. Should the displays be multi-focal orvari-focal? How fast do display update rates need to be to supportregistered augmented reality systems? How do we exploit the propertiesof the human visual system to render more efficiently?The traditional computer graphics pipeline puts emphasis on generatingfull images at the display size and rate, where that display might be1920x1080 at 60Hz or higher. For a near-eye display, it is clear thatsuch a pipeline is nowhere near suitable: already one consumer HMD isdemanding 2480x1080 at 90Hz, roughly double the bandwidth that a commondesktop display requires. At these rates, there will still be problemswith visual acuity and latency (e.g. there is an inherent 11ms displaylag). Future HMDs are touted with resolutions of 8K pixels across, butthis will require a very significant increase in graphics compute andthus power consumption. Not only is this expensive, it could limit theform factor of the device and thus user-acceptance of the technology.There is very exciting work going on in display technologies at themoment. For example, Varjo (a project partner) are building a HMD thathas a moving display insert that tracks the eye; Facebook have built ademonstration display that uses a focal surface to generate multiplefocal depths in the same image.In comparison, the graphics pipeline is taken mostly as a given.Recently, the proposers, along with a small group of colleagues in thefield, have started to the challenge the status quo. We don't proposeto ditch the highly optimised compute units in graphics cards, butrather to study frameworks within they can be exploited more readily.We believe that by reformulating the graphics pipeline and payingattention to the very specific needs of near-eye displays, we canradically reduce the power required from GPUs, and thus make near-eyedisplay more usable.We will focus on three connected challenges that we have labelled thelatency, redundancy and bandwidth challenges. First, we will targetextremely low latency displays. We will develop systems that achieve>1000 fps visual output, with latency under 1ms and study how theseimpact visual response. Second, we will explore stronger decoupling offrame-based rendering from display. We note that in near-eye displaysmost pixels are wasted, and thus we target novel spatial and temporalalgorithms that reduce redundancy. Third, to exploit redundancy moregenerally, we need to use it to reduce bandwidth between graphics cardand display. Taking inspiration from the concept of surface lightfields, our concept of ambient fields will render to buffers that areexpected to be valid for re-rendering to the user for 10-100ms. In summary, we believe that the current graphics pipeline and itsassociated implementation in GPUs is unsuited to drive near-eyedisplays. We want near-eye displays to be low-cost, power efficientand highly acceptable to users. To achieve this, we propose newalgorithms that can use GPU capabilities more effectively. We targetreducing redundant compute to enable lower latency and lower bandwidthrequirements through the graphics pipeline.