CCRI: New: An Open End-to-End Extended Reality System Infrastructure: Enabling Domain-Specific Edge Systems Research

延展实境（XR），包括虚拟、增强和混合现实（AR、VR、MR），也被称为沉浸式计算，预计将渗透到大多数人类的努力中--它将影响我们教学、开展科学、行医、娱乐、培训专业人员、社交互动等的方式。许多人说它将是大多数计算的下一个接口。虽然目前的XR系统存在于今天，但它们远未提供接近人类感知能力的无绳体验。在性能、功率和可用性方面，所需和可实现的之间存在几个数量级的差距，需要系统研究人员进行深入的创新。与此同时，随着Dennard缩放和摩尔定律的终结，应用驱动的专业化或特定于领域的计算已经成为满足新兴应用需求的关键架构技术。 然而，为了真正实现高效的特定领域计算的承诺，特别是XR领域，需要系统研究人员将他们的投资组合扩展到单个加速器的专业化之外。相反，研究人员必须开发专门用于特定领域系统的科学，该系统可能由多个子域组成，需要多个并行异构加速器，这些加速器相互交互以共同满足最终用户的需求。XR领域特定系统研究的一个关键障碍是（直到我们的工作）没有覆盖整个XR工作流程的开源基准或测试平台来推动此类研究。该项目为XR设备开发了一个开源的端到端基础设施。它建立在最初的研究原型ILLIXR（伊利诺伊州延展实境试验台）的基础上。该系统的设计包含用于完整XR工作流程的最先进组件，可协调这些组件调度的可扩展运行时，以及广泛的遥测支持，以测量性能，功率和端到端体验质量指标。该系统是可扩展的并且支持各种操作系统（例如，Linux、Android）和异构平台（例如，NVIDIA Jetson、高通Snapdragon等），传感器（例如，摄像机、伊穆斯等），以及各种XR应用。它为计算堆栈的所有部分提供了新的研究机会，以解决以前不可能实现的端到端XR系统创新。系统研究人员受益于使用基础设施来推动后摩尔领域特定系统的新研究，包括计算机体系结构，编程语言，编译器，运行时系统以及安全和隐私等领域。端到端基础设施推动了针对端到端用户体验进行优化的协同设计系统中的新技术。对于应用程序，XR涵盖多个子领域，如计算机视觉，机器人，图形，信号处理和机器学习。这些领域的算法研究人员可以对新算法进行原型设计和测试，这些算法针对端到端系统效率进行了优化，而无需担心实现堆栈的其余部分，特别是XR研究人员将能够设计针对端到端用户体验进行优化的系统。这项工作解决了计算中两个最重要的问题-处理摩尔定律的终结和设计实现沉浸式计算潜力的系统。这两个奖项都有可能对整个社会产生巨大的影响。这个奖项反映了NSF的法定使命，并被认为是值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估的支持。

Extended reality (XR), which encompasses virtual, augmented, and mixed reality (AR, VR, MR) and also referred to as immersive computing, is expected to pervade most human endeavors --- it will affect the way we teach, conduct science, practice medicine, entertain ourselves, train professionals, interact socially, and more. Many have said it will be the next interface for most of computing. While current XR systems exist today, they are far from providing a tetherless experience approaching perceptual abilities of humans. There is a gap of several orders of magnitude between what is needed and achievable in performance, power, and usability, requiring deep innovations from system researchers. At the same time, with the end of Dennard scaling and Moore's law, application-driven specialization or domain-specific computing has emerged as a key architectural technique to meet the requirements of emerging applications, Computer architects have responded with an explosion of research on highly efficient accelerators, targeting machine learning and other domains. To truly achieve the promise of efficient domain-specific computing in general and for the XR domain in particular, however, requires systems researchers to broaden their portfolio beyond specialization for individual accelerators. Instead, researchers must develop the science for specializing for a domain-specific system which may consist of multiple sub-domains requiring multiple parallel heterogeneous accelerators that interact with each other to collectively meet end-user demands. A key obstacle to domain-specific systems research for XR is that (until our work) there have been no open source benchmarks or testbeds covering the entire XR workflow to drive such research. This project develops an open source end-to-end infrastructure for XR devices. It builds on an initial research prototype, ILLIXR (Illinois Extended Reality Testbed). The system is being designed to contain state-of-the-art components for a complete XR workflow, an extensible runtime that orchestrates the scheduling of these components, and extensive telemetry support to measure performance, power, and end-to-end quality of experience metrics. The system is extensible and supports a variety of operating systems (e.g., Linux, Android) and heterogeneous platforms (e.g., NVIDIA Jetson, Qualcomm Snapdragon, etc.), sensors (e.g., cameras, IMUs, etc.), and various XR applications. It enables new research opportunities in all parts of the computing stack to tackle end-to-end XR system innovations that were previously not possible. Systems researchers benefit from using the infrastructure to drive new research in post-Moore domain-specific systems, in the areas of computer architecture, programming languages, compilers, runtime systems, and security and privacy. The end-to-end infrastructure drives new techniques in co-designed systems that are optimized for end-to-end user experiences. For applications, XR encompasses multiple sub-domains such as computer vision, robotics, graphics, signal processing, and machine learning. Algorithms researchers in these areas can prototype and test new algorithms that are optimized for end-to-end system efficiencies without worrying about implementing the rest of the stack, and XR researchers in particular will be able to design systems optimized for the end-to-end user experience. This work addresses two of the most important problems in computing -- dealing with the end of Moore's law and designing systems that achieve the potential of immersive computing. Both have the potential for tremendous impact on society at large.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.