CCRI: New: An Open End-to-End Extended Reality System Infrastructure: Enabling Domain-Specific Edge Systems Research

扩展现实(XR)包括虚拟、增强和混合现实(AR、VR、MR)，也被称为沉浸式计算，预计将渗透到大多数人类努力-它将影响我们教学、进行科学、行医、娱乐、培训专业人员、社交互动等的方式。许多人说，它将成为大多数计算的下一个界面。虽然目前的XR系统目前已经存在，但它们远远不能提供接近人类感知能力的无绳体验。在性能、功率和可用性方面，需要的和可实现的之间存在着几个数量级的差距，这需要系统研究人员进行深入的创新。与此同时，随着Dennard Scaling和Moore定律的终结，应用程序驱动的专业化或特定于领域的计算已成为满足新兴应用程序要求的关键体系结构技术，计算机架构师以针对机器学习和其他领域的高效加速器的研究爆炸式增长作为回应。然而，为了真正实现有效的特定于领域的计算的承诺，特别是对于XR领域，需要系统研究人员扩大他们的产品组合，而不仅仅是针对单个加速器的专业化。相反，研究人员必须发展专门针对特定领域的系统的科学，该系统可能由多个子域组成，需要多个并行的、不同种类的加速器，这些加速器相互作用以共同满足最终用户的需求。XR领域特定系统研究的一个关键障碍是(在我们的工作之前)还没有涵盖整个XR工作流程的开放源码基准或试验台来推动此类研究。该项目为XR设备开发了一个开源的端到端基础设施。它建立在最初的研究原型ILLIXR(伊利诺伊州扩展现实测试床)的基础上。该系统旨在包含用于完整XR工作流程的最先进组件、协调这些组件调度的可扩展运行时，以及广泛的遥测支持，以衡量性能、功率和端到端体验质量指标。该系统是可扩展的，并且支持各种操作系统(例如，Linux、Android)和异类平台(例如，NVIDIA Jetson、高通骁龙等)、传感器(例如，相机、IMU等)和各种XR应用。它使计算堆栈的所有部分都有新的研究机会，以应对以前不可能实现的端到端XR系统创新。系统研究人员受益于使用基础设施来推动后摩尔域特定系统在计算机体系结构、编程语言、编译器、运行时系统以及安全和隐私领域的新研究。端到端基础设施推动了联合设计系统中的新技术，这些系统针对端到端用户体验进行了优化。对于应用程序，XR包含多个子领域，如计算机视觉、机器人、图形学、信号处理和机器学习。这些领域的算法研究人员可以制作和测试针对端到端系统效率进行优化的新算法，而无需担心实现堆栈的其余部分，尤其是XR研究人员将能够设计出针对端到端用户体验进行优化的系统。这项工作解决了计算中最重要的两个问题--处理摩尔定律的终结和设计实现沉浸式计算潜力的系统。这两个奖项都有可能对整个社会产生巨大的影响。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Extended reality (XR), which encompasses virtual, augmented, and mixed reality (AR, VR, MR) and also referred to as immersive computing, is expected to pervade most human endeavors --- it will affect the way we teach, conduct science, practice medicine, entertain ourselves, train professionals, interact socially, and more. Many have said it will be the next interface for most of computing. While current XR systems exist today, they are far from providing a tetherless experience approaching perceptual abilities of humans. There is a gap of several orders of magnitude between what is needed and achievable in performance, power, and usability, requiring deep innovations from system researchers. At the same time, with the end of Dennard scaling and Moore's law, application-driven specialization or domain-specific computing has emerged as a key architectural technique to meet the requirements of emerging applications, Computer architects have responded with an explosion of research on highly efficient accelerators, targeting machine learning and other domains. To truly achieve the promise of efficient domain-specific computing in general and for the XR domain in particular, however, requires systems researchers to broaden their portfolio beyond specialization for individual accelerators. Instead, researchers must develop the science for specializing for a domain-specific system which may consist of multiple sub-domains requiring multiple parallel heterogeneous accelerators that interact with each other to collectively meet end-user demands. A key obstacle to domain-specific systems research for XR is that (until our work) there have been no open source benchmarks or testbeds covering the entire XR workflow to drive such research. This project develops an open source end-to-end infrastructure for XR devices. It builds on an initial research prototype, ILLIXR (Illinois Extended Reality Testbed). The system is being designed to contain state-of-the-art components for a complete XR workflow, an extensible runtime that orchestrates the scheduling of these components, and extensive telemetry support to measure performance, power, and end-to-end quality of experience metrics. The system is extensible and supports a variety of operating systems (e.g., Linux, Android) and heterogeneous platforms (e.g., NVIDIA Jetson, Qualcomm Snapdragon, etc.), sensors (e.g., cameras, IMUs, etc.), and various XR applications. It enables new research opportunities in all parts of the computing stack to tackle end-to-end XR system innovations that were previously not possible. Systems researchers benefit from using the infrastructure to drive new research in post-Moore domain-specific systems, in the areas of computer architecture, programming languages, compilers, runtime systems, and security and privacy. The end-to-end infrastructure drives new techniques in co-designed systems that are optimized for end-to-end user experiences. For applications, XR encompasses multiple sub-domains such as computer vision, robotics, graphics, signal processing, and machine learning. Algorithms researchers in these areas can prototype and test new algorithms that are optimized for end-to-end system efficiencies without worrying about implementing the rest of the stack, and XR researchers in particular will be able to design systems optimized for the end-to-end user experience. This work addresses two of the most important problems in computing -- dealing with the end of Moore's law and designing systems that achieve the potential of immersive computing. Both have the potential for tremendous impact on society at large.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.