RINGS: Bumblebee: A Neural Network Transformer Architecture for Summarization and Prediction in Interactive XR Applications

扩展现实(XR)应用通过虚拟和增强现实等技术将虚拟内容与物理世界紧密结合在一起。这些应用是计算密集型、延迟敏感型和带宽密集型应用，这使它们成为下一代通信架构的理想驱动因素。当前的尽力而为网络技术难以满足XR工作负载的高要求，从而导致网络数据包丢失或延迟、明显的抖动和糟糕的体验质量。幸运的是，在现代XR系统中，过去和未来的数据包之间存在统计相关性，这些数据包未得到充分利用。该项目的目标是开发一种快速、弹性和自适应的通用神经网络变压器架构，用于交互式XR应用程序，使用机器学习(ML)来建模、历史总结、适应、预测并随后利用流时间序列数据。所提出的建模策略是基于变压器的，但它可以表示任意远古时代的上下文关系，这通常是不可行的，它还可以通过一种新颖的自适应最终神经网络层来快速适应不断变化的统计信息。这项工作将导致一个系统，可以透明地插入网络控制消息，以减少带宽通过预测和外推。它还将有助于掩盖无线通道上的分组延迟和丢弃，改善网络协调，这对许多XR应用程序至关重要，如AR制导手术、搜索和救援、数字网真和汽车平视显示器。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Extended Reality (XR) applications tightly couple virtual content with the physical world through technologies such as virtual and augmented reality. These applications are compute-intensive, latency-sensitive, and bandwidth-hungry, making them ideal drivers for next-generation communication architectures. Current best-effort networking techniques struggle to meet the high demand of XR workloads leading to dropped or delayed network packets, observable jitter, and poor Quality-of-Experience. Fortunately, there are statistical correlations between past and future packets that go under-exploited in modern XR systems. The goal of this project is to develop a fast, resilient, and adaptive general neural network transformer-based architecture for interactive XR applications using machine learning (ML) to model, historically summarize, adapt to, predict, and then utilize streaming time-series data. The proposed modeling strategy is transformer-based, but it can represent context relationships going arbitrarily far back in time, something ordinarily infeasible, and it can also rapidly adapt to changing statistics via a novel adaptive final neural network layer. This work will result in a system that can transparently interpose network control messages to reduce bandwidth through forecasting and extrapolation. It will also help to mask packet delays and drops over wireless channels improving network coordination critical to many XR applications like AR guided surgery, search and rescue, digital telepresence and automotive heads up displays.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.