RINGS: Bumblebee: A Neural Network Transformer Architecture for Summarization and Prediction in Interactive XR Applications

延展实境（XR）应用通过虚拟和增强现实等技术将虚拟内容与物理世界紧密耦合。这些应用程序是计算密集型、延迟敏感型和带宽需求型的，使其成为下一代通信架构的理想驱动程序。当前的尽力而为网络技术难以满足XR工作负载的高需求，导致网络数据包丢失或延迟、可观察到的抖动和较差的体验质量。幸运的是，在现代XR系统中，过去和未来的数据包之间存在统计相关性。该项目的目标是为交互式XR应用程序开发一个快速，弹性和自适应的通用神经网络转换器架构，使用机器学习（ML）来建模，历史总结，适应，预测，然后利用流式时间序列数据。所提出的建模策略是基于transformer的，但它可以表示任意远的上下文关系，这通常是不可行的，并且它还可以通过一个新的自适应最终神经网络层快速适应不断变化的统计数据。这项工作将导致一个系统，可以透明地过滤网络控制消息，以减少带宽通过预测和推断。 它还将有助于屏蔽无线信道上的数据包延迟和丢失，改善对许多XR应用（如AR引导手术、搜索和救援、数字远程呈现和汽车抬头显示器）至关重要的网络协调。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Extended Reality (XR) applications tightly couple virtual content with the physical world through technologies such as virtual and augmented reality. These applications are compute-intensive, latency-sensitive, and bandwidth-hungry, making them ideal drivers for next-generation communication architectures. Current best-effort networking techniques struggle to meet the high demand of XR workloads leading to dropped or delayed network packets, observable jitter, and poor Quality-of-Experience. Fortunately, there are statistical correlations between past and future packets that go under-exploited in modern XR systems. The goal of this project is to develop a fast, resilient, and adaptive general neural network transformer-based architecture for interactive XR applications using machine learning (ML) to model, historically summarize, adapt to, predict, and then utilize streaming time-series data. The proposed modeling strategy is transformer-based, but it can represent context relationships going arbitrarily far back in time, something ordinarily infeasible, and it can also rapidly adapt to changing statistics via a novel adaptive final neural network layer. This work will result in a system that can transparently interpose network control messages to reduce bandwidth through forecasting and extrapolation. It will also help to mask packet delays and drops over wireless channels improving network coordination critical to many XR applications like AR guided surgery, search and rescue, digital telepresence and automotive heads up displays.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.