CAREER: Scarlet: Learned Protocols and Functional Architectures for Low-Latency Internet Video

该项目将创建新的方法来设计在互联网上实时传输视频和音频的系统，改进商业最先进的系统(例如，Zoom、Google Meet、Skype或FaceTime)。这个项目背后的假设是，学术计算机科学中的新技术(如纯功能体系结构、机器学习和低延迟实时分布式系统设计)已准备好成功应用于这一领域。最终目标将是创建新的系统，让人们感觉就像在同一个地方，能够在互联网上表演音乐或戏剧，或者进行强有力的对话。该项目计划在三个领域进行科学研究：协议、编解码器和压缩。对于协议，其核心思想是应用模型预测控制和现场学习来研究单个网络路径，并细粒度地预测其逐个数据包的动态，以便控制近期行为。对于编解码器，主要假设是将机器学习紧密地结合到编码器的内部码率控制算法中，可以减少端到端延迟，并产生对传输协议有用的代理目标函数。如果这些努力成功，这将为压缩提供一个机会：低延迟可以用来替代带宽，实时凝视信息和视线相关的压缩可以带来数倍的压缩增益和通过真实路径传输“视网膜质量”的360x180(虚拟现实)视频流。该项目将包括一系列现场在线戏剧和音乐表演，使用通过互联网连接的远程演员和观众。这些表演将成为研究项目的学习机会和强制功能，为不能旅行的表演者提供一个身体搭配的机会，并在日常环境中公开展示计算机科学研究的好处。该项目将包括为斯坦福大学本科生开设的关于视频和音频传输技术开发的新课程。该项目将在https://github.com/stanford-stagecast.维护一个源代码存储库代码将在编写时公开发布；数据和结果将在学术文献中发布，也将公开可用(从存储库链接)。该项目网站将维护至少三年。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project will create new approaches to the design of systems that transmit video and audio over the Internet in real time, improving on the commercial state-of-the-art (e.g., Zoom, Google Meet, Skype, or FaceTime). The hypothesis underlying this project is that new techniques in academic computer science (such as purely functional architectures, machine learning, and low-latency real-time distributed systems design) are ready to be applied successfully to this area. The ultimate goal will be to create new systems that allow people to feel like they are in the same place, able to perform music or theater or have a robust conversation while separated over the Internet.The project plans to pursue scientific inquiries in three areas: protocols, codecs, and compression. For protocols, the core idea is to apply Model-Predictive Control and in-situ learning to study an individual network path and predict its packet-by-packet dynamics at a fine grain, in order to control near-term behavior. For codecs, the main hypothesis is that incorporating machine learning tightly in an encoder's inner rate-control algorithm can reduce end-to-end latency and produce a proxy objective function that is useful to the transport protocol. If these efforts are successful, this opens up an opportunity for compression: the idea that low latency can be used to substitute for bandwidth, and that real-time gaze information and gaze-contingent compression can lead to manyfold compression gains and "retina-quality" 360x180 (virtual reality) video streaming over realistic path capacities.The project will include in a series of live online theatrical and musical performances, using far-flung actors and audience members connected over the Internet. The performances will serve as a learning opportunity and forcing function for the research project, an opportunity for performers who can't travel to be physically collocated, and a public demonstration of the benefits of computer-science research in an everyday setting. The project will include new classes taught for undergraduates at Stanford University on the development of video and audio transmission technology.The project will maintain a source-code repository at https://github.com/stanford-stagecast. Code will be released publicly as it is written; data and results will be published in the academic literature and will be available publicly (linked from the repository) as well. The project website will be maintained for at least three years.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.