NeTS: Small: Supporting Next Generation Adaptive Multi-Lens Stereoscopic Video Streaming

本项目研究下一代多镜头立体视频的协调适配和网络流媒体框架。未来的立体相机可能由多个（超过两个）镜头组成，以提供更密集的视点采样，以改善捕捉和观看场景依赖显示之间的不匹配，随着立体设备和显示器变得越来越普遍，这个问题将会越来越严重。使用合适的流媒体系统，可以提供更丰富的用户体验，既可以适应网络，又可以减少立体成像的负面副作用，如3D眼睛疲劳。多镜头立体视频框架需要三个相互关联的组成部分的融合：压缩和表示；检索、流化和自适应；以及观看场景相关的优化。这三个组件将共同为立体成像提供低延迟、观看场景优化的体验。考虑到这种架构，该项目将：*研究多镜头压缩技术，以支持有效的检索和观看场景优化。多镜头立体数据的压缩既要实现高压缩，又要便于自适应和子集检索。*为建议的检索友好压缩格式开发自适应流技术。关键的目标是避免引入工件（由于质量下降），使其进入立体领域。该适应层将与观看场景相关的优化相协调，以便在检索最佳数据的同时也具有网络适应性。*开发基于观看场景的最佳显示管理技术。为了最大化观看体验并最小化负面影响，观看场景相关优化将从阵列中选择图像子集以最佳匹配观看场景。新的视图合成技术可以用来帮助最小化图像数据检索，但提供良好的用户体验。更广泛的影响：根据观看场景动态提供优化的立体内容，可以减少人们观看立体内容时产生的3D疲劳和头痛等副作用。线程架构也可以用于其他多视图系统（例如，视频传感器网络），其中视频的显示只需要交付相机视图的子集。最后，显示管理工作的结果还可以与眼动追踪相结合，实现自由视角和无玻璃立体观看体验。

This project investigates a coordinated adaptation and network streaming framework for next generation multi-lens stereoscopic video. Future stereoscopic cameras may be made up of multiple (more than two) lenses to provide a denser sampling of viewpoints to ameliorate the mismatch between capture and viewing scenario dependent display, a problem that will grow as stereoscopic devices and displays become more ubiquitous. With an appropriate streaming system, a much richer user experience can be delivered that is both network adaptive and reduces the negative side-effects of stereoscopic imaging such as 3D eye fatigue. A multi-lens stereoscopic video framework will require the confluence of three interrelated components: compression and representation; retrieval, streaming and adaptation; and viewing scenario dependent optimization. These three components will act together to provide a low-latency, viewing scenario optimized experience for stereoscopic imaging. With this architecture in mind, this project will:* Investigate multi-lens compression technologies to support efficient retrieval and viewing scenario optimization. The compression of the multi-lens stereoscopic data needs to achieve high compression, but at the same time, be amenable to adaptation and subset retrieval. * Develop adaptive streaming techniques for the proposed retrieval-friendly compression format. The key objective is to avoid introducing artifacts (due to the decreased quality) that make their way into the stereoscopic field. This adaptation layer will be coordinated with a viewing scenario dependent optimization in order to retrieve the best data possible while also being network adaptive.* Develop viewing scenario dependent optimal display management technologies. To maximize viewing experience and minimize negative side effects, viewing scenario dependent optimization will select a subset of images from the array to best match the viewing scenario. Novel view synthesis techniques may be used to help minimize the image data retrieved, yet provide good user experience.Broader Impact: Dynamically delivering optimized stereoscopic content according to the viewing scenario can reduce side effects such as 3D fatigue and headaches that people have from viewing stereoscopic content. The threaded architecture can also be used for other multi-view systems (e.g., video sensor networks), where the display of video requires only a subset of the camera views to be delivered. Finally, the results from the display management work can also be combined with eye tracking to enable free-view and free-glass stereo viewing experiences.