NeTS: Small: Supporting Next Generation Adaptive Multi-Lens Stereoscopic Video Streaming

本计画探讨下一代多镜头立体视讯之协调适应与网路串流架构。 未来的立体相机可以由多个（多于两个）透镜组成，以提供视点的更密集采样，从而改善捕获和观看场景相关显示之间的失配，这是随着立体设备和显示器变得更加普遍而增长的问题。 通过适当的流传输系统，可以提供更丰富的用户体验，该用户体验既是网络自适应的，又减少了立体成像的负面副作用，例如3D眼睛疲劳。 多镜头立体视频框架将需要三个相互关联的组件的融合：压缩和表示;检索，流传输和适应;以及观看场景相关的优化。 这三个组件将共同作用，为立体成像提供低延迟、观看场景优化的体验。 考虑到这一架构，该项目将：* 研究多镜头压缩技术，以支持有效的检索和观看场景优化。多镜头立体数据的压缩需要实现高压缩，但同时要服从自适应和子集检索。* 为拟议的检索友好压缩格式开发自适应流技术。 关键目标是避免引入伪影（由于质量下降），使其进入立体场。 该适配层将与观看场景相关优化相协调，以便在检索最佳数据的同时还具有网络自适应性。*开发视景方案相关的最佳显示管理技术。 为了最大化观看体验并最小化负面影响，观看场景相关优化将从阵列中选择图像子集以最佳匹配观看场景。新颖的视图合成技术可用于帮助最小化检索的图像数据，同时提供良好的用户体验。更广泛的影响：根据观看场景动态地提供优化的立体内容可以减少人们因观看立体内容而产生的副作用，例如3D疲劳和头痛。 线程架构还可以用于其他多视图系统（例如，视频传感器网络），其中视频的显示仅需要传送摄像机视图的子集。 最后，显示管理工作的结果还可以与眼动跟踪相结合，以实现自由视图和自由玻璃立体观看体验。

This project investigates a coordinated adaptation and network streaming framework for next generation multi-lens stereoscopic video. Future stereoscopic cameras may be made up of multiple (more than two) lenses to provide a denser sampling of viewpoints to ameliorate the mismatch between capture and viewing scenario dependent display, a problem that will grow as stereoscopic devices and displays become more ubiquitous. With an appropriate streaming system, a much richer user experience can be delivered that is both network adaptive and reduces the negative side-effects of stereoscopic imaging such as 3D eye fatigue. A multi-lens stereoscopic video framework will require the confluence of three interrelated components: compression and representation; retrieval, streaming and adaptation; and viewing scenario dependent optimization. These three components will act together to provide a low-latency, viewing scenario optimized experience for stereoscopic imaging. With this architecture in mind, this project will:* Investigate multi-lens compression technologies to support efficient retrieval and viewing scenario optimization. The compression of the multi-lens stereoscopic data needs to achieve high compression, but at the same time, be amenable to adaptation and subset retrieval. * Develop adaptive streaming techniques for the proposed retrieval-friendly compression format. The key objective is to avoid introducing artifacts (due to the decreased quality) that make their way into the stereoscopic field. This adaptation layer will be coordinated with a viewing scenario dependent optimization in order to retrieve the best data possible while also being network adaptive.* Develop viewing scenario dependent optimal display management technologies. To maximize viewing experience and minimize negative side effects, viewing scenario dependent optimization will select a subset of images from the array to best match the viewing scenario. Novel view synthesis techniques may be used to help minimize the image data retrieved, yet provide good user experience.Broader Impact: Dynamically delivering optimized stereoscopic content according to the viewing scenario can reduce side effects such as 3D fatigue and headaches that people have from viewing stereoscopic content. The threaded architecture can also be used for other multi-view systems (e.g., video sensor networks), where the display of video requires only a subset of the camera views to be delivered. Finally, the results from the display management work can also be combined with eye tracking to enable free-view and free-glass stereo viewing experiences.