NeTS-NBD: Application-Level Broadband Protocol Framework for Mutli-Camera 3D Video Delivery

随着多媒体计算和通信技术的发展，智能房间中的远程沉浸式环境正在成为协作交互的推动因素。有许多智能房间系统的例子，提供令人兴奋的基础设施，以加强教学，远程教育，资源共享和本地小组活动。例如，用于支持智能房间基础设施的iLAND、Roomware、EasyLiving、Interactive Workspaces、Oxygen和Gaia系统。这些智能环境的特点是单用户设备高度集中，从手持设备到高性能高清电视等离子显示器，通过异构网络（如802.11无线以太网或千兆以太网）连接。然而，大多数这些智能房间设备包括多媒体设备，可以帮助创建2D视频会议环境，但不包括3D远程沉浸式环境。例子包括VRVS，一个面向网络的系统，用于视频会议和IP上的协同工作，Polycom, Netmeeting， vic和vat通过MBone IP多播广播音频和视频。另一方面，在智能房间环境之外，出现了昂贵的沉浸式环境，这些环境要么不支持轻松的远程沉浸，要么难以通过互联网连接。例如CAVE环境、AccessGrid（一种沉浸式组对组协作系统）、TIDE（一种用于协作可视化和探索的远程沉浸式虚拟环境）、Colliseum（一种桌面远程沉浸式系统）。本研究的目标是将联合2D和3D远程沉浸式协作置于智能房间中，考虑在一个智能房间中使用多摄像头2D/3D源，通过LAN和/或WAN将视频数据传输到其他智能房间，其中视频流要么呈现为3D远程沉浸式视频，要么在多个显示器上以多视图方式显示为2D视频流。如果我们考虑一个真正的远程沉浸式环境，那么这个目标的挑战是巨大的：(1)从3D相机捕获的视频流的视频分辨率为640x480像素或更高，每个像素包括RGB和深度信息，编码为每像素分辨率5字节，每秒10帧。这种视频特性意味着每帧处理和传输1.536兆字节，每秒15.36兆字节，每秒122.480兆字节。(2)我们考虑10个摄像机来实现令人满意的远程沉浸式感知，即3D视频流以180度半圆放置在一个房间中，这意味着在一个智能房间中存储和传输15.36兆字节或1.22480千兆比特每秒。(3)所有3D流必须以同步方式到达，以实现正确渲染为最终的3D远程沉浸式视频。目前的多媒体协议交付解决方案仍然考虑(a)更小尺寸的2D视频流，(b)没有联合2D/3D视频，以及(c)如果要传输大量数据，则通常需要专用的网络基础设施（例如，在超级计算应用程序或虚拟现实应用程序的情况下）。因此，迫切需要了解如何在智能房间中组织如此大量的时间敏感视频数据，哪些算法和协议功能必须成为环境的一部分，以协助3D多媒体交付，管理和控制。提出了一种统一的应用级宽带协议框架，用于智能房间之间的多摄像头3D视频传输，以实现时间敏感的3D视频流发送目标。这个应用层宽带框架将探索增强的TCP或类似udp的多媒体增强传输层之上的协议设计空间，作为对改变路由器和传输协议功能的努力的有力补充。pi将使用跨层方法仔细查看驻留在会话层和应用程序层中的服务功能。由于多摄像头3D视频流从房间环境发送并在房间环境中接收，因此可以使用LAN/WAN计算和通信基础设施以分布式方式托管各种服务功能。他们将探索协议空间中特定于内容和特定于中间件的服务功能，例如(a)基于颜色减少的压缩方法，(b)带宽/延迟管理服务，(c)配置服务，以调整发送摄像机的数量，带宽/延迟管理原因，(d)基于用户的定制服务

With recent advances in multimedia computing and communication, tele-immersive environments in smart room are emerging as enablers for collaborative interaction. There are many examples of smart room systems that offer exciting infrastructures to enhance teaching, distance education, resource sharing and local group activities. Examples include the iLAND, Roomware, EasyLiving, Interactive Workspaces, Oxygen and Gaia systems for support of smart room infrastructures. These smart environments are characterized by a high concentration of single user devices, ranging from handheld devices to high-performance HDTV plasma displays, connected via heterogeneous networks such as 802.11 wireless Ethernet or Gigabit Ethernet. However, most of these smart room devices include multimedia devices that can assist in creating 2D video conferencing environments, but not 3D teleimmersive environments. Examples include VRVS, a web oriented system for video conferencing and collaborative work over IP, Polycom, Netmeeting, vic and vat to broadcast audio and video over MBone IP multicast. On the other hand, outside of smart room environments, expensive immersive environments emerged, which either do not support easily tele-immersion or are complex to be connected via Internet. Examples include CAVE environment, AccessGrid, an immersive group-to-group collaboration system, TIDE, a tele-immersive virtual environment for collaborative visualization and exploration, Colliseum, a desktop tele-immersive system.The goal of this research is to place a joint 2D and 3D tele-immersive collaboration into smart rooms, consider multicamera 2D/3D sources in one smart room which transmit the video data over LAN and/or WAN to other smart room(s) where the video streams are either rendered into a 3D tele-immersive video or displayed as 2D video streams in a multi-view fashion on multiple displays. The challenges of this goal are tremendous if we consider a true tele-immersive environment as follows: (1) video resolution of a captured video stream from a 3D camera is 640x480 pixels or higher, each pixel includes RGB and depth information encoded as 5 bytes per pixel resolution, and 10 frames per second. This video characteristic means to process and transmit 1.536Mbytes per frame, 15.36 Mbytes per second, 122.480 MBit per second per stream, (2) we consider ten cameras to achieve a satisfactory teleimmersive perception, i.e., 3D video streams are placed in one room in a 180 degree half-circle, which means 15.36 Mbytes or 1.22480 Gigabits per second to store and transmit in/from/to one smart room, and (3) all 3D streams must arrive in a synchronous manner to achieve a proper rendering into a final 3D tele-immersive video. The current multimedia protocol delivery solutions still consider (a) much smaller sizes of 2D video streams, (b) no joint 2D/3D videos, and (c) if high volume data are to be transmitted, then often dedicated networking infrastructure are in place (e.g., in case of supercomputing applications or Virtual Reality applications). Hence, there is a strong need to understand how to organize such large volumes of time-sensitive video data in smart rooms, what algorithms and protocol functions must be part of the environment to assist in 3D multimedia delivery, management and control.The principal investigators (PIs) propose a unified application-level broadband protocol framework for multi-camera 3D video delivery between smart rooms to reach the goal of sending 3D video streams in time-sensitive manner. This application-level broadband framework will explore the protocol design space above the augmented TCP or UDP-like multimedia-enhanced transport layer as a strong complement to efforts to change the router and transport protocol functions. The PIs will carefully look at service functions residing in the session layer and application layer using a cross-layering approach. Since the multi-camera 3D video streams are sent from a room environment and also received in a room environment, a LAN/WAN computing and communication infrastructure will be available to host the various service functions in a distributed manner. They will explore content-specific and middleware-specific service functions in the protocol space such as (a) a color-reduction-based compression approach, (b) bandwidth/delay management services, (c) configuration service to tune the number of sending cameras for bandwidth/delay management reasons, (d) user-based customization service to tune