III-COR-Small: Developing Novel Mosaic Generation Methods for Object-Based Multimedia Information Systems

本项目为交互式、基于对象的语义多媒体信息检索系统的一个重要组成部分：“马赛克”生成提供了解决方案。马赛克可以被认为是场景的静态组件（或背景），它在一系列帧中不会改变，通过计算帧之间的全局运动，根据全局运动进行变形，然后混合帧来获得。马赛克生成在许多应用中发挥着重要作用，包括基于对象的编码（场景中的对象也被编码或压缩，独立于规则的矩形帧编码）、视频压缩、视频索引、对象跟踪、虚拟环境、安全监控、广域监控、全景视频、交通监控、对象识别、以及人类行为分析，因为这些应用通常需要从背景（或马赛克）中减去实际场景来确定前景对象。由于传统的马赛克生成方法需要对包含运动物体的视频进行对象分割，因此它们不适合实时马赛克生成，特别是对于需要精灵编码的视频编码器（基于在马赛克或精灵上分层物体的编码）。该项目(i)为更大的视频域开发马赛克生成解决方案；（ii）对包含多个镜头的视频进行分类，生成马赛克；（iii）通过生成真实的基础，开发出马赛克生成的客观评价方法。精灵融合方法混合了使用对齐帧差异生成的自信和保守精灵，而没有对象分割，从而消除了对象遮挡问题，特别是对于摄像机跟踪对象的视频。由于精灵融合方法对一对帧只计算一次全局运动，不需要对象分割，因此非常适合实时拼接生成或视频编码器的精灵编码。当为序列构建马赛克时，存储全局运动向量以便能够生成（或扭曲）相对于另一帧的任何帧，从而允许为空间交互提供多个自由度，包括上下，左右，前后移动和旋转。具有当前（或活动）场景扭曲的多个自由度通过从马赛克中再生任意帧，然后将活动（当前场景）覆盖在复制帧的顶部，形成交互式视频复制的基础。这个项目的教育部分包括开发一门新的多媒体课程，吸引所有大一学生，以提高他们对计算机科学的兴趣。本课程涵盖计算机科学的基本概念，包括媒体类型、色彩模型、存储设备、多媒体创作和互联网。计划为K-12学生举办多媒体研讨会。这个研究项目的结果将通过互联网（http://www.cs.uah.edu/~raygun/projects/mosaics.htm）传播，包括一个视频数据库系统（视频集和它的任何真值注释），与其他研究人员共享测试数据，并鼓励各研究机构之间开放的基于指标的评估。此计划将促进使用互动式视像再现的逼真影像系统的发展，并具有广泛的应用。

This project provides solutions to one of the important components of an interactive, object-based, semantic multimedia information retrieval system: "mosaic" generation. A mosaic can be considered as a static component (or background) of a scene that does not change over a sequence of frames and is obtained by computing the global motion between frames, warping according to the global motion, and then blending the frames. Mosaic generation plays an important role in many applications including object-based coding (where objects in a scene are also coded or compressed independent of regular rectangular frame coding), video compression, video indexing, object tracking, virtual environments, security surveillance, wide-area surveillance, panoramic video, traffic monitoring, object recognition, and human behavior analysis since these applications usually require the subtraction of actual scenes from the background (or the mosaic) to determine the foreground objects. Since traditional mosaic generation methods require object segmentation for videos containing moving objects, they are not suitable for real-time mosaic generation and especially for video encoders that require sprite coding (coding based on layering objects on top of a mosaic or a sprite). This project (i) develops mosaic generation solutions for larger domains of videos; (ii) generates mosaics for videos containing many shots by classifying video shots; and (iii) develops objective evaluation methods for mosaic generation by producing ground-truths. Sprite fusion method blends assertive and conservative sprites that are generated using the aligned frame differences without object segmentation thus eliminating the object occlusion problem especially for videos where the camera tracks an object. Since the sprite fusion method computes the global motion once for a pair of frames and does not require object segmentation, it suits well for real-time mosaic generation or sprite coding for video encoders. As the mosaic is constructed for a sequence, the global motion vectors are stored to be able to generate (or warp) any frame with respect to another frame thus allowing to provide multiple degrees-of-freedom for spatial interactions including up-down, left-right, move forward-backward, and rotation. The multiple degrees-of-freedom with the warping of the current (or active) scene forms the basis of interactive video reproduction by regenerating any frame from the mosaic and then overlaying the active (current scene) on top of the reproduced frame. The educational component of this project includes development of a new course on multimedia that appeals to any freshman students in order to grow interest in computer science. This course covers fundamental concepts in computer science including types of media, color models, storage devices, multimedia authoring, and internet. A multimedia workshop is planned for K-12 students. The results of this research project will be disseminated via Internet (http://www.cs.uah.edu/~raygun/projects/mosaics.htm), including a video database system (video sets and any truth annotation for it) to share test data with other researchers and to encourage open metrics-based evaluations across research institutions. This project will foster the development of photo-realistic visualization systems using interactive video reproduction, with a wide range of applications.