Operational Refinement of Computation for Multimedia Coding Systems

目前的多媒体编码系统在系统资源退化的情况下无法提供无缝的质量退化。例如，如果一个人在便携式视频播放器上观看视频，或者正在打一个非常重要的电话，这会耗尽系统资源（电池），当前的系统不允许在视觉（音频）质量与电池寿命（计算）之间进行无缝权衡。今天，用户实际上面临着数字世界的开/关情况，而人们会强烈选择模拟世界，在模拟世界中，能量或计算资源（复杂性）与多媒体质量（例如视觉或听觉失真）相交换。我们建议从根本上改变传统多媒体编码算法的计算方式，基于一种新的范式，我们称之为多媒体编码系统计算的操作细化。关键原理是基于改变多媒体编码算法的实现，使新的计算增量细化原理成为可能：在对多媒体信息（如图像/视频/音频）进行细化的情况下，算法计算对先前计算的结果进行细化，从而导致输出的增量计算。与现有系统相比，输入/输出多媒体信号的增量处理或重建具有三个关键优势。首先，复杂性-失真的权衡可以公式化，因为每个细化层都以额外的复杂性为代价提高输出结果的质量（减少失真）。其次，每个细化输入/输出层通常由有限动态范围的数据组成（例如，单比特精度数据）。因此，可以根据源统计数据更准确地对处理任务的复杂性进行建模。第三，每个细化层可以安排在实现体系结构的不同部分，并且所有层的计算可以并行化。这有望显著提高执行速度和硬件利用率。这个提议来得正是时候。在像素级或采样级的图像传感器上合并基于逐次逼近的模数转换器的新型采样和捕获设备已经有了大量的研究。这使得基于样本或基于位平面的输入多媒体数据捕获成为可能。同时，最近的结果表明，图像显示使大量亮度阴影的增量细化无闪烁是可能的。这使得增量生成的输出可以直接由显示监视器使用。这些电路理论和设计方面的新发展在解决增量处理输入数据的系统的捕获和显示方面似乎非常有希望。总之，传统系统提供全有或全无的多媒体表示；当资源不可用时，不能任意中断计算并检索最终结果的有意义近似值。对比现有的范例，我们首次提出了一种新的尽力而为的信号处理和多媒体系统。这种类型的系统应用于所有环境中，在这些环境中，由于环境的限制，资源可能缺乏或不确定，基于用户的选择，或者最后，由构造决定。例如：*能源有限的便携式多媒体系统，*资源受限的自适应监视或监控应用程序，*容错多媒体算法和系统设计，以及*逐步定价方案和质量可升级硬件的逐步升级。

Multimedia coding systems today cannot provide seamless quality degradation under degraded system resources. For example, if one watches a video on a portable video player, or is in the middle of a very important phone call, and this is draining the system resources (battery), current systems do not allow for seamless trade-offs in visual (audio) quality vs battery life (computation). Today the user is practically facing the on/off situation of the digital world, while one would strongly opt for an analogue world, where energy or computational resources (complexity) are traded off with multimedia quality (e.g. visual or audible distortion).We propose to fundamentally alter the way conventional multimedia coding algorithms are computed based on a new paradigm that we call Operational Refinement of Computation for Multimedia Coding Systems . The key principle is based on altering the realization of multimedia coding algorithms to enable the new principle of incremental refinement of computation: under a refinement of the multimedia information (e.g. images/video/audio), the algorithm computation refines the previously-computed result thereby leading to incremental computation of the output. The incremental processing or reconstruction of the input/output multimedia signals enables three key advantages in comparison to existing systems. Firstly, complexity-distortion trade-offs can be formulated since every refinement layer improves upon the quality of the output result (reduces distortion) at the cost of additional complexity. Secondly, each refinement input/output layer typically consists of data with limited dynamic-range (e.g. single-bit precision data). Hence, the complexity of the processing tasks can be modelled more accurately in function of the source statistics. Thirdly, each refinement layer can be scheduled in a different part of the implementation architecture and the computation of all layers can be parallelized. This is expected to increase the execution speed and hardware utilization significantly.This proposal comes at an excellent time. There has been a flurry of research on novel sampling and capturing devices that merge successive-approximation based analogue-to-digital converters with image sensors at the pixel or sample level. This enables the sample-based, or bitplane-based capturing of the input multimedia data. At the same time, very recent results demonstrated that image displays enabling the incremental refinement of a large number of luminance shades without flicker are possible. This enables the incrementally-produced output to be directly consumed by the display monitor. These novel developments in circuit theory and design seem very promising in solving the capturing and display aspects for systems that process the input data incrementally.In summary, conventional systems provide an all or nothing multimedia representation; the computation cannot be interrupted arbitrarily when resources become unavailable and retrieve a meaningful approximation of the final result. Contrasting the existing paradigm, we propose to investigate, for the first time, a new category of best-effort signal processing and multimedia systems. Applications of this type of systems are in all environments where resources may bescarce or uncertain due to environmental constraints, based on user choice, or, finally, by construction. Examples are:* portable multimedia systems with limited energy resources,* resource-constrained adaptive surveillance or monitoring applications with always on features,* fault tolerant multimedia algorithm and system design, and* progressive pricing schemes and progressive upgrades for quality-upgradeable hardware.