Developing Information Theoretic Tools for Multimedia Multimodal Information Processing

多模式信息融合分析从几种类型的感觉数据或模式中提取的信号，通常比单一模式获得更好和更可靠的性能。在许多应用中，尤其是在生物识别和多媒体信息处理中，一致地理解和处理来自不同模式的信息的需求变得越来越苛刻。如果不能很好地解决多模式信息融合问题，人类对信息时代的掌控将仍然是一个遥不可及的梦想。*多模式融合有两大优势使多模式系统更加可靠：1)不同模式中嵌入的信息通常是互补的，多模式处理比单一模式(如人类生物计算机)提供更多的信息；2)当一种模式被噪声或环境破坏时，我们可以依靠其他模式来部分弥补缺失的信息。*文献回顾表明，迄今为止的大部分研究缺乏分析基础，主要是出于特殊需求。这样的方法是临时的、不系统的，而且往往很难推广。如果没有适当的方法，信息融合领域充其量将继续是经验性的。因此，迫切需要系统地解决多模式信息融合问题。在这个项目中，我们提出了一种新的基于信息熵估计的框架。熵是衡量无序性或不可预测性的指标。基于熵的方法不是假定流行的二阶统计量，而是允许任意分布和高阶统计量。在信息融合中，熵与置信度密切相关。一般来说，高熵表示对相应通道的置信度较低，反之亦然。将证据与通过后验熵估计的可靠性相结合，以提供一种手段来预测多模式信息的集成是否将导致更好的性能，为手头的应用选择最合适的融合策略，并使融合系统容忍单个模态的质量变化，甚至容忍模态的子集的临时破坏。*拟议研究的预期结果是为开发最先进的多媒体信息处理系统提供知识和工具。所获得的知识将为加拿大带来巨大的社会效益，并为广泛的应用提供使能技术，例如新媒体制作和交付、文化遗产保护、基础设施规划、熟练专业人员的培训、营销和广告、电子健康、智能家居、安全/监控、游戏和电子娱乐，以及利用身临其境的多媒体技术创造逼真的体验。**

Multimodal information fusion analyzes signals extracted from several types of sensory data or modalities and normally leads to better and more reliable performance than that obtained from a single modality. The need to coherently understand and process information from different modalities is becoming more and more demanding in many applications, particularly in biometrics and in multimedia information processing. Without a satisfactory resolution to multimodal information fusion, human mastery of the information era will remain an elusive dream.******There are two major advantages for multimodal fusion that make multimodal systems more reliable: 1) the information embedded in different modalities is usually complementary and multimodal processing is more informative than single modality, e.g. the human biological computing machine; and 2) When one modality becomes corrupted by noise or the environment, we can rely on other modalities to partially make up for the missing information.******A literature review reveals that the bulk of research to date lacks analytical foundations and has largely been motivated by special needs. Such approaches are ad hoc, unsystematic, and often, difficult to generalize. Without a proper approach, the field of information fusion will continue to be empirical at best. There is, therefore, an urgent call for a systematic resolution to multimodal information fusion. In this program, we propose a new framework based on the estimation of information entropy. Entropy is a measure of disorder or unpredictability. Instead of assuming the popular second order statistics, entropy based methods allow arbitrary distributions and higher order statistics. Entropy is closely linked to confidence level in information fusion. In general, high entropy indicates a low confidence in the corresponding modality, and vice versa. The evidence is combined with the estimated reliability through posterior entropy to provide a means to predict if the integration of multimodal information will lead to better performance, selecting the most appropriate fusion strategy for the application on hand and making a fusion system tolerant to quality changes of individual modalities, and even to the temporary corruption of a subset of the modalities. ******The expected outcome of the proposed research is to provide knowledge and tools for developing state-of-art multimedia information processing systems. The gained knowledge will generate significant societal benefits to Canada and provide enabling technology for a broad range of applications, such as new media production and delivery, preservation of cultural heritage, infrastructure planning, training of skilled professionals, marketing and advertising, e-health, smart homes, security/surveillance, gaming and e-entertainment, and creating life-like experience with immersive multimedia technology. **