Developing Information Theoretic Tools for Multimedia Multimodal Information Processing

多模态信息融合分析从几种类型的传感数据或模态中提取的信号，通常会导致比从单一模态中获得的性能更好和更可靠的性能。在许多应用中，特别是在生物识别和多媒体信息处理中，对连贯地理解和处理来自不同模态的信息的需求变得越来越高。如果没有一个令人满意的解决方案，多模态信息融合，人类掌握信息时代将仍然是一个遥不可及的梦想。多模态融合有两个主要的优点，使得多模态系统更加可靠：1）嵌入在不同模态中的信息通常是互补的，多模态处理比单一模态（例如人类生物计算机器）具有更多的信息;以及2）当一种模态被噪声或环境破坏时，我们可以依靠其他方式来部分弥补缺失的信息。一项文献审查表明，迄今为止的大部分研究缺乏分析基础，主要是出于特殊需要。这种方法是临时的、不系统的，而且往往难以推广。如果没有适当的方法，信息融合领域将继续是经验主义的。因此，迫切需要系统地解决多模态信息融合问题。在这个计划中，我们提出了一个新的框架的基础上估计的信息熵。熵是无序或不可预测性的度量。而不是假设流行的二阶统计量，基于熵的方法允许任意分布和高阶统计量。在信息融合中，熵与置信度密切相关。通常，高熵指示对应模态的低置信度，反之亦然。证据相结合的估计可靠性，通过后熵提供一种手段来预测，如果多模态信息的整合将导致更好的性能，选择最合适的融合策略，手头上的应用程序，并使融合系统容忍个别模态的质量变化，甚至暂时腐败的一个子集的模态。** 拟议研究的预期成果是为开发最先进的多媒体信息处理系统提供知识和工具。所获得的知识将为加拿大带来重大的社会效益，并为广泛的应用提供技术支持，如新媒体制作和交付、文化遗产保护、基础设施规划、熟练专业人员培训、营销和广告、电子健康、智能家居、安全/监控、游戏和电子娱乐，以及利用沉浸式多媒体技术创造逼真的体验。 **

Multimodal information fusion analyzes signals extracted from several types of sensory data or modalities and normally leads to better and more reliable performance than that obtained from a single modality. The need to coherently understand and process information from different modalities is becoming more and more demanding in many applications, particularly in biometrics and in multimedia information processing. Without a satisfactory resolution to multimodal information fusion, human mastery of the information era will remain an elusive dream.******There are two major advantages for multimodal fusion that make multimodal systems more reliable: 1) the information embedded in different modalities is usually complementary and multimodal processing is more informative than single modality, e.g. the human biological computing machine; and 2) When one modality becomes corrupted by noise or the environment, we can rely on other modalities to partially make up for the missing information.******A literature review reveals that the bulk of research to date lacks analytical foundations and has largely been motivated by special needs. Such approaches are ad hoc, unsystematic, and often, difficult to generalize. Without a proper approach, the field of information fusion will continue to be empirical at best. There is, therefore, an urgent call for a systematic resolution to multimodal information fusion. In this program, we propose a new framework based on the estimation of information entropy. Entropy is a measure of disorder or unpredictability. Instead of assuming the popular second order statistics, entropy based methods allow arbitrary distributions and higher order statistics. Entropy is closely linked to confidence level in information fusion. In general, high entropy indicates a low confidence in the corresponding modality, and vice versa. The evidence is combined with the estimated reliability through posterior entropy to provide a means to predict if the integration of multimodal information will lead to better performance, selecting the most appropriate fusion strategy for the application on hand and making a fusion system tolerant to quality changes of individual modalities, and even to the temporary corruption of a subset of the modalities. ******The expected outcome of the proposed research is to provide knowledge and tools for developing state-of-art multimedia information processing systems. The gained knowledge will generate significant societal benefits to Canada and provide enabling technology for a broad range of applications, such as new media production and delivery, preservation of cultural heritage, infrastructure planning, training of skilled professionals, marketing and advertising, e-health, smart homes, security/surveillance, gaming and e-entertainment, and creating life-like experience with immersive multimedia technology. **