Multi-Modality-Enriched Video: Potential, Strategies and Applications

视觉传感器无处不在，并已在许多不同的应用中找到了自己的方法。它们以令人难以置信的速度捕获数据，并与其他传感器集成到单一设备中。例如，典型的智能手机有一个摄像头(视觉传感器)、一个GPS传感器、一个加速计、一个陀螺仪和一个磁力计。这些整合的传感器产生多模式丰富的视频信号：视频信号以及一个或多个1-D或2-D非视频信号。非视频信号可以为视频信号处理提供附加的上下文，或者直接提供辅助信息以便于视频信号处理和通信。例如，运动传感器可以提供对运动估计以及视频压缩有用的运动数据。此外，多模式丰富的视频使我们能够开发新功能和创造新的用户体验。丰富的非视频信号具有巨大的潜力，可以改善我们基于视频进行沟通和决策的方式。然而，这种潜力的实现取决于管理、分析和从这些多通道数据集中提取有用信息的技术的进步。我们利用丰富的视觉传感器获取的不同信号模式之间的相互依赖是相当直观的。目前，还没有确定在特定视频处理任务(例如，视频压缩、检测、跟踪)中使用多于一种信号模式是否有任何好处以及使用哪种信号模式以及如何使用它的既定理论基础。在这项探索基金的研究中，将探索一条新的研究路线，以便系统地利用不同信号模式之间的相互依赖关系来进行视频压缩、后处理、检测和分类。将采用基于张量的框架来描述传感模式之间的相互作用。这项拟议的研究有望为系统分析和处理多模式丰富的视频提供新的方法。这项研究将带来更好的视频通信和更好的基于视频的跟踪、识别和分类。这项研究产生的新知识将进入对加拿大非常重要的领域的新技术和产品，如教育、医疗保健、制造业和汽车工业。这一领域的加拿大公司将受益于新的知识，以及将参与研究的高素质人员。

Visual sensors are ubiquitous and have found their ways into many different applications. They capturing data at an incredible pace and are being integrated with other sensors into single devices. For example a typical smartphone has a camera (visual sensor), a GPS sensor, an accelerometer, a gyroscope, and a magnetometer. These consolidated sensors generate multi-modality-enriched video signal: video signal as well as one or multiple 1-D or 2-D non-video signals. The non-video signals can provide additional context for video signal processing or directly provide side information to facilitate video signal processing and communication. For example a motion sensor can provide motion data useful for motion estimation and thus video compression. Moreover, multi-modality enriched video enables us to develop new functionalities and create new user experiences. The enriching non-video signals have a tremendous potential for improving the way we communicate and make decision based on video. However, realization of this potential hinges upon advancement in techniques for managing, analyzing, and extracting useful information from these multi-modality data sets. Our utilization of interdependency between different signal modality acquired by enriched visual sensors is rather intuitive. Currently, there is no established rationale for determining if there are any benefits to use more than one signal modality in a particular video processing task (e.g. video compression, detection, tracking) and which one to use and how to use it. In this Discovery Grant research, a new line of inquiry will be pursued in order to systematically utilize the inter dependencies between different signal modalities for video compression, post processing, detection and classification. A tensor based framework will be adopted that will characterize the interaction between sensing modalities. The proposed research is expected to yield new approaches for systematic analysis and processing of multi-modality enriched video. The research will lead to superior video communication and better video-based tracking, recognition and classifications. The new knowledge arising from this research will find its way into new technologies and products in areas of great importance to Canada such as education, healthcare, manufacturing and auto industry. Canadian companies in this area will benefit from the new knowledge and also the highly qualified personnel that will participate in the research.