Deep Weakly-Supervised Neural Networks for Cross-Domain Video Recognition and Localization

廉价传感器的激增正在为创新的人工智能技术和应用铺平道路。例如，来自分布式RGB、IR、LiDAR和深度传感器的数据通常被集成在例如移动机器人、自动驾驶和视频监控应用中，以增强感知。在这样的应用中，需要具有成本效益的系统来从通过多个不同传感器捕获的海量数据中识别个人、对象及其行为。除了计算复杂性之外，由于跨域移动、背景杂乱、光照变化、遮挡等，在真实世界的场景中，跨多个不同域(传感器模式和操作条件)的视频识别可能会降级。基于卷积神经网络的深度学习模型在许多视觉识别应用中提供了最先进的性能，但在真实世界的应用中，当对具有有限或没有注释的噪声数据进行训练时，以及在源和目标(操作)数据之间存在域偏移的情况下，其性能可能会下降。该研究计划的主要长期目标是研究和开发数字图书馆模型，以便在有限的监督下跨多个不同的领域进行准确的视频识别和定位。考虑到收集和注释用于训练的目标数据的成本很高，这些模型将依赖弱监督学习(WSL)来维持高水平的性能。具体目标在于基于两个主轴的基于弱标记视频的深度度量学习(DML)模型，两个主轴-(1)跨域适应和识别，(2)用于解释的时空对象定位。在Axis 1中，提出了知识提取和多头方法来训练DML模型，用于跨不同源域和目标域的弱监督领域自适应。在Axis 2中，提出了全分辨率类激活映射方法和视觉转换器来训练DML模型用于弱监督目标定位和解释。每个轴将主要从使用多个分布式RGB和/或IR摄像机的视频重新识别应用的角度进行检查，其中视频序列跨不同的域相关联。该计划将解决与人工智能工程相关的具体研究问题，并开发有助于几个领域的最先进技术的创新方法，最显著的是在数字图书馆和计算机视觉方面。尽管该项目将主要关注基于视频的识别，但这些模型与广泛的任务和应用程序相关。鉴于最近数字图书馆的流行和众多商业应用，该计划将提供极好的机会，与学术界合作，与加拿大公司合作，培训HQP，制作有影响力的出版物，并启动通过其他拨款机制资助的项目。

The proliferation of inexpensive sensors are paving the way for innovative AI technologies and applications. For instance, data from distributed RGB, IR, LiDAR and depth sensors are often integrated in, e.g., mobile robotics, autonomous driving, and video surveillance applications, to enhance perception. In such applications, cost-effective systems are required to recognize individuals, objects, and their behaviours from the massive amounts of data captured over multiple different sensors. Beyond the computational complexity, video recognition across multiple different domains (sensor modalities and operational conditions) may be degraded in real-world scenarios, due to cross-domain shifts, background clutter, variations in illumination, occlusion, etc. Deep learning (DL) models based on convolutional neural networks provide state-of-the-art performance in many visual recognition applications, yet their performance can decline in real-world applications when training on noisy data with limited or no annotations, and in the presence of domain shift between source and target (operational) data. The main long-term objective of this research program is to investigate and develop DL models for accurate video recognition and localization across multiple diverse domains with limited supervision. Given the high cost of collection and annotation of target data for training, these models will rely on weakly-supervised learning (WSL) to sustain a high level of performance. The specific objective consists in developing deep metric learning (DML) models based on weakly-labeled videos according to two main axes two main axes - (1) cross-domain adaptation and recognition, and (2) spatio-temporal object localization for interpretation. In Axis 1, knowledge distillation and multi-head methods are proposed to train DML models for weakly-supervised domain adaptation across different source and target domains. In Axis 2, methods for full-resolution class activation mapping methods and vision transformers are proposed to train DML models for weakly-supervised object localization and interpretation. Each axis will mainly be examined from a perspective of video re-identification applications using multiple distributed RGB and/or IR cameras, where video sequences are associated across different domains. This program will address concrete research problems related to the engineering of AI, and develop innovative methods that contribute to the state-of-art in several fields, most notably in DL and computer vision. Although this program will mostly focus on video-based recognition, these models are relevant in a wide range of tasks and applications. Given the recent popularity of DL, and it numerous commercial applications, this program will offer excellent opportunities for collaboration with academics, partnering with Canadian companies, training HQP, producing high impact publications, and initiating projects funded through other granting mechanisms.