Deep Weakly-Supervised Neural Networks for Cross-Domain Video Recognition and Localization

廉价传感器的普及为创新的人工智能技术和应用铺平了道路。例如，来自分布式RGB、IR、LiDAR和深度传感器的数据通常被集成在例如，移动的机器人、自动驾驶和视频监控应用，以增强感知。在这些应用中，需要具有成本效益的系统来从多个不同传感器捕获的大量数据中识别个人，对象及其行为。除了计算复杂性，跨多个不同领域的视频识别由于跨域移位、背景杂波、照明变化、遮挡等，在现实世界的场景中，传感器模态（传感器模态和操作条件）可能会退化。基于卷积神经网络的深度学习（DL）模型在许多视觉识别应用中提供了最先进的性能，然而，当在具有有限注释或没有注释的噪声数据上进行训练时，以及在源和目标（操作）数据之间存在域转移的情况下，它们的性能在现实应用中可能下降。 该研究计划的主要长期目标是研究和开发DL模型，用于在有限监督的情况下跨多个不同领域进行准确的视频识别和定位。鉴于收集和注释目标数据以进行训练的成本很高，这些模型将依赖于弱监督学习（WSL）来维持高水平的性能。具体目标包括根据两个主轴开发基于弱标记视频的深度度量学习（DML）模型-（1）跨域适应和识别，以及（2）用于解释的时空对象定位。在Axis 1中，提出了知识蒸馏和多头方法来训练DML模型，以实现跨不同源和目标域的弱监督域自适应。在轴2中，提出了全分辨率类激活映射方法和视觉变换器的方法来训练DML模型，以进行弱监督对象定位和解释。每个轴将主要从使用多个分布式RGB和/或IR相机的视频重新识别应用的角度进行检查，其中视频序列跨不同的域相关联。该计划将解决与人工智能工程相关的具体研究问题，并开发创新方法，为多个领域的最新技术做出贡献，最引人注目的是DL和计算机视觉。虽然该计划将主要侧重于基于视频的识别，但这些模型与广泛的任务和应用相关。鉴于DL最近的流行，以及它的许多商业应用，该计划将提供与学术界合作，与加拿大公司合作，培训HQP，制作高影响力出版物，并通过其他资助机制启动项目的绝佳机会。

The proliferation of inexpensive sensors are paving the way for innovative AI technologies and applications. For instance, data from distributed RGB, IR, LiDAR and depth sensors are often integrated in, e.g., mobile robotics, autonomous driving, and video surveillance applications, to enhance perception. In such applications, cost-effective systems are required to recognize individuals, objects, and their behaviours from the massive amounts of data captured over multiple different sensors. Beyond the computational complexity, video recognition across multiple different domains (sensor modalities and operational conditions) may be degraded in real-world scenarios, due to cross-domain shifts, background clutter, variations in illumination, occlusion, etc. Deep learning (DL) models based on convolutional neural networks provide state-of-the-art performance in many visual recognition applications, yet their performance can decline in real-world applications when training on noisy data with limited or no annotations, and in the presence of domain shift between source and target (operational) data. The main long-term objective of this research program is to investigate and develop DL models for accurate video recognition and localization across multiple diverse domains with limited supervision. Given the high cost of collection and annotation of target data for training, these models will rely on weakly-supervised learning (WSL) to sustain a high level of performance. The specific objective consists in developing deep metric learning (DML) models based on weakly-labeled videos according to two main axes two main axes - (1) cross-domain adaptation and recognition, and (2) spatio-temporal object localization for interpretation. In Axis 1, knowledge distillation and multi-head methods are proposed to train DML models for weakly-supervised domain adaptation across different source and target domains. In Axis 2, methods for full-resolution class activation mapping methods and vision transformers are proposed to train DML models for weakly-supervised object localization and interpretation. Each axis will mainly be examined from a perspective of video re-identification applications using multiple distributed RGB and/or IR cameras, where video sequences are associated across different domains. This program will address concrete research problems related to the engineering of AI, and develop innovative methods that contribute to the state-of-art in several fields, most notably in DL and computer vision. Although this program will mostly focus on video-based recognition, these models are relevant in a wide range of tasks and applications. Given the recent popularity of DL, and it numerous commercial applications, this program will offer excellent opportunities for collaboration with academics, partnering with Canadian companies, training HQP, producing high impact publications, and initiating projects funded through other granting mechanisms.