Deep Weakly-Supervised Neural Networks for Cross-Domain Video Recognition and Localization

用于跨域视频识别和定位的深度弱监督神经网络

• 批准号: RGPIN-2022-05397
• 负责人: Granger, Eric
• 金额: $ 2.55万
• 依托单位: École de technologie supérieure
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=753929
• 关键词: Deep; Weakly; Supervised; Neural; Networks

The proliferation of inexpensive sensors are paving the way for innovative AI technologies and applications. For instance, data from distributed RGB, IR, LiDAR and depth sensors are often integrated in, e.g., mobile robotics, autonomous driving, and video surveillance applications, to enhance perception. In such applications, cost-effective systems are required to recognize individuals, objects, and their behaviours from the massive amounts of data captured over multiple different sensors. Beyond the computational complexity, video recognition across multiple different domains (sensor modalities and operational conditions) may be degraded in real-world scenarios, due to cross-domain shifts, background clutter, variations in illumination, occlusion, etc. Deep learning (DL) models based on convolutional neural networks provide state-of-the-art performance in many visual recognition applications, yet their performance can decline in real-world applications when training on noisy data with limited or no annotations, and in the presence of domain shift between source and target (operational) data. The main long-term objective of this research program is to investigate and develop DL models for accurate video recognition and localization across multiple diverse domains with limited supervision. Given the high cost of collection and annotation of target data for training, these models will rely on weakly-supervised learning (WSL) to sustain a high level of performance. The specific objective consists in developing deep metric learning (DML) models based on weakly-labeled videos according to two main axes two main axes - (1) cross-domain adaptation and recognition, and (2) spatio-temporal object localization for interpretation. In Axis 1, knowledge distillation and multi-head methods are proposed to train DML models for weakly-supervised domain adaptation across different source and target domains. In Axis 2, methods for full-resolution class activation mapping methods and vision transformers are proposed to train DML models for weakly-supervised object localization and interpretation. Each axis will mainly be examined from a perspective of video re-identification applications using multiple distributed RGB and/or IR cameras, where video sequences are associated across different domains. This program will address concrete research problems related to the engineering of AI, and develop innovative methods that contribute to the state-of-art in several fields, most notably in DL and computer vision. Although this program will mostly focus on video-based recognition, these models are relevant in a wide range of tasks and applications. Given the recent popularity of DL, and it numerous commercial applications, this program will offer excellent opportunities for collaboration with academics, partnering with Canadian companies, training HQP, producing high impact publications, and initiating projects funded through other granting mechanisms.

廉价传感器的激增正在为创新的人工智能技术和应用铺平道路。例如，来自分布式 RGB、IR、LiDAR 和深度传感器的数据通常集成在移动机器人、自动驾驶和视频监控应用中，以增强感知。在此类应用中，需要具有成本效益的系统从多个不同传感器捕获的大量数据中识别个人、物体及其行为。除了计算复杂性之外，跨多个不同域（传感器模式和操作条件）的视频识别在现实场景中可能会由于跨域变化、背景杂波、照明变化、遮挡等而降低。基于卷积神经网络的深度学习 (DL) 模型在许多视觉识别应用中提供了最先进的性能，但在使用噪声数据进行训练时，其性能可能会在实际应用中下降 有限或没有注释，并且存在源数据和目标（操作）数据之间的域转移。 该研究项目的主要长期目标是研究和开发深度学习模型，以便在有限的监督下跨多个不同领域进行准确的视频识别和定位。鉴于训练目标数据的收集和注释成本高昂，这些模型将依赖弱监督学习（WSL）来维持高水平的性能。具体目标在于根据两个主轴（1）跨域适应和识别，以及（2）用于解释的时空对象定位，开发基于弱标记视频的深度度量学习（DML）模型。在轴 1 中，提出了知识蒸馏和多头方法来训练 DML 模型，以实现跨不同源域和目标域的弱监督域适应。在 Axis 2 中，提出了全分辨率类激活映射方法和视觉转换器来训练 DML 模型以进行弱监督对象定位和解释。每个轴将主要从使用多个分布式 RGB 和/或 IR 摄像机的视频重新识别应用的角度进行检查，其中视频序列跨不同域关联。该项目将解决与人工智能工程相关的具体研究问题，并开发有助于多个领域最先进的创新方法，尤其是深度学习和计算机视觉领域。尽管该计划主要关注基于视频的识别，但这些模型与广泛的任务和应用相关。鉴于深度学习最近的流行及其众多的商业应用，该计划将为与学术界合作、与加拿大公司合作、培训 HQP、制作高影响力出版物以及启动通过其他资助机制资助的项目提供极好的机会。