Wide area multi-target tracking over multiband imaging sensor networks for military applications

该博士项目旨在开发第一个用于军事环境的广域多目标跟踪系统。该系统将允许在地面和/或无人机上探测、跟踪和监视多个同时移动的目标，这些目标跨越非重叠和多种视觉传感器网络，包括红外和RGB。通过提出基于深度学习范式的整体跟踪框架来实现这一目标。我们的方法由一种多模态再识别算法和一种相机内多目标跟踪算法组成，该算法能够在传感器之间移动时保持目标的身份，并将其结合在统一的深度神经网络架构中。传感技术的进步彻底改变了军事行动，使远程监测和干预成为可能，同时减少了飞行员和士兵的危险。在这种情况下，目标探测和跟踪是战术作战和侦察所需的一项基本任务。传感器在现场的日益普及不仅增加了其覆盖范围，而且其成功的代价是增加了处理大数据流的资源。目标跟踪的自动化可以在克服人为限制的同时有效地解决这种情况。该领域的大多数现有研究假设跟踪仅限于单个传感器的视场，而不是解决在非重叠传感器网络上跟踪的复杂性。这不仅减少了覆盖范围，而且假设在任务期间没有发生视觉接触中断，从而增加了失去关键目标的机会。在这个博士项目中，我们的目标是通过提出跟踪的整体视图来解决以前的技术限制，其中目标检测，多目标跟踪和跨传感器的目标再识别在统一的框架中一起解决。待解决的场景包括观察多个物体，并在安装在地面和/或无人机上的多个非重叠摄像机（红外和RGB）之间传输。为了实现这一目标，我们提出了一种基于深度神经网络的统一广域跟踪框架。该框架将在3个工作包中开发：多波段再识别算法：在这个工作包中，我们将开发一个再识别系统，能够在目标在非重叠传感器上移动时保持目标的身份，以及在长时间遮挡后重新出现。所提出的系统基于深度学习Siamese架构，可以为该任务实现特征提取和度量学习的联合优化。考虑到军事场景的特殊性，网络将使用RGB、红外和运动通道的不同组合进行训练，以避免过度依赖视觉线索，并在传感器仅提供部分信息时允许重新识别。2. 基于LSTM和循环网络的多目标跟踪：该工作包提出了深度学习在多目标跟踪中的新应用，以解决多目标跟踪中涉及的固有数据关联和优化问题。最近在文献中提出了使用神经网络进行跟踪的非常初步的尝试，但仅限于单目标跟踪或使用模拟结果。我们的目标是将这些方法扩展到一个有效的多目标跟踪系统。3. 广域跟踪框架：在这个工作包中，之前的交付成果将结合在一个独特的深度神经架构中。由于在两个组件中使用了神经网络，这将成为可能，这些组件可以在这里连接到一个统一的架构中。这将允许端到端学习范式，从而最大限度地提高所有组件的性能。这个工作包的结果将是第一个基于深度学习的广域框架。

This PhD project aims to develop the first wide-area multi-target tracking system for military environments. This system will allow detecting, tracking and monitoring multiple simultaneous targets moving across a network of non-overlapping and diverse vision sensors, including infrared and RGB, on the ground and/or on UAVs. This goal is achieved by proposing a holistic tracking framework based on the Deep Learning Paradigm. Our approach is composed of a multimodal re-identification algorithm, able to preserve the identity of the targets when moving across sensors, and a within-camera multi-target tracking algorithm, which are combined in a unified Deep Neural Network architecture. Technological advances in sensing have fully transformed military operations, allowing remote monitoring and intervention while reducing the dangers for pilots and soldiers. In this context, target detection and tracking is a fundamental task required for tactical operations and reconnaissance. The increasing ubiquity of sensors on the field increases not only the coverage but its success at the cost of increasing the resources to process the large data flow. The automatisation of target tracking allows addressing efficiently this scenario while overcoming human limitations. Most existing research in the field assumes tracking is confined within a single sensor's field of view, rather than addressing the complexity of tracking over a network of non-overlapping sensors. This not only reduced coverage but also assumes no interruption of visual contact happens during the duration of the mission, resulting on increasing chances of losing a critical target. In this PhD project, we aim to tackle the previous limitations of the state of the art by proposing a holistic view of tracking, where target detection, multi target tracking and target re-identification across sensors are solved together in a unified framework. The scenario to be solved comprises multiple objects being observed and transiting between multiple non-overlapping cameras, both infrared and RGB, mounted on the ground and/or on UAVs. To achieve this goal, we propose a unified wide area tracking framework based on Deep Neural networks. This framework will be developed in 3 work-packages:1. Multi-band re-identification algorithm: In this work-package we will develop a re-identification system able to preserve the identity of the targets when they move across non overlapping sensors, as well as reappearances after long occlusions. The proposed system is based on Deep learning Siamese architectures that enable the join optimisation of feature extraction and metric learning for this task. Given the particularities of military scenarios, the network will be trained using different combinations of RGB, infrared and motion channels in order to avoid over-dependency on visual clues and allow the re-identification when only partial information is given by the sensor. 2. Multi target tracking based on LSTM and recurrent networks: This work-package proposes the novel use of Deep Learning for multi-target tracking for solving the inherent data association and optimisation problem involved in multi-target tracking. Very preliminary attempts to tracking using neural networks have been recently proposed in the literature, but limited to single-target tracking or using simulated results. We aim to extend those approaches to an effective multi-target tracking system. 3. Wide area tracking framework: In this work-package, the previous deliverables will be combined in a unique Deep Neural architecture. This will be possible thanks to the use of neural networks in both components that can be here joined into a unified architecture. This will allow an end-to-end learning paradigm that will maximise the performance of all the components. The result of this work-package will be the first proposed wide-area framework based on deep learning.