Wide area multi-target tracking over multiband imaging sensor networks for military applications

该博士项目旨在开发第一个用于军事环境的广域多目标跟踪系统。该系统将允许在地面和/或无人机上通过非重叠和多样化的视觉传感器网络(包括红外和RGB)检测、跟踪和监控多个同时移动的目标。这一目标是通过提出一个基于深度学习范式的整体跟踪框架来实现的。我们的方法由多模式重新识别算法和摄像机内多目标跟踪算法组成，这两种算法结合在一个统一的深度神经网络体系结构中。传感技术的进步彻底改变了军事行动，允许远程监控和干预，同时降低了飞行员和士兵的危险。在这种背景下，目标检测和跟踪是战术作战和侦察所需要的一项基本任务。传感器在现场的日益普遍不仅增加了覆盖范围，而且以增加处理大量数据流的资源为代价增加了其成功。目标跟踪的自动化允许有效地处理这种情况，同时克服人类的限制。该领域的大多数现有研究都假设跟踪局限在单个传感器的视野内，而不是解决在非重叠传感器网络上进行跟踪的复杂性。这不仅减少了覆盖范围，而且假设在任务期间不会中断视觉联系，从而增加了失去关键目标的机会。在这个博士项目中，我们的目标是通过提出一种跟踪的整体观点来解决先前技术水平的局限性，其中目标检测、多目标跟踪和跨传感器的目标重新识别在一个统一的框架中被一起解决。要解决的场景包括在安装在地面和/或无人机上的多个非重叠的红外和RGB摄像机之间观察和过渡多个对象。为了实现这一目标，我们提出了一种基于深度神经网络的统一广域跟踪框架。该框架将分3个工作包来开发：1.多波段再识别算法：在这个工作包中，我们将开发一个重新识别系统，该系统能够保持目标在非重叠传感器上移动时的身份，以及在长时间遮挡后的重现。提出的系统基于深度学习暹罗体系结构，使特征提取和度量学习的联合优化能够完成这一任务。鉴于军事场景的特殊性，网络将使用RGB、红外和运动通道的不同组合进行训练，以避免过度依赖视觉线索，并允许在传感器仅给出部分信息时进行重新识别。2.基于LSTM和递归网络的多目标跟踪：该工作包提出了将深度学习用于多目标跟踪的新方法，以解决多目标跟踪中固有的数据关联和优化问题。最近在文献中提出了使用神经网络进行跟踪的非常初步的尝试，但仅限于单目标跟踪或使用模拟结果。我们的目标是将这些方法扩展到一个有效的多目标跟踪系统。3.广域跟踪框架：在这个工作包中，以前的交付成果将合并到一个独特的深度神经体系结构中。这将是可能的，这要归功于在这两个组件中使用神经网络，这些组件可以在这里连接到一个统一的架构中。这将允许端到端的学习范例，使所有组件的性能最大化。这一一揽子工作的结果将是提出的第一个基于深度学习的广域框架。

This PhD project aims to develop the first wide-area multi-target tracking system for military environments. This system will allow detecting, tracking and monitoring multiple simultaneous targets moving across a network of non-overlapping and diverse vision sensors, including infrared and RGB, on the ground and/or on UAVs. This goal is achieved by proposing a holistic tracking framework based on the Deep Learning Paradigm. Our approach is composed of a multimodal re-identification algorithm, able to preserve the identity of the targets when moving across sensors, and a within-camera multi-target tracking algorithm, which are combined in a unified Deep Neural Network architecture. Technological advances in sensing have fully transformed military operations, allowing remote monitoring and intervention while reducing the dangers for pilots and soldiers. In this context, target detection and tracking is a fundamental task required for tactical operations and reconnaissance. The increasing ubiquity of sensors on the field increases not only the coverage but its success at the cost of increasing the resources to process the large data flow. The automatisation of target tracking allows addressing efficiently this scenario while overcoming human limitations. Most existing research in the field assumes tracking is confined within a single sensor's field of view, rather than addressing the complexity of tracking over a network of non-overlapping sensors. This not only reduced coverage but also assumes no interruption of visual contact happens during the duration of the mission, resulting on increasing chances of losing a critical target. In this PhD project, we aim to tackle the previous limitations of the state of the art by proposing a holistic view of tracking, where target detection, multi target tracking and target re-identification across sensors are solved together in a unified framework. The scenario to be solved comprises multiple objects being observed and transiting between multiple non-overlapping cameras, both infrared and RGB, mounted on the ground and/or on UAVs. To achieve this goal, we propose a unified wide area tracking framework based on Deep Neural networks. This framework will be developed in 3 work-packages:1. Multi-band re-identification algorithm: In this work-package we will develop a re-identification system able to preserve the identity of the targets when they move across non overlapping sensors, as well as reappearances after long occlusions. The proposed system is based on Deep learning Siamese architectures that enable the join optimisation of feature extraction and metric learning for this task. Given the particularities of military scenarios, the network will be trained using different combinations of RGB, infrared and motion channels in order to avoid over-dependency on visual clues and allow the re-identification when only partial information is given by the sensor. 2. Multi target tracking based on LSTM and recurrent networks: This work-package proposes the novel use of Deep Learning for multi-target tracking for solving the inherent data association and optimisation problem involved in multi-target tracking. Very preliminary attempts to tracking using neural networks have been recently proposed in the literature, but limited to single-target tracking or using simulated results. We aim to extend those approaches to an effective multi-target tracking system. 3. Wide area tracking framework: In this work-package, the previous deliverables will be combined in a unique Deep Neural architecture. This will be possible thanks to the use of neural networks in both components that can be here joined into a unified architecture. This will allow an end-to-end learning paradigm that will maximise the performance of all the components. The result of this work-package will be the first proposed wide-area framework based on deep learning.