Model design for analyzing ultra-low framerate surveillance videos

该项目的总体目标是开发应用于超低帧速率(低于1帧/秒)的交通视频的下一代自动场景理解技术。其目标是(1)定位和识别实时交通摄像头拍摄的车辆，以及(2)开发允许将视频分析方法直接部署到摄像头上的压缩技术。我的研究计划旨在将这两个组成部分整合到实时视频监控系统中，以便收集高级交通统计数据。 一个重要的挑战是，交通图像具有较大的光照变化、压缩伪影、任意的车辆方向和比例、较低的分辨率、各种天气条件和类别间的相似性。因此，我们将利用深度卷积神经网络(CNN)与大数据集相结合的能力。然而，简单地使用现成的CNN并不适合实现上述目标。首先，最先进的CNN太大了，需要太多的处理能力来存储在今天的监控摄像头上。其次，由于交通数据本身的性质，它们所包含的车辆类型严重不平衡。因此，它们包含的汽车是其他车辆(如摩托车和公路列车)的100倍，这导致机器学习方法对特定类别的车辆召回率较低。此外，为该项目收集的数据集太大(400万至500万张图像)，无法手动勾勒出每幅图像中的每辆车的轮廓。取而代之的是，这些图像带有元信息，其中包括一天中的时间、场景中出现的车辆类型、整体密度、全球背景以及每辆车上的一个2D点。因此，这项研究计划旨在设计端到端可训练系统，以适应交通数据每周注释和不平衡的性质，以及将其部署到监控摄像机上的压缩技术。这包括创新的分割和定位技术，通过强化学习的网络超参数估计，以及具有稀疏性和正交性约束的优化方法。这些模式有可能迅速转移到加拿大公司，并极大地改善日常交通状况。该项目还将为一个要求严格的、高度多学科的领域培养2名硕士和3名博士。

The overarching objective of this project is to develop the next generation of automatic scene understanding techniques applied to ultra-low frame-rate traffic videos (less than 1 frame per second). The goals are to (1) localize and recognize vehicles pictured by live traffic cameras, and (2) develop compression techniques that shall allow the deployment of video analytics methods directly onto cameras. My research program aims at the integration of these two components into real-time video surveillance systems that will allow the gathering of high-level traffic statistics. One important challenge is the fact that traffic images come with large illumination variations, compression artifacts, arbitrary vehicle orientation and scale, poor resolution, various weather conditions and inter-class similarities. As such, we shall leverage the power of deep convolutional neural networks (CNN) combined with large datasets. However, the plain use of off-the-shell CNNs is ill-suited to reach the aforementioned goals. First, state-of-the-art CNN are too large and require too much processing power to be stored on today's surveillance cameras. Second, by their very nature, traffic datasets are heavily imbalanced in the type of vehicles they contain. As such, they contain 100 times more cars than other vehicles (like motorcycles and road trains) which cause machine-learning methods to have a low recall on certain category of vehicles. Furthermore, the dataset that will be gathered for this project is so large (4 to 5 million images) that each vehicle in each image cannot be manually outlined. Instead, the images come with meta-information containing the time of the day, the type of vehicles present in the scene, the overall density, the global context, and a 2d dot on each vehicle. This research program thus aims at designing end-to-end trainable systems adapted to the weekly-annotated and unbalanced nature of traffic data as well as compression techniques for deploying it onto surveillance cameras. This includes innovative segmentation and localization techniques, network hyperparameters estimation through reinforcement learning, and optimization methods with sparsity and orthogonality constraints. These models have the potential to be transferred rapidly to Canadian companies and greatly improve daily traffic conditions. This program will also train 2 M.Sc and 3 Ph.D. for a demanding, highly multidisciplinary field.