Model design for analyzing ultra-low framerate surveillance videos

该项目的总体目标是开发下一代应用于超低帧速率交通视频（低于每秒1帧）的自动场景理解技术。目标是（1）定位和识别实时交通摄像机拍摄的车辆，以及（2）开发压缩技术，将视频分析方法直接部署到摄像机上。 我的研究计划旨在将这两个组件集成到实时视频监控系统中，以便收集高级别的交通统计数据。一个重要的挑战是，交通图像来与大的照明变化，压缩文物，任意车辆方向和规模，分辨率差，各种天气条件和类间的相似性。因此，我们将利用深度卷积神经网络（CNN）与大型数据集相结合的强大功能。 然而，简单地使用现成的CNN不适合实现上述目标。首先，最先进的CNN太大，需要太多的处理能力来存储在今天的监控摄像头上。 其次，就其本质而言，交通数据集在其包含的车辆类型方面严重不平衡。因此，它们包含的汽车是其他车辆（如摩托车和公路列车）的100倍，这导致机器学习方法对某些类别的车辆的召回率较低。 此外，为该项目收集的数据集非常大（400万至500万张图像），无法手动勾勒每张图像中的每辆车。 相反，这些图像包含了元信息，包括一天中的时间、场景中的车辆类型、整体密度、全球背景以及每辆车上的2d点。因此，该研究计划旨在设计端到端的可训练系统，以适应交通数据的每周注释和不平衡性质，以及将其部署到监控摄像头上的压缩技术。这包括创新的分割和定位技术，通过强化学习进行网络超参数估计，以及具有稀疏性和正交性约束的优化方法。 这些模型有可能迅速转移到加拿大公司，并大大改善日常交通状况。 该项目还将培养2名理科硕士和3名博士。一个要求很高的多学科领域。

The overarching objective of this project is to develop the next generation of automatic scene understanding techniques applied to ultra-low frame-rate traffic videos (less than 1 frame per second). The goals are to (1) localize and recognize vehicles pictured by live traffic cameras, and (2) develop compression techniques that shall allow the deployment of video analytics methods directly onto cameras. My research program aims at the integration of these two components into real-time video surveillance systems that will allow the gathering of high-level traffic statistics.******One important challenge is the fact that traffic images come with large illumination variations, compression artifacts, arbitrary vehicle orientation and scale, poor resolution, various weather conditions and inter-class similarities. As such, we shall leverage the power of deep convolutional neural networks (CNN) combined with large datasets. However, the plain use of off-the-shell CNNs is ill-suited to reach the aforementioned goals. First, state-of-the-art CNN are too large and require too much processing power to be stored on today's surveillance cameras. Second, by their very nature, traffic datasets are heavily imbalanced in the type of vehicles they contain. As such, they contain 100 times more cars than other vehicles (like motorcycles and road trains) which cause machine-learning methods to have a low recall on certain category of vehicles. Furthermore, the dataset that will be gathered for this project is so large (4 to 5 million images) that each vehicle in each image cannot be manually outlined. Instead, the images come with meta-information containing the time of the day, the type of vehicles present in the scene, the overall density, the global context, and a 2d dot on each vehicle. This research program thus aims at designing end-to-end trainable systems adapted to the weekly-annotated and unbalanced nature of traffic data as well as compression techniques for deploying it onto surveillance cameras. This includes innovative segmentation and localization techniques, network hyperparameters estimation through reinforcement learning, and optimization methods with sparsity and orthogonality constraints. These models have the potential to be transferred rapidly to Canadian companies and greatly improve daily traffic conditions. This program will also train 2 M.Sc and 3 Ph.D. for a demanding, highly multidisciplinary field.