RI:Small: Improve Visual Tracking by Large Scale Learning, Diagnosis, and Evaluation

视频理解与分析具有广泛的应用前景。作为视频理解的基石，视觉跟踪提供了感兴趣对象的在线运动信息，例如自动驾驶中的行人行走，生物工程研究中的移动细胞，以及医疗干预中的变形导丝等等。然而，尽管基于深度学习的视觉跟踪系统最近取得了进展，但在最先进的算法和现实世界的应用之间仍然存在着巨大的差距。一种猜测是，深度学习的优势没有得到充分挖掘，特别是由于缺乏大规模的质量跟踪数据集。目前，发布的最大的全注释跟踪数据集包含的视频不到2,000个，这几乎不足以有效地学习一个健壮的跟踪模型。本课题通过直接和显式地研究跟踪算法的大规模学习来解决这一问题，旨在从精度、效率、健壮性以及泛化能力等方面提高跟踪系统的性能。所产生的数据集、基准测试、诊断工具包、跟踪算法和时间建模技术将公开可用，有望对计算机视觉及相关领域产生重大贡献。本研究的总体目标是通过大规模学习来推动视觉目标跟踪的前沿。该项目将研究活动分为三个阶段。首先，构建具有完整标注的大规模质量跟踪数据集。在这些数据集的基础上，将建立一个在线基准平台，并开发一个跟踪诊断工具包，以研究视觉跟踪中的挑战因素。这些结果将为促进视觉跟踪研究提供数据基础、测试平台和分析工具。其次，将致力于通过优化追踪器架构或整合对这些因素的预测，来提高深度追踪器对各种挑战因素的稳健性。第三，将通过两种方式开发有效的深度时间模型：一种是在时空联合CNN结构中隐式编码时间信息，另一种是开发注意力引导的双记忆模型。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Video understanding and analysis has a wide range of applications. As a cornerstone in video understanding, visual tracking provides online motion information of objects of interests, such as walking pedestrians in autonomous driving, moving cells in bioengineering study, and deforming guidewire in medical intervention, to name a few. Despite recent advances in deep learning-based visual tracking systems, however, a significant gap remains between state-of-the-art algorithms and real-world applications. A conjecture is that the advantage of deep learning is not fully explored, especially due to the lack of large-scale quality tracking datasets. Currently, the largest published fully annotated tracking dataset contains less than 2,000 videos, which are hardly sufficient for effectively learning a robust tracking model. This project confronts the issue by directly and explicitly working on large-scale learning of tracking algorithms, and aims to improving tracker systems from various aspects including accuracy, efficiency, robustness, as well as generalization capability. The produced datasets, benchmark, diagnosis toolkit, tracking algorithms and temporal modeling techniques, will be made publicly available and expected to generate significant contributions to the computer vision and related fields.The overall goal of this research is to push the frontier of visual object tracking though large-scale learning. The project divides the research activities into three thrusts. Firstly, large-scale quality tracking datasets will be constructed with full annotation. Based on such datasets, an online benchmark platform will be derived and a tracking diagnosis toolkit be developed for studying challenge factors in visual tracking. These results will provide the data basis, test beds, and analytic tools for facilitating research in visual tracking. Secondly, efforts will be devoted to improving the robustness of deep trackers against various challenge factors, by either optimizing tracker architectures or integrating predictions of these factors. Thirdly, effective deep temporal models will be developed in two ways: one implicitly encodes temporal information in joint spatial-temporal CNN structures, while the other develops attention-guided dual memory models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.