CRII: RI: Learning Predictive Representations from Unlabeled Video

该项目研究的计算机系统可以预测物体和人的移动方式，即使在物体和人因遮挡而看不到的情况下，例如口袋里的钥匙。预测模型有可能使许多影响健康、安全和机器人的新应用程序成为可能，从而提高整个人口的效率、安全和福利。为了实现这一点，该研究对计算机视觉算法进行了研究，这些算法可以从大量视频数据中自动学习用于预测的视觉模式。这个计算机软件将能够跟踪被遮挡遮挡的对象，准确地在视频中表示阴影，并预测未来对象的运动。该项目将为研究生和本科生提供研究机会，并增加机器智能研究的多样性。这个项目的成果将转化为课程材料，教授学生计算机科学和机器学习。这项研究致力于将预测模型稳健地推广到真实世界视频的自然多样性和复杂性。虽然大量的带注释的数据集推动了视觉场景识别的快速发展，但机器对事件和动态的理解仍然具有挑战性，因为视频理解所需的知识量是巨大的，并且可能是模糊的。相反，研究人员的目标是利用大量原始的、未标记的视频来创建机器算法，以便高效地学习预测事件、对象和人的未来行为。在研究团队和其他人的高度竞争框架的基础上，该项目将利用无标签视频中的自然冗余，如颜色一致性和重复运动，在没有人类监督的情况下训练深度卷积神经网络。研究小组建议对时空记忆模型进行扩展，以处理这种情况，并提出学习颜色恒定表示的方法，以提高跟踪性能。调查人员还提出了分析工具来衡量和可视化出现的表现，使新的方法能够量化预测模型中的表现。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The project studies computer systems that predict how objects and people will move, even when they are out-of-sight due to occlusion, for example keys inside pockets. Predictive models have the potential to enable many new applications impacting health, security, and robotics, which can improve the efficiency, safety, and welfare of the overall population. To achieve this, the research investigates computer vision algorithms that learn the visual patterns for prediction automatically from large amounts of video data. This computer software will be able to track objects obscured by occlusion, accurately represent shadows in video, and forecast object movements into the future. The project will provide research opportunities for both graduate and undergraduate students, and increase the diversity in machine intelligence research. Outcomes from this project will translate into course material to teach students in computer science and machine learning. This research focuses on robustly generalizing predictive models to the natural diversity and complexity of real-world video. While large annotated datasets fuel rapid advancements in visual scene recognition, machine understanding of events and dynamics remains challenging because the amount of knowledge required for video understanding is vast and potentially ambiguous. Instead, the investigators aim to capitalize on large amounts of raw, unlabeled video in order to create machine algorithms that efficiently learn to predict the future behaviors of events, objects, and people. Building off highly competitive frameworks from the research team and others, this project will leverage natural redundancy in unlabeled video, such as color coherency and repetitive motion, to train deep convolutional neural networks without human supervision. The research team proposes extensions to spatiotemporal memory models to handle such situations, and methods to learn representations of color constancy that will improve tracking performance. The investigators also propose analysis tools to measure and visualize the representation that emerges, enabling new methods to quantify performance in predictive models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.