CRII: RI: Learning Predictive Representations from Unlabeled Video

CRII：RI：从未标记的视频中学习预测表示

• 批准号: 1850069
• 负责人: Carl Vondrick
• 金额: $ 17.5万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-06-15 至 2022-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1850069&HistoricalAwards=false
• 关键词: CRII; RI; Learning; Predictive; Representations

The project studies computer systems that predict how objects and people will move, even when they are out-of-sight due to occlusion, for example keys inside pockets. Predictive models have the potential to enable many new applications impacting health, security, and robotics, which can improve the efficiency, safety, and welfare of the overall population. To achieve this, the research investigates computer vision algorithms that learn the visual patterns for prediction automatically from large amounts of video data. This computer software will be able to track objects obscured by occlusion, accurately represent shadows in video, and forecast object movements into the future. The project will provide research opportunities for both graduate and undergraduate students, and increase the diversity in machine intelligence research. Outcomes from this project will translate into course material to teach students in computer science and machine learning. This research focuses on robustly generalizing predictive models to the natural diversity and complexity of real-world video. While large annotated datasets fuel rapid advancements in visual scene recognition, machine understanding of events and dynamics remains challenging because the amount of knowledge required for video understanding is vast and potentially ambiguous. Instead, the investigators aim to capitalize on large amounts of raw, unlabeled video in order to create machine algorithms that efficiently learn to predict the future behaviors of events, objects, and people. Building off highly competitive frameworks from the research team and others, this project will leverage natural redundancy in unlabeled video, such as color coherency and repetitive motion, to train deep convolutional neural networks without human supervision. The research team proposes extensions to spatiotemporal memory models to handle such situations, and methods to learn representations of color constancy that will improve tracking performance. The investigators also propose analysis tools to measure and visualize the representation that emerges, enabling new methods to quantify performance in predictive models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目研究预测物体和人如何移动的计算机系统，即使它们由于遮挡而看不见，例如口袋里的钥匙。预测模型有可能实现许多影响健康、安全和机器人的新应用，从而提高整体人口的效率、安全和福利。为了实现这一目标，该研究研究了计算机视觉算法，这些算法可以从大量视频数据中自动学习用于预测的视觉模式。这种计算机软件将能够跟踪被遮挡遮挡的物体，准确地表示视频中的阴影，并预测物体未来的运动。该项目将为研究生和本科生提供研究机会，并增加机器智能研究的多样性。该项目的成果将转化为课程材料，教授学生计算机科学和机器学习。这项研究的重点是鲁棒地将预测模型推广到现实世界视频的自然多样性和复杂性。虽然大型注释数据集推动了视觉场景识别的快速发展，但对事件和动态的机器理解仍然具有挑战性，因为视频理解所需的知识量巨大且可能模糊。相反，研究人员的目标是利用大量原始的、未标记的视频来创建机器算法，这些算法可以有效地学习预测事件、物体和人的未来行为。基于研究团队和其他人的高度竞争框架，该项目将利用未标记视频中的自然冗余，例如颜色一致性和重复运动，在没有人类监督的情况下训练深度卷积神经网络。研究小组提出了时空记忆模型的扩展来处理这种情况，以及学习颜色恒定性表示的方法，这将提高跟踪性能。研究人员还提出了分析工具来衡量和可视化出现的代表性，使新的方法来量化预测模型的性能。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

项目成果

Metric Learning for Adversarial Robustness

• 发表时间: 2019-09
• 期刊: Translational Psychiatry
• 影响因子: 6.8
• 作者: Chengzhi Mao;Ziyuan Zhong;Junfeng Yang;Carl Vondrick;Baishakhi Ray

Real-Time Neural Voice Camouflage

• 发表时间: 2021-12
• 期刊: ArXiv
• 作者: Mia Chiquier;Chengzhi Mao;Carl Vondrick