NRI: FND: Learning Visual Dynamics from Interaction

NRI：FND：从交互中学习视觉动力学

• 批准号: 1925157
• 负责人: Carl Vondrick
• 金额: $ 75万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1925157&HistoricalAwards=false
• 关键词: NRI; FND; Learning; Visual; Dynamics

This project studies robots that utilize the nearby and available physical objects to perform tasks, such as building a bridge out of miscellaneous rubble in a disaster area. Resourceful robots have the potential to enable many new applications in emergency response, healthcare, and manufacturing, which will improve the welfare, security, and efficiency of the overall population. The research investigates how the patterns between multiple senses, such as vision, sound, and touch, will help teach the robot to solve interaction tasks without needing a human teacher, which is expected to improve the flexibility and versatility of autonomous robots. The project will provide research and educational opportunities for both graduate and undergraduate students in computer science and mechanical engineering. Outcomes from this research will translate into new educational materials in computer vision, machine learning, and robotics. This research investigates robots that interact with realistic environments in order to learn reusable representations for navigation and manipulation tasks. While there has been significant advancements leveraging machine learning for computer vision and robotics problems, a central challenge in both fields is generalizing to the realistic complexity and diversity of the physical world. Although simulation has proved instrumental in developing platforms for machine interaction, the unconstrained world is vast, making it computationally difficult to simulate. Instead, the investigators aim to capitalize on the inherent structure of physical environments through the natural synchronization of modalities and context to efficiently learn self-supervised representations and policies for interaction with unconstrained environments. The investigators also plan several evaluations to analyze the generalization capabilities of such algorithms.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目研究利用附近可用的物理物体执行任务的机器人，例如在灾区用杂乱的瓦砾建造一座桥。足智多谋的机器人具有在应急响应、医疗保健和制造业中实现许多新应用的潜力，这将提高整个人口的福利、安全和效率。这项研究调查了视觉、声音和触觉等多种感官之间的模式如何帮助教机器人在不需要人类教师的情况下解决交互任务，这有望提高自主机器人的灵活性和多功能性。该项目将为计算机科学和机械工程的研究生和本科生提供研究和教育机会。这项研究的结果将转化为计算机视觉、机器学习和机器人学的新教材。这项研究调查了与现实环境交互的机器人，以便学习导航和操作任务的可重复使用的表示法。虽然利用机器学习解决计算机视觉和机器人问题已经取得了重大进展，但这两个领域的一个核心挑战是将其推广到物理世界的现实复杂性和多样性。尽管模拟已被证明有助于开发机器交互的平台，但不受约束的世界是广阔的，这使得模拟在计算上很困难。相反，研究人员的目标是通过模态和上下文的自然同步来利用物理环境的内在结构，以有效地学习自我监督的表示法和策略，以便与不受限制的环境进行交互。调查人员还计划进行几次评估，以分析此类算法的泛化能力。这一裁决反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

SurfsUp: Learning Fluid Simulation for Novel Surfaces

• DOI: 10.1109/iccv51070.2023.01308
• 发表时间: 2023-04
• 期刊: 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: Arjun Mani;I. Chandratreya;Elliot Creager;Carl Vondrick;R. Zemel

Beyond Categorical Label Representations for Image Classification

• 发表时间: 2021-04
• 期刊: ArXiv
• 作者: Boyuan Chen-;Yu Li;Sunand Raghupathi;H. Lipson

Smile Like You Mean It: Driving Animatronic Robotic Face with Learned Models

• DOI: 10.1109/icra48506.2021.9560797
• 发表时间: 2021-05
• 期刊: 2021 IEEE International Conference on Robotics and Automation (ICRA)
• 作者: Boyuan Chen-;Yuhang Hu;Lianfeng Li;S. Cummings;Hod Lipson

Muscles in Action

• DOI: 10.1109/iccv51070.2023.02019
• 发表时间: 2022-12
• 期刊: 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: Mia Chiquier;Carl Vondrick

Learning the Predictability of the Future

• DOI: 10.1109/cvpr46437.2021.01242
• 发表时间: 2021-01
• 期刊: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: D'idac Sur'is;Ruoshi Liu;Carl Vondrick