NRI: Real-Time Semantic Computer Vision for Co-Robotics

NRI：协作机器人的实时语义计算机视觉

• 批准号: 1637941
• 负责人: Nuno Vasconcelos
• 金额: $ 71.91万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-09-01 至 2020-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1637941&HistoricalAwards=false
• 关键词: NRI; Real; Time; Semantic; Computer

This project develops real-time object recognition algorithms that generate extensive semantic object descriptions as a side effect of recognition. This side-information includes perceived object costs, attributes (object properties), and affordances (actions afforded by objects). With these, the act of recognizing a "door knob" would automatically produce the information that this is a "flexible" object, "made of metal," which "can be grasped" and "can be twisted," but "cannot be eaten." For robotics, this information is sometimes more important than the recognition of the object itself. The project enables robots to perform zero shot learning, e.g. learn to recognize door knobs by simply being told that these are objects that "are flexible, made of metal, can be grasped and twisted but not eaten." The research has applicability in areas such as manufacturing, intelligent systems, assisted living, and homeland security. Educationally, the project provides an exciting opportunity for undergraduate research.This research develops new methods for top-down (task-driven) regularization of deep learning algorithms, though a combination of structural and loss-based regularizers. Structural regularizers constrain object and scene recognition models to guarantee speed and automatic generation of rich mid-level semantic (MLS) descriptions as a side effect of recognition. Loss-based regularizers penalize errors in the multiple semantic outputs of these models, enabling simultaneously high performance in object recognition, MLS predictions, and zero-shot learning. The resulting learning algorithms will endow robots with human-like abilities to infer rich MLS descriptions of objects and scenes as a "side effect" of object recognition and scene classification, in real-time. These contributions will be developed in the context of a new co-robotics problem, person-following unmanned aerial vehicles, where computer vision plays a mission critical role for tasks such as control and semantic motion planning but whose requirements in terms of speed and MLS inference are far superior to what is feasible today.

该项目开发实时对象识别算法，生成广泛的语义对象描述作为识别的副作用。这些附带信息包括感知对象成本、属性（对象属性）和功能（对象提供的操作）。有了这些，识别“门把手”的行为将自动产生这样的信息：这是一个“柔性”物体，“由金属制成”，“可以抓住”和“可以扭曲”，但“不能吃”。对于机器人来说，这些信息有时比识别物体本身更重要。该项目使机器人能够进行零射击学习，例如，通过简单地告诉机器人门把手是“柔性的，由金属制成的，可以抓住和扭曲，但不能吃掉”的物体，机器人就能学会识别门把手。该研究在制造业、智能系统、辅助生活和国土安全等领域具有适用性。在教育方面，该项目为本科生研究提供了一个令人兴奋的机会。本研究开发了深度学习算法的自上而下（任务驱动）正则化的新方法，尽管结合了结构和基于损失的正则化器。作为识别的副作用，结构正则器约束对象和场景识别模型以保证速度和自动生成丰富的中级语义（MLS）描述。基于损失的正则化器对这些模型的多个语义输出中的错误进行惩罚，同时在对象识别、MLS预测和零射击学习方面实现高性能。由此产生的学习算法将赋予机器人类似人类的能力，以实时推断物体和场景的丰富MLS描述，作为物体识别和场景分类的“副作用”。这些贡献将在一个新的协同机器人问题的背景下发展，即人跟随无人机，其中计算机视觉在控制和语义运动规划等任务中起着关键作用，但其在速度和MLS推理方面的要求远远优于今天的可行性。

项目成果

Semantically Consistent Regularization for Zero-Shot Recognition

• DOI: 10.1109/cvpr.2017.220
• 发表时间: 2017-04
• 期刊: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Pedro Morgado;N. Vasconcelos

Catastrophic Child's Play: Easy to Perform, Hard to Defend Adversarial Attacks

• DOI: 10.1109/cvpr.2019.00945
• 发表时间: 2019-06
• 期刊: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Chih-Hui Ho;Brandon Leung;Erik Sandström;Yen Chang;N. Vasconcelos

Deep Scene Image Classification with the MFAFVNet

• DOI: 10.1109/iccv.2017.613
• 发表时间: 2017-10
• 期刊: 2017 IEEE International Conference on Computer Vision (ICCV)
• 作者: Yunsheng Li;Mandar Dixit;N. Vasconcelos

Object based Scene Representations using Fisher Scores of Local Subspace Projections

• 发表时间: 2016-12
• 作者: Mandar Dixit;N. Vasconcelos

Self-Supervised Generation of Spatial Audio for 360 Video

• 发表时间: 2018-09
• 期刊: ArXiv
• 作者: Pedro Morgado;N. Vasconcelos;Timothy R. Langlois;Oliver Wang