NRI: Real-Time Semantic Computer Vision for Co-Robotics

NRI：协作机器人的实时语义计算机视觉

• 批准号: 1637941
• 负责人: Nuno Vasconcelos
• 金额: $ 71.91万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-09-01 至 2020-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1637941&HistoricalAwards=false
• 关键词: NRI; Real; Time; Semantic; Computer

This project develops real-time object recognition algorithms that generate extensive semantic object descriptions as a side effect of recognition. This side-information includes perceived object costs, attributes (object properties), and affordances (actions afforded by objects). With these, the act of recognizing a "door knob" would automatically produce the information that this is a "flexible" object, "made of metal," which "can be grasped" and "can be twisted," but "cannot be eaten." For robotics, this information is sometimes more important than the recognition of the object itself. The project enables robots to perform zero shot learning, e.g. learn to recognize door knobs by simply being told that these are objects that "are flexible, made of metal, can be grasped and twisted but not eaten." The research has applicability in areas such as manufacturing, intelligent systems, assisted living, and homeland security. Educationally, the project provides an exciting opportunity for undergraduate research.This research develops new methods for top-down (task-driven) regularization of deep learning algorithms, though a combination of structural and loss-based regularizers. Structural regularizers constrain object and scene recognition models to guarantee speed and automatic generation of rich mid-level semantic (MLS) descriptions as a side effect of recognition. Loss-based regularizers penalize errors in the multiple semantic outputs of these models, enabling simultaneously high performance in object recognition, MLS predictions, and zero-shot learning. The resulting learning algorithms will endow robots with human-like abilities to infer rich MLS descriptions of objects and scenes as a "side effect" of object recognition and scene classification, in real-time. These contributions will be developed in the context of a new co-robotics problem, person-following unmanned aerial vehicles, where computer vision plays a mission critical role for tasks such as control and semantic motion planning but whose requirements in terms of speed and MLS inference are far superior to what is feasible today.

该项目开发实时对象识别算法，生成广泛的语义对象描述作为识别的副作用。这些辅助信息包括感知的对象成本、属性（对象属性）和启示（对象提供的动作）。有了这些，识别“门把手”的行为将自动产生这样的信息：这是一个“柔性”物体，“由金属制成”，“可以抓住”和“可以扭曲”，但“不能吃”。“对于机器人来说，这些信息有时比物体本身的识别更重要。该项目使机器人能够进行零射击学习，例如，只需告诉它们这些是“灵活的，由金属制成的，可以抓住和扭曲但不能吃的”物体，就可以学会识别门把手。“这项研究在制造、智能系统、辅助生活和国土安全等领域具有适用性。在教育方面，该项目为本科生研究提供了一个令人兴奋的机会。这项研究开发了自上而下（任务驱动）深度学习算法正则化的新方法，尽管结合了结构正则化器和基于损失的正则化器。结构正则化器约束对象和场景识别模型，以保证速度和自动生成丰富的中级语义（MLS）描述作为识别的副作用。基于损失的正则化器惩罚这些模型的多个语义输出中的错误，同时实现对象识别，MLS预测和零射击学习的高性能。由此产生的学习算法将赋予机器人类似人类的能力，以实时推断对象和场景的丰富MLS描述，作为对象识别和场景分类的“副作用”。这些贡献将在一个新的合作机器人问题的背景下，人以下无人驾驶飞行器，其中计算机视觉起着使命的关键作用，如控制和语义运动规划，但其要求的速度和MLS推理是远远优于上级什么是可行的今天。

项目成果

Semantically Consistent Regularization for Zero-Shot Recognition

• DOI: 10.1109/cvpr.2017.220
• 发表时间: 2017-04
• 期刊: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Pedro Morgado;N. Vasconcelos

Catastrophic Child's Play: Easy to Perform, Hard to Defend Adversarial Attacks

• DOI: 10.1109/cvpr.2019.00945
• 发表时间: 2019-06
• 期刊: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Chih-Hui Ho;Brandon Leung;Erik Sandström;Yen Chang;N. Vasconcelos

Deep Scene Image Classification with the MFAFVNet

• DOI: 10.1109/iccv.2017.613
• 发表时间: 2017-10
• 期刊: 2017 IEEE International Conference on Computer Vision (ICCV)
• 作者: Yunsheng Li;Mandar Dixit;N. Vasconcelos

Object based Scene Representations using Fisher Scores of Local Subspace Projections

• 发表时间: 2016-12
• 作者: Mandar Dixit;N. Vasconcelos

Self-Supervised Generation of Spatial Audio for 360 Video

• 发表时间: 2018-09
• 期刊: ArXiv
• 作者: Pedro Morgado;N. Vasconcelos;Timothy R. Langlois;Oliver Wang