NRI: Real-Time Semantic Computer Vision for Co-Robotics

该项目开发了实时对象识别算法，该算法生成大量的语义对象描述作为识别的副作用。该辅助信息包括感知的对象成本、属性(对象属性)和可负担性(对象所承担的动作)。有了这些，识别“门把手”的动作会自动产生信息，这是一个“灵活的”物体，“由金属制成”，“可以抓住”，“可以扭曲”，但“不能吃”。对于机器人来说，这种信息有时比物体本身的识别更重要。该项目使机器人能够进行零射击学习，例如，只需被告知门把手“灵活，由金属制成，可以抓住和扭曲，但不能吃”，就能学会识别门把手。这项研究在制造业、智能系统、辅助生活和国土安全等领域具有适用性。在教育方面，该项目为本科生的研究提供了一个令人兴奋的机会。该研究开发了自顶向下(任务驱动)的深度学习算法正则化的新方法，通过结构正则化和基于损失的正则化的组合。结构正则化约束对象和场景识别模型以保证作为识别的副作用的丰富的中层语义(MLS)描述的速度和自动生成。基于损失的正则化器惩罚这些模型的多个语义输出中的错误，同时在目标识别、最小二乘预测和零射击学习中实现高性能。由此产生的学习算法将赋予机器人类似人类的能力，以实时推断对象和场景的丰富MLS描述，作为对象识别和场景分类的“副作用”。这些贡献将在一个新的合作机器人问题的背景下发展出来，即跟随人的无人机，其中计算机视觉在控制和语义运动规划等任务中发挥着关键作用，但其在速度和MLS推理方面的要求远远优于当今可行的要求。

This project develops real-time object recognition algorithms that generate extensive semantic object descriptions as a side effect of recognition. This side-information includes perceived object costs, attributes (object properties), and affordances (actions afforded by objects). With these, the act of recognizing a "door knob" would automatically produce the information that this is a "flexible" object, "made of metal," which "can be grasped" and "can be twisted," but "cannot be eaten." For robotics, this information is sometimes more important than the recognition of the object itself. The project enables robots to perform zero shot learning, e.g. learn to recognize door knobs by simply being told that these are objects that "are flexible, made of metal, can be grasped and twisted but not eaten." The research has applicability in areas such as manufacturing, intelligent systems, assisted living, and homeland security. Educationally, the project provides an exciting opportunity for undergraduate research.This research develops new methods for top-down (task-driven) regularization of deep learning algorithms, though a combination of structural and loss-based regularizers. Structural regularizers constrain object and scene recognition models to guarantee speed and automatic generation of rich mid-level semantic (MLS) descriptions as a side effect of recognition. Loss-based regularizers penalize errors in the multiple semantic outputs of these models, enabling simultaneously high performance in object recognition, MLS predictions, and zero-shot learning. The resulting learning algorithms will endow robots with human-like abilities to infer rich MLS descriptions of objects and scenes as a "side effect" of object recognition and scene classification, in real-time. These contributions will be developed in the context of a new co-robotics problem, person-following unmanned aerial vehicles, where computer vision plays a mission critical role for tasks such as control and semantic motion planning but whose requirements in terms of speed and MLS inference are far superior to what is feasible today.