NRI: FND: Towards Scalable and Self-Aware Robotic Perception

机器人视觉系统应该是快速的，以提高机器人对视觉世界中事件的反应时间，能够同时解决多个视觉问题，并意识到它们的局限性。这些特性对于机器人的安全和协作至关重要。更快的反应时间(例如，汽车检测障碍物的速度更快，在撞到它们之前有更大的停车空间)和自我意识(例如，机器人应该选择停下来，在它认为太难成功的情况下操作)增强了安全性。协作通过可伸缩性(允许合作机器人解决更多问题，因此行为更像人类合作者)和自我意识(简化了人类与具有不同技能的机器人或机器人团队之间的任务分工)而得到加强。然而，这些性质并不是计算机视觉研究的重点，计算机视觉研究主要解决解决单个任务的网络设计，通常需要大量计算和相对较低的帧速率，并且简单地尝试处理所有示例，而不考虑它们有多难。该项目解决了所有这些挑战，为更高效、可扩展和自我感知的新一代机器人感知系统奠定了基础。这项研究在社会相关领域具有适用性，如制造业、自动驾驶汽车、智能系统、辅助生活、国土安全等。在教育方面，该项目将为研究生和本科生的研究提供令人兴奋的机会。该项目追求的是由几个综合贡献组成的研究议程，这些贡献促进了机器人视觉深度学习的最新水平。这包括1)新的神经网络量化技术，解决了网络权重和激活的量化，导致可以完全用二进制运算实现的深度学习模型，显著提高了所有人工智能计算的速度；2)利用广泛的参数共享来实现任务生态的可扩展推理的新的网络族，大大增加了可以高速缓存在处理器中的网络的数量，从而增加了机器人可以同时解决的视觉问题的数量；3)用于自我意识深度学习的新网络架构，能够评估每个示例的难度、预测失败并拒绝处理太难的示例，以降低发生灾难性错误的可能性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Robot vision systems should be fast, to enhance the reaction times of robots to events in the visual world, capable of solving multiple vision problems simultaneously, and aware of their limitations. These properties are critical for robotic safety and collaboration. Safety is enhanced by faster reaction times (e.g. a car faster to detect obstacles has more room to stop before hitting them) and self-awareness (e.g., a robot should choose to stop to operate in situations that it deems too hard to be successful in). Collaboration is enhanced by scalability (which allows co-robots to solve more problems and thus behave more like human collaborators) and self-awareness (which simplifies the division of tasks between humans and robots, or teams of robots, with different skills). However, these properties have not been the focus of computer vision research, which has mostly addressed the design of networks that solve single tasks, usually requiring heavy computation and relatively low frame rates, and simply attempt to process all examples without any consideration for how difficult they are. This project addresses all these challenges, laying the foundation for a new generation of robotic perception systems that are more efficient, scalable, and self-aware. The research has applicability in areas of societal relevance, such as manufacturing, self-driving vehicles, intelligent systems, assisted living, homeland security, etc. Educationally, the project will provide exciting opportunities for both graduate and undergraduate research.This project pursues a research agenda composed of several integrated contributions that advance the state of the art in deep learning for robotic vison. This includes 1) novel neural network quantization techniques that address the quantization of both network weights and activations, leading to deep learning models that can be fully implemented with binary operations, significantly enhancing the speed of all AI computations; 2) new families of networks that exploit extensive parameter sharing to achieve scalable inference for task ecologies, substantially increasing the number of networks that can be cached in a processor and, therefore, the number of vision problems that can be solved simultaneously by a robot; 3) new network architectures for self-aware deep learning, capable of assessing the difficulty of each example, predicting failures, and refusing to process examples that are too difficult, so as to mitigate the possibility of catastrophic errors.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.