NRI: FND: Towards Scalable and Self-Aware Robotic Perception

机器人视觉系统应该是快速的，以提高机器人对视觉世界事件的反应时间，能够同时解决多个视觉问题，并意识到它们的局限性。这些特性对于机器人的安全和协作至关重要。更快的反应时间（例如，一辆汽车更快地检测到障碍物，在撞到它们之前有更多的空间停下来）和自我意识（例如，机器人在它认为难以成功的情况下应该选择停下来）提高了安全性。可扩展性（允许协作机器人解决更多问题，从而表现得更像人类合作者）和自我意识（简化具有不同技能的人与机器人或机器人团队之间的任务划分）增强了协作。然而，这些属性并不是计算机视觉研究的焦点，它主要解决解决单个任务的网络设计，通常需要大量的计算和相对较低的帧率，并且简单地尝试处理所有示例，而不考虑它们有多困难。该项目解决了所有这些挑战，为更高效、可扩展和自我意识的新一代机器人感知系统奠定了基础。该研究在制造业、自动驾驶汽车、智能系统、辅助生活、国土安全等社会相关领域具有适用性。在教育方面，该项目将为研究生和本科生的研究提供令人兴奋的机会。该项目的研究议程由几个综合贡献组成，这些贡献推动了机器人视觉深度学习的最新进展。这包括1)新颖的神经网络量化技术，该技术解决了网络权重和激活的量化问题，从而产生了可以用二进制操作完全实现的深度学习模型，显著提高了所有人工智能计算的速度；2)利用广泛的参数共享来实现任务生态的可扩展推理的新网络家族，大大增加了可以在处理器中缓存的网络数量，从而增加了机器人可以同时解决的视觉问题的数量；3)用于自我意识深度学习的新网络架构，能够评估每个示例的难度，预测失败，并拒绝处理过于困难的示例，从而降低灾难性错误的可能性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Robot vision systems should be fast, to enhance the reaction times of robots to events in the visual world, capable of solving multiple vision problems simultaneously, and aware of their limitations. These properties are critical for robotic safety and collaboration. Safety is enhanced by faster reaction times (e.g. a car faster to detect obstacles has more room to stop before hitting them) and self-awareness (e.g., a robot should choose to stop to operate in situations that it deems too hard to be successful in). Collaboration is enhanced by scalability (which allows co-robots to solve more problems and thus behave more like human collaborators) and self-awareness (which simplifies the division of tasks between humans and robots, or teams of robots, with different skills). However, these properties have not been the focus of computer vision research, which has mostly addressed the design of networks that solve single tasks, usually requiring heavy computation and relatively low frame rates, and simply attempt to process all examples without any consideration for how difficult they are. This project addresses all these challenges, laying the foundation for a new generation of robotic perception systems that are more efficient, scalable, and self-aware. The research has applicability in areas of societal relevance, such as manufacturing, self-driving vehicles, intelligent systems, assisted living, homeland security, etc. Educationally, the project will provide exciting opportunities for both graduate and undergraduate research.This project pursues a research agenda composed of several integrated contributions that advance the state of the art in deep learning for robotic vison. This includes 1) novel neural network quantization techniques that address the quantization of both network weights and activations, leading to deep learning models that can be fully implemented with binary operations, significantly enhancing the speed of all AI computations; 2) new families of networks that exploit extensive parameter sharing to achieve scalable inference for task ecologies, substantially increasing the number of networks that can be cached in a processor and, therefore, the number of vision problems that can be solved simultaneously by a robot; 3) new network architectures for self-aware deep learning, capable of assessing the difficulty of each example, predicting failures, and refusing to process examples that are too difficult, so as to mitigate the possibility of catastrophic errors.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.