Collaborative Research: RI: Medium: Introspective Perception and Planning for Long-Term Autonomy

在真实的人类环境中构建和部署自主服务机器人一直是人工智能和机器人学的长期挑战。这种机器人可以帮助人类进行日常活动，并对社会产生变革性的影响。然而，为了成功地实现它们的好处，机器人必须认识到它们的局限性，当不确定一个动作时，它们必须能够请求人类的帮助。当机器人部署在新的环境中时，开发人员无法完全预见机器人可能会犯什么错误，这些错误的根本原因可能是什么，人类对机器人能力的信心可能会因这些错误而发生变化，以及机器人学习自主克服错误和减少对人类帮助的依赖程度有多好。该项目通过引入能力感知自主性来开发针对这些挑战的全面解决方案，使机器人能够了解环境、情况和任务的哪些方面会导致不同程度的成功。当请求人类帮助时，有能力的机器人可以提供证据来解释它的信心水平。当让机器人自主行动时，感知能力的机器人可以假设应急行动，以帮助减少自身的不确定性，并保持高度自信的自主。因此，该项目改变了研究人员和从业者在非结构化环境中部署机器人的能力，在非结构化环境中，在部署之前可以获得有限的知识。这使机器人专业知识有限的工人能够在非结构化环境中更安全地部署机器人，并随着时间的推移教会它们逐渐独立。项目团队还将在德克萨斯大学奥斯汀分校和马萨诸塞州阿默斯特分校开发新的课程材料，指导本科学生研究人员，特别关注代表性不足的群体，开展推广活动，向在校学生介绍机器人编程，为学生提供综合感知和规划研究的会议研讨会和教程，并加强学术界和产业界之间的合作。该项目通过引入满足内省感知和规划的六个核心特性的方法，解决了建立能力感知系统的需求。它们包括：1)依靠不同类型的一致性度量自主监督内省知觉训练的方法；2)通过考虑感知数据中的局部和全局线索来学习识别知觉错误的原因的方法；3)从日志中分析行动和观察序列以了解行动对内省知觉的影响的方法；4)认识到不同程度的自主性的内省规划方法，每个自主性水平都与对自主操作的某些限制有关；5)认识到不同形式的人类帮助的成本并能够学会随着时间的推移将对人类的依赖降到最低的内省规划方法；以及6)一种内省的规划方法，可以从人类对负面副作用的反馈中学习，并试图解释它们并减轻它们的影响。该项目确定了这些不同组件之间交互的关键模式，使机器人能够自主学习绕过其限制进行规划，并将对人类的依赖降至最低。该团队在德克萨斯大学奥斯汀分校和马萨诸塞大学阿默斯特分校进行了全面的评估，包括在高保真模拟中基于单个部件的测试，以及在德克萨斯大学奥斯汀分校和马萨诸塞大学阿默斯特分校广泛部署服务移动机器人，同时执行直接支持这两所大学设施的关键挑战任务。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Building and deploying autonomous service robots in real human environments has been a long-standing challenge in artificial intelligence and robotics. Such robots can assist humans in everyday activities and offer a transformational impact on society. In order to successfully realize their benefits, however, robots must be cognizant of their limitations, and when uncertain about an action, they must be able to ask for human assistance. When robots are deployed in novel environments, developers cannot fully foresee what errors the robots may make, what may be the root causes of such errors, how human confidence in the robots’ abilities may change as a result of such errors, and how well the robots may learn to autonomously overcome errors and reduce the reliance on human assistance. This project develops a comprehensive solution to these challenges by introducing competence-aware autonomy, enabling robots to learn what aspects of the environment, the situation, and the task lead to varying levels of success. When asking for human assistance, a competence-aware robot can offer evidence to explain its level of confidence. When left to act autonomously, a competence-aware robot can hypothesize contingency actions to help reduce its own uncertainty and to remain autonomous with high confidence. Consequently, the project transforms the ability of researchers and practitioners to deploy robots in unstructured environments where limited knowledge is available prior to deployment. This enables workers with limited robotics expertise to deploy robots more safely in unstructured environments and teach them over time to be progressively independent. The project team will also develop new course materials at UT Austin and UMass Amherst, mentor undergraduate student researchers with special attention to underrepresented groups, perform outreach activities to introduce programming with robots to grade school students, develop conference workshops and tutorials on integrated perception and planning research, and strengthen collaborations between academia and industry.The project addresses the need to build competency-aware systems by introducing approaches to satisfy six core properties of introspective perception and planning. They include: 1) an approach to autonomously supervise the training of introspective perception by relying on different types of consistency metrics; 2) an approach to learn to identify causal factors of perception errors by considering both local and global cues in sensed data; 3) an approach to analyze sequences of actions and observations from logs to learn the impact of actions on introspective perception; 4) an introspective planning approach that is cognizant of different levels of autonomy, each associated with certain restrictions on autonomous operation; 5) an introspective planning approach that is cognizant of the cost of different forms of human assistance and can learn to minimize the reliance on humans over time; and 6) an introspective planning approach that can learn from human feedback about negative side effects and can attempt to explain them and mitigate their impact. The project identifies key patterns of interaction between these different components to enable a robot to autonomously learn to plan around its limitations and minimize the reliance on humans. The team conducts a comprehensive evaluation consisting of individual part-based testing in high-fidelity simulation, and extensive real-world deployments of service mobile robots across the University of Texas at Austin and University of Massachusetts Amherst campuses, while performing key challenge tasks that directly support the facilities of both universities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.