Collaborative Research: RI: Medium: Introspective Perception and Planning for Long-Term Autonomy

在真实的人类环境中构建和部署自主服务机器人一直是人工智能和机器人技术领域的一个长期挑战。 这些机器人可以帮助人类进行日常活动，并对社会产生变革性的影响。 然而，为了成功地实现它们的好处，机器人必须认识到它们的局限性，当不确定某个行动时，它们必须能够寻求人类的帮助。 当机器人被部署在新的环境中时，开发人员无法完全预见机器人可能会犯什么错误，这些错误的根本原因是什么，人类对机器人能力的信心可能会因这些错误而改变，以及机器人可以学习如何自主克服错误并减少对人类帮助的依赖。 该项目通过引入能力感知的自主性，为这些挑战开发了一个全面的解决方案，使机器人能够学习环境，情况和任务的哪些方面导致不同程度的成功。 当请求人类帮助时，具有能力意识的机器人可以提供证据来解释其信心水平。 当自主行动时，能力感知机器人可以假设应急行动，以帮助减少自身的不确定性，并保持高度自信的自主性。 因此，该项目改变了研究人员和从业者在部署前知识有限的非结构化环境中部署机器人的能力。 这使得机器人专业知识有限的工作人员能够在非结构化环境中更安全地部署机器人，并随着时间的推移教他们逐渐独立。 项目团队还将在UT Austin和UMass Amherst开发新的课程材料，指导本科生研究人员，特别关注代表性不足的群体，开展外展活动，向小学生介绍机器人编程，开发关于综合感知和规划研究的会议研讨会和教程，该项目通过引入满足内省感知和规划的六个核心特性的方法，解决了建立能力感知系统的需要。它们包括：1）通过依赖不同类型的一致性度量来自主监督内省感知训练的方法; 2）通过考虑感知数据中的局部和全局线索来学习识别感知错误的因果因素的方法; 3）分析动作序列和来自日志的观察以学习动作对内省感知的影响的方法; 4）认识到不同自主水平的内省规划方法，每个自主水平与自主操作的某些限制相关联; 5）认识到不同形式的人类协助的成本并且可以学习随着时间的推移最小化对人类的依赖的内省规划方法;以及6）内省规划方法，其可以从关于负面副作用的人类反馈中学习，并且可以尝试解释它们并减轻它们的影响。 该项目确定了这些不同组件之间的关键交互模式，使机器人能够自主学习围绕其局限性进行规划，并最大限度地减少对人类的依赖。 该团队进行了全面的评估，包括高保真模拟中基于单个部件的测试，以及德克萨斯大学奥斯汀分校和马萨诸塞州阿默斯特大学校园内服务移动的机器人的广泛真实部署，同时执行直接支持两所大学设施的关键挑战任务。该奖项反映了NSF的法定使命，并通过使用基金会的学术价值和更广泛的影响审查标准。

Building and deploying autonomous service robots in real human environments has been a long-standing challenge in artificial intelligence and robotics. Such robots can assist humans in everyday activities and offer a transformational impact on society. In order to successfully realize their benefits, however, robots must be cognizant of their limitations, and when uncertain about an action, they must be able to ask for human assistance. When robots are deployed in novel environments, developers cannot fully foresee what errors the robots may make, what may be the root causes of such errors, how human confidence in the robots’ abilities may change as a result of such errors, and how well the robots may learn to autonomously overcome errors and reduce the reliance on human assistance. This project develops a comprehensive solution to these challenges by introducing competence-aware autonomy, enabling robots to learn what aspects of the environment, the situation, and the task lead to varying levels of success. When asking for human assistance, a competence-aware robot can offer evidence to explain its level of confidence. When left to act autonomously, a competence-aware robot can hypothesize contingency actions to help reduce its own uncertainty and to remain autonomous with high confidence. Consequently, the project transforms the ability of researchers and practitioners to deploy robots in unstructured environments where limited knowledge is available prior to deployment. This enables workers with limited robotics expertise to deploy robots more safely in unstructured environments and teach them over time to be progressively independent. The project team will also develop new course materials at UT Austin and UMass Amherst, mentor undergraduate student researchers with special attention to underrepresented groups, perform outreach activities to introduce programming with robots to grade school students, develop conference workshops and tutorials on integrated perception and planning research, and strengthen collaborations between academia and industry.The project addresses the need to build competency-aware systems by introducing approaches to satisfy six core properties of introspective perception and planning. They include: 1) an approach to autonomously supervise the training of introspective perception by relying on different types of consistency metrics; 2) an approach to learn to identify causal factors of perception errors by considering both local and global cues in sensed data; 3) an approach to analyze sequences of actions and observations from logs to learn the impact of actions on introspective perception; 4) an introspective planning approach that is cognizant of different levels of autonomy, each associated with certain restrictions on autonomous operation; 5) an introspective planning approach that is cognizant of the cost of different forms of human assistance and can learn to minimize the reliance on humans over time; and 6) an introspective planning approach that can learn from human feedback about negative side effects and can attempt to explain them and mitigate their impact. The project identifies key patterns of interaction between these different components to enable a robot to autonomously learn to plan around its limitations and minimize the reliance on humans. The team conducts a comprehensive evaluation consisting of individual part-based testing in high-fidelity simulation, and extensive real-world deployments of service mobile robots across the University of Texas at Austin and University of Massachusetts Amherst campuses, while performing key challenge tasks that directly support the facilities of both universities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.