CHS: Medium: Leveraging Human Interaction to Efficiently Learn and Use Multimodal Object Affordances

这项研究的目标是使机器人能够有效地识别、推理和预测日常人类环境中常见物体的可视性。当机器人进入一个新的环境时，它不应该需要从头开始学习所有的领域知识。相反，它应该能够利用关于它所看到的对象的一般常识，以及通过定位交互获得的领域特定知识，来有效地推断周围环境中对象的属性和可见性。ppi的最终目标是让普通人更容易接触到机器人。项目成果将为研究界提供研究基础设施和新的数据来源，并为扩大参与和STEM教育推广创造机会。本科和研究生教育将受到影响，因为这些研究将补充pi的人工智能、机器人和人力资源研究所课程所涵盖的材料和项目。PI在实验室中有本科生研究助理的记录，PI thomas是本科生人工智能俱乐部的指导老师。这两个pi都有在科学技术领域指导和促进女性发展的悠久历史。他们将在这个项目中继续他们开源软件开发的传统；所有来自这项研究的数据都将公开。该项目包含端到端研究议程，探索如何表示、获取特定于领域的信息性知识库（pi称之为位置信息性网络（SAN）），然后将其用于复杂任务的推理。SAN表示：pi将使用马尔可夫逻辑网络建立SAN知识表示，将物理对象属性与对象属性和可视性联系起来。这种表示将作为其余工作的统一基础。*从语义知识来源构建SAN: pi将开发自动化技术，利用现有的通用语义知识资源，基于机器人所做的对象和位置观察来构建特定领域的SAN。结果将是一个马尔可夫逻辑网络，它表示关于机器人环境的抽象概念知识，包括分类标签、对象属性和可视性。*通过定位交互优化SAN：接下来，pi将开发技术，在机器人存在的环境中物理接地SAN的抽象功能概念。他们将开发用于学习对象、位置、属性的特定表示的技术，以及通过与环境和人类用户的位置交互实现支持所需的控制器。*使用SAN的功能推理：在最后的研究推力中，pi将开发算法技术，利用统一的SAN表示，使机器人能够执行高级任务规划，适应环境变化，并在多个上下文中推广领域独立的知识。在完成这项工作后，机器人将能够进入一个新的环境，并且1)使用它在场景中识别的对象来初始化一个特定领域的SAN，该SAN包含有关周围环境中对象的属性和可见性的抽象知识，2)通过探索环境和与人的交互来逐步完善生成的SAN， 3)利用生成的表示来执行环境中的复杂任务。包括预测新对象的可视性，抽象任务计划的基础，以及执行计划修复。本研究的主要贡献不是为给定领域和对象集生成特定的SAN知识库，而是提供了一种与领域无关的方法，通过该方法，机器人可以为任何新环境构建SAN。

The goal of this research is to enable robots to effectively identify, reason about and predict the affordances of common objects found in everyday human environments. When a robot enters a new environment, it should not need to learn all domain knowledge from scratch. Instead, it should be able to leverage general commonsense knowledge about the objects it sees, as well as domain-specific knowledge it acquires through situated interaction, to reason effectively about the attributes and affordances of objects in the surrounding environment. The PIs' ultimate objective is to make robots more accessible to everyday people. Project outcomes will contribute research infrastructure and novel data sources for the research community, as well as create an opportunity for broadening participation and STEM educational outreach. Undergraduate and graduate education will be impacted because the research will supplement the material and projects covered in the PIs' AI, robotics and HRI courses. The PIs have a track record of including undergraduate research assistants in their labs, and PI Thomaz serves as faculty advisor to the undergraduate AI Club. Both PIs have an extensive history of mentoring and promoting women in science and technology. And they will continue their tradition of open source software development in this project; all data deriving from this research will be made publicly available.This project encompasses an end-to-end research agenda that explores how a domain-specific affordance knowledge base, which the PIs call a Situated Affordance Network (SAN), can be represented, acquired, and then used for reasoning about complex tasks.* SAN Representation: The PIs will use Markov logic networks to establish the SAN knowledge representation, which relates physical object properties to object attributes and affordances. This representation will serve as the unifying foundation for the remainder of the work.* SAN construction from semantic knowledge sources: The PIs will develop automated techniques for leveraging existing, general purpose semantic knowledge resources to construct a domain-specific SAN based on object and location observations made by the robot. The outcome will be a Markov logic network that represents abstract conceptual knowledge about the robot's environment, including categorical labels, object attributes and affordances.* SAN refinement through situated interaction: Next, the PIs will develop techniques for physically grounding the SAN's abstract affordance concepts in the environment in which the robot exists. They will develop techniques for learning specific representations of objects, locations, attributes, and the controllers needed to achieve affordances through situated interaction with the environment and with the human user.* Affordance reasoning using SAN: In the final research thrust, the PIs will develop algorithmic techniques that leverage the unified SAN representation to enable the robot to perform high level task planning, adapt to changes in the environment and generalize domain-independent knowledge across multiple contexts.At the completion of this work, a robot will be able to enter a novel environment, and 1) use objects that it recognizes in the scene to initialize a domain-specific SAN that contains abstract knowledge about the attributes and affordances of objects in the surrounding environment, 2) incrementally refine the resulting SAN through exploration of the environment and interaction with a human, and 3) leverage the resulting representation to perform complex tasks in the environment, including prediction of the affordances of novel objects, grounding of abstract task plans, and performing plan repair. The main contribution of this research is not the specific SAN knowledge base that has been generated for a given domain and object set, but rather the domain-independent method by which a robot can construct a SAN for any new environment.