NRI: INT: COLLAB: Shared Autonomy for Unstructured Underwater Environments through Vision and Language

现有的水下机器人系统通常提供两种操作模式之一-完全遥操作或完全自主。到目前为止，遥操作是最常见的，特别是涉及与环境交互的任务，如抓取和操纵。自主权仅限于非接触式调查任务和受控实验室环境。在遥控和自主之间操作的能力将提高在水下环境中执行任务的效率和效力。这项研究将开发和评估一个新的共享自主框架。这项研究利用了人类和机器人的不同性质。这项工作将减少对多名训练有素的操作员的需求。它有可能极大地降低水下任务的成本。这项研究的贡献将影响人类与机器人在各种应用中的合作方式，包括空间探索、救灾和辅助机器人。随着机器人系统作为我们海洋科学和探索的替代品发挥着越来越大的作用，利用人类和机器人互补性质的能力对科学发现变得至关重要。这项研究将开发新的模型和算法，利用多个非公度传感和控制模式来实现复杂非结构化环境中的智能共享自主。这项研究的新奇之处在于，使用自然语言和视觉作为弱监督的补充形式，使机器人能够从讲述的人类演示中机会主义地学习人类协作的感觉运动操纵策略。这些方法的基础是它们能够基于与人工操作员的交互在现场改进这些策略。总之，这些模型和算法将提高水下科学探索的效率和有效性。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Existing underwater robotic systems typically provide one of two operating modes---full teleoperation or full autonomy. Teleoperation is by far the most common, particularly for tasks involving interaction with the environment, such as grasping and manipulation. Autonomy is restricted to non-contact survey missions and to controlled laboratory settings. The ability to operate between teleoperation and autonomy will improve the efficiency and effectiveness of tasks performed in underwater environments. This research will develop and evaluate a novel shared autonomy framework. The research leverages the different nature of humans and robots. This work will reduce the need for multiple, highly trained operators. It has the potential to drastically reduce the cost of underwater missions. The contributions of this research will impact the way in which humans work together with robots within a wide variety of applications, including space exploration, disaster relief, and assistive robotics.As robotic systems play an ever-larger role as our surrogates for marine science and exploration, the ability to leverage the complementary nature of humans and robots becomes critical for scientific discovery. This research will develop new models and algorithms that exploit multiple non-commensurate sensing and control modalities to realize intelligent shared autonomy in complex unstructured environments. Novel to this research is the use of natural language and vision as complementary forms of weak supervision to enable robots to learn human-collaborative sensorimotor manipulation policies opportunistically from narrated human demonstrations. Fundamental to these methods is their ability to then refine these policies in situ based upon interaction with a human operator. Together, these models and algorithms will enhance the efficiency and effectiveness of underwater scientific exploration.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.