NRI: INT: COLLAB: Shared Autonomy for Unstructured Underwater Environments through Vision and Language

现有的水下机器人系统通常提供两种操作模式之一——完全远程操作或完全自主。远距操作是迄今为止最常见的，特别是涉及与环境交互的任务，如抓取和操作。自主性仅限于非接触式调查任务和受控的实验室设置。在远程操作和自主之间操作的能力将提高在水下环境中执行任务的效率和有效性。本研究将开发和评估一个新的共享自治框架。这项研究利用了人类和机器人的不同天性。这项工作将减少对多名训练有素的操作人员的需求。它有可能大幅降低水下任务的成本。这项研究的贡献将影响人类在各种应用中与机器人一起工作的方式，包括太空探索、救灾和辅助机器人。随着机器人系统作为海洋科学和探索的替代品发挥着越来越大的作用，利用人类和机器人互补性的能力对科学发现至关重要。本研究将开发新的模型和算法，利用多种非相称的传感和控制方式，在复杂的非结构化环境中实现智能共享自治。本研究的新颖之处在于使用自然语言和视觉作为弱监督的补充形式，使机器人能够从叙述的人类演示中机会主义地学习人类合作的感觉运动操作策略。这些方法的基础是它们能够根据与人工操作员的交互在现场改进这些策略。这些模型和算法将共同提高水下科学探测的效率和效果。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Existing underwater robotic systems typically provide one of two operating modes---full teleoperation or full autonomy. Teleoperation is by far the most common, particularly for tasks involving interaction with the environment, such as grasping and manipulation. Autonomy is restricted to non-contact survey missions and to controlled laboratory settings. The ability to operate between teleoperation and autonomy will improve the efficiency and effectiveness of tasks performed in underwater environments. This research will develop and evaluate a novel shared autonomy framework. The research leverages the different nature of humans and robots. This work will reduce the need for multiple, highly trained operators. It has the potential to drastically reduce the cost of underwater missions. The contributions of this research will impact the way in which humans work together with robots within a wide variety of applications, including space exploration, disaster relief, and assistive robotics.As robotic systems play an ever-larger role as our surrogates for marine science and exploration, the ability to leverage the complementary nature of humans and robots becomes critical for scientific discovery. This research will develop new models and algorithms that exploit multiple non-commensurate sensing and control modalities to realize intelligent shared autonomy in complex unstructured environments. Novel to this research is the use of natural language and vision as complementary forms of weak supervision to enable robots to learn human-collaborative sensorimotor manipulation policies opportunistically from narrated human demonstrations. Fundamental to these methods is their ability to then refine these policies in situ based upon interaction with a human operator. Together, these models and algorithms will enhance the efficiency and effectiveness of underwater scientific exploration.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.