Epistemic Uncertainty Estimation in Multi-Agent Reinforcement Learning

多智能体强化学习中的认知不确定性估计

• 批准号: 2747642
• 金额: --
• 依托单位: University of Oxford
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2022
• 资助国家: 英国
• 起止时间: 2022 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2747642
• 关键词: Epistemic; Uncertainty; Estimation; Multi; Agent

This project falls in the EPSRC artificial intelligence (AI) and robotics research area.Reinforcement Learning (RL) is a technique to train an AI agent through a system of rewards. The agent is allowed to interact and execute actions in an environment, and rewards are awarded when the agent successfully completes the intended task.RL has a multitude of applications, some examples include robotics, recommendations systems and healthcare.Multi-Agent Reinforcement Learning (MARL) focuses on how multiple agents interact with each other in a common environment.Each agent is motivated by their own rewards and interest, but agents can collaborate to achieve common goals or compete with each other, resulting in complex group dynamics.The study of MARL is increasingly more relevant, as AI agents become widespread in many aspects of our daily lives (e.g. self-driving cars).Aims and ObjectivesIn real-world scenarios, it is common for agents to not have perfect knowledge of the world around them, and modelling uncertainty is fundamental to avoid catastrophic and dangerous failures. This means, the agent should know what it does not know.The goal of this project is providing an accurate estimate of epistemic uncertainty in the MARL setting.Epistemic uncertainty refers to uncertainty caused by a lack of knowledge. In the MARL scenario, this includes information about the environment or other agents' motivations and behaviour. This type of uncertainty can be reduced by taking actions to explore the environment or interact with other agents.Obtaining a correct and calibrated uncertainty estimate could lead to safer interactions and collaboration between agents. Crucially, this includes interactions between AI agents and humans.A relevant application of this would be self-driving cars interacting with human drivers or other self-driving cars using different software. Modelling uncertainty over all other agents' behaviours, regardless of what software they run or what their intentions are, is fundamental for effective and safe collaboration.Novelty of the research methodologyCurrent RL techniques are incredibly successful, but fail to model uncertainty. In contrast, Bayesian models offer a theoretically grounded framework to reason about model uncertainty, but are often impossible to use in all but the simplest environments, due to their extremely high computational costs. Recently, multiple techniques have been proposed to circumvent this challenge and approximate Bayesian inference, such as dropout in Neural Networks (Gal, 2016) and Deep Ensembles (Lakshminarayanan, 2017).Gal, Yarin, and Zoubin Ghahramani. "Dropout as a bayesian approximation: Representing model uncertainty in deep learning." international conference on machine learning. PMLR, 2016.Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. "Simple and scalable predictive uncertainty estimation using deep ensembles." Advances in neural information processing systems 30 (2017).

该项目属于 EPSRC 人工智能 (AI) 和机器人研究领域。强化学习 (RL) 是一种通过奖励系统训练 AI 代理的技术。允许代理在环境中交互并执行操作，当代理成功完成预期任务时会授予奖励。强化学习有多种应用，包括机器人、推荐系统和医疗保健。多代理强化学习（MARL）重点关注多个代理如何在公共环境中相互交互。每个代理都受到自己的奖励和兴趣的激励，但代理可以协作以实现共同目标或相互竞争， 随着人工智能代理在我们日常生活的许多方面（例如自动驾驶汽车）变得广泛，MARL 的研究变得越来越重要。目的和目标在现实世界场景中，代理通常对周围的世界没有完美的了解，对不确定性进行建模是避免灾难性和危险故障的基础。这意味着，智能体应该知道它不知道什么。该项目的目标是在 MARL 设置中提供认知不确定性的准确估计。认知不确定性是指由于缺乏知识而引起的不确定性。在 MARL 场景中，这包括有关环境或其他代理的动机和行为的信息。通过采取行动探索环境或与其他智能体交互，可以减少这种类型的不确定性。获得正确且经过校准的不确定性估计可以导致智能体之间更安全的交互和协作。至关重要的是，这包括人工智能代理和人类之间的交互。其相关应用是自动驾驶汽车与人类驾驶员或使用不同软件的其他自动驾驶汽车进行交互。对所有其他代理行为的不确定性进行建模，无论他们运行什么软件或他们的意图是什么，都是有效和安全协作的基础。研究方法的新颖性当前的强化学习技术非常成功，但无法对不确定性进行建模。相比之下，贝叶斯模型提供了一个基于理论的框架来推理模型的不确定性，但由于其极高的计算成本，通常不可能在除了最简单的环境之外的所有环境中使用。最近，人们提出了多种技术来规避这一挑战并近似贝叶斯推理，例如神经网络中的 dropout（Gal，2016）和深度集成（Lakshminarayanan，2017）。Gal、Yarin 和 Zoubin Ghahramani。 “作为贝叶斯近似的 Dropout：代表深度学习中的模型不确定性。”机器学习国际会议。 PMLR，2016。Lakshminarayanan、Balaji、Alexander Pritzel 和 Charles Blundell。 “使用深度集成进行简单且可扩展的预测不确定性估计。”神经信息处理系统的进展 30 (2017)。