Partially Observable Multi-agent Inverse Reinforcement Learning

强化学习的一个主要挑战是逼近奖励结构，逆强化学习通过从专家演示中学习奖励结构来解决这个问题。这种方法已经取得了很大的成功，然而，很少有文献关注多智能体系统的逆强化学习方法，尽管多智能体系统的使用正在上升。目前多智能体系统的逆强化学习方法是在一个完全可观察的环境中对系统进行建模，这在现实环境中是不切实际的。在部分可观察的环境中，我们假设观察到的数据可能是有噪声的，甚至是不正确的，这更好地模拟了真实系统。为了解决这一知识差距，本研究将解决的主要研究问题是，哪些算法可以用于可以建模为多智能体部分可观察马尔可夫决策过程的环境？为了扩展这个问题，我们将评估这些算法在模拟和实际环境中的性能。为了解决这些问题，我们将首先利用对问题的理论理解来设计算法。然后，我们将算法应用于模拟游戏，并与其他算法比较性能。这些游戏将成为部分可观察的多智能体强化学习的标准基准游戏。我们将对使用真实数据的实际示例进行相同的处理。使用真实世界数据的目的是看看我们在多智能体设置中如何准确地模拟人类行为。最后，我们将探讨如何使用多智能体逆强化学习来更好地改进传统的多智能体强化学习方法，为这些问题提供更系统的奖励结构建模方法。这些新方法将同样在模拟和实际环境中进行评估。通过在部分可观察环境中研究多智能体强化学习，它使我们能够更好地将逆强化学习应用于现实世界系统。一个主要的潜在好处在于开发与人类兼容的人工智能系统。当人类与人工智能交互时，我们可以将其归类为多智能体系统。为了适当地设计有利于人类的人工智能系统，我们需要准确地建模人类的行为和需求，这可以通过逆强化学习来实现。另一个重要的应用是对人类需求的估计。在经济学、心理学和工程学中，为了设计满足这些需求的系统，了解人类的需求是很重要的。这项研究将使我们能够在其他智能生物存在的现实场景中准确地模拟这些人类行为。此外，通过使用这些估计和模型，我们还可以通过模拟人类行为和测试人类将与新系统交互来设计更好的工程系统。这项研究将与EPSRC工程和人工智能研究领域保持一致。将人工智能集成到工程系统（如未来的智能城市）中的主要因素之一是人类与这些系统进行智能交互的能力。这项研究将直接解决这个问题，通过更准确地模拟人类行为，并能够训练人工智能代理像人类一样行事。通过这种方式，我们可以设计出在未来的工程系统中满足人类需求的系统。

A major challenge of reinforcement learning is approximating the reward structure, which inverse reinforcement learning addresses by learning the reward structure from expert demonstrations. This method has shown a lot of success, however, few literature exists that focuses on inverse reinforcement learning methods for multi-agent systems, although the use of multi-agent systems is on the rise. Current inverse reinforcement learning methods for multi-agent systems model the systems in a fully observable environment, which can be impractical to use in a real world environment. In a partially observable environment, we assume that data that is observed can be noisy or even incorrect, which better models real systems. To address this knowledge gap, the main research question that will be addressed by this research is what algorithms can be used for environments that can be modelled as multi-agent partially observable Markov decision processes? To extend this question, we will evaluate the performance of these algorithms in simulated and practical environments. To solve these questions, we will first design algorithms using the theoretical understanding of the problem. Then, we will apply the algorithms to simulated games, and evaluate the performance against other algorithms. These games will be standard benchmark games for partially observable multi-agent reinforcement learning. The same will be done with a practical example using real data. The goal of using real world data is to see how accurately we can model human behaviour in a multi-agent setting. Finally, we will explore how the use of multi-agent inverse reinforcement learning to better improve traditional multi-agent reinforcement learning methods by having a more systematic way of modelling reward structures for these problems. These new methods will similarly be evaluated on simulated and practical environments. By studying multi-agent reinforcement learning in a partially observable environment, it allows us to much better apply inverse reinforcement learning to real world systems. One of the main potential benefits lies in the development of human compatible AI systems. Whenever a human interacts with an AI, we can classify that as a multi-agent system. In order to appropriately design the AI system for the benefit of the human, we need to model the human behaviours and needs accurately, which can be done with inverse reinforcement learning. Another important application is the estimation of human needs. In economics, psychology, and engineering, it is important to understand what humans need in order to design systems that address those needs. This research will allow us to accurately model these human behaviours in realistic scenarios in the presence of other intelligent beings. Additionally, by using these estimations and models, we can then also design better engineering systems by simulating human behaviour and test humans will interact with the new systems. This research will align with the EPSRC engineering and artificial intelligence research areas. One of the main factors for integrating artificial intelligence in engineered systems, such as future smart cities, is the ability for humans to interact intelligently with such systems. This research will directly address that by having a more accurate way of modelling human behaviour and being able to train artificially intelligent agents to behave like humans. In this way, we can design systems that will address the human needs in future engineering systems.