Learning, Innovation, and Explanation in Self-Organising Multi-Agent Systems

最初的研究计划侧重于多智能体系统中的学习、创新和解释。关于多智能体系统的设计，已经确定了三个层次：智能体（个体）层次、智能体间（社会）层次和系统层次。先前的工作提出了社会激励机制，以促进代理间级别的集体行动和自组织，以及基于gp的技术，用于代理和系统级别的适应和近似优化。我们希望在所有三个层面探索学习和创新，以应对环境中潜在的动态变化并确保可持续性。我们注意到，如果我们将系统实体视为控制代理，那么可以应用于代理级别的方法（例如用于策略优化的强化学习）与可以应用于系统级别的方法是相同的。到目前为止，我们的研究主要集中在构建一个单一的系统范围的政策，所有的代理都必须遵守。然而，在多中心的视角下，代理应该能够制定自己的政策和行动计划，并参与系统政策的制定。这需要结合不同感官经验和意见的机制，以集体学习的方式使用沟通和知识转移协议，以实现社会学习和社会创新。我们在这一阶段提出的研究问题是：即使在动态环境中，个体和集体的系统政策应该如何制定和解释，以持续改善和保持系统性能和代理满意度？公式化既指分布式学习算法和协议，使代理能够构建个人和集体策略，也指策略的表示方式。可解释性是指产生新策略的学习算法和策略本身的可理解性。理解为什么选择了一个策略，以及为什么它比其他可能被选择的候选策略更好，对于系统设计者和用户能够信任它的建议并收集有关问题领域的有价值的知识非常重要。一些学习模型本质上是可理解的，而另一些则不是，并且有可能一些不可理解的模型可以被理解、解释和解释。可理解的模型可以为人类本质上难以理解的问题领域提供新的视角，并导致采用新的和更好的策略来搜索理论上无限的规则集空间，这些规则集是建立政策的基础。在这个阶段，我们已经确定了以下高级任务：-回顾以前在多智能体系统设计方面的工作，重点是上面确定的三个层次：代理、社会和系统层面-识别先前工作中的知识缺口，即关于政策的多中心构建和可解释性-根据文献综述和关于知识缺口的结论撰写调查论文-识别可能的应用程序和系统，作为政策构建和选择的运行示例-迭代地提出三个层面的政策构建解决方案：单一系统政策（当前工作）；多代理策略；本项目旨在为集体自适应系统的设计提出一种新的范式，其中系统策略不是手工制作的，设计者也不是硬编码不可变的规则集，而是以提高系统性能和智能体满意度为明确目标的自动构建的结果。研究领域：控制工程、工程设计、人工智能技术、软件工程

The initial research proposal focuses on learning, innovation, and explanation in multi-agent systems. Three levels have been identified concerning the design of multi-agent systems: the agent (individual) level, the inter-agent (social) level, and the system level. Previous work has proposed socially-inspired mechanisms to promote collective action and self-organisation at the inter-agent level and GP-based techniques for adaptation and approximate optimisation at the agent and system levels. We would like to explore learning and innovation at all three levels as ways to handle potentially dynamic changes in the environment and ensure sustainability.We note that the methods which can be applied to the agent level, such as reinforcement learning for policy optimisation, are the same sort of methods which can be applied to the system level if we consider the system entity as a control agent. So far, our research has focused on constructing a single system-wide policy by which all agents must abide. However, in a polycentric perspective, the agents should be able to formulate their own policies and plans of action and to participate in the formulation of system policies. This would require mechanisms for combining different sensory experiences and opinions, using communication and knowledge transfer protocols in a collective learning fashion in order to achieve social learning and social innovation.The research question we propose at this stage is: how should individual and collective system policies be formulated and explained so as to sustainably improve and maintain system performance and agent satisfaction, even in dynamic environments? Formulation refers both to the distributed learning algorithms and protocols which enable the agents to construct individual and collective policies and to the way the policies are represented. Explainability refers to the intelligibility of both the learning algorithms which result in the creation of new policies and the policies themselves. Understanding why a policy has been selected and why it is better than other candidate policies which could also have been selected is important for system designers and users to be able to trust its recommendations and gather valuable knowledge regarding the problem domain. Some learning models are inherently intelligible while others are not and it is possible that some unintelligible models can be made understandable, explainable, and interpretable. Intelligible models could shed a new light upon problem domains which are intrinsically hard for humans to understand and lead to the adoption of new and better strategies for searching the theoretically infinite space of sets of rules upon which policies are built. At this stage we have identified the following high-level tasks:- Review previous work on the design of multi-agent systems, focusing on the three levels identified above: the agent, the social, and the system levels- Identify knowledge gaps in previous work, namely regarding polycentric construction of policies and explainability- Write survey paper based on both the literature review and the conclusions about knowledge gaps- Identify possible applications and systems to use as running examples for policy construction and selection- Iteratively propose solutions to three levels of policy construction: single system policy (current work); multiple agent policies; multiple agent policies and a collectively formulated system policyThis project aims at proposing a new paradigm for the design of collective adaptive systems in which system policies are not handcrafted and designers do not hardcode immutable rule sets, but rather the result of automatic construction with the explicit goal of improving system performance and agent satisfaction.EPSRC Research Areas: Control Engineering, Engineering design, Artificial intelligence technologies, Software engineering