Distributed Adaptive Systems: Feedback Control with Evolutionary Games

提出的研究是发展反馈控制理论方法的学习和适应的多智能体系统，并调查其潜力作为分析和设计工具的分布式自适应系统。这项提议的工作源于加州大学洛杉矶分校最近关于反馈控制在分布式学习博弈论框架中的作用的创新。最近的这项工作克服了博弈论学习中长期存在的感知障碍，并为分布式系统设计开辟了新的机会。提出的研究将强调多智能体学习的基础数学模型。有待探索的研究方向有：基于反馈控制的游戏学习高级分析：反馈控制方法在游戏学习中的引入是最近的事情，还有许多重要的未解决的问题。提出的感兴趣的主题包括基于反馈控制方法的战略优势，异构学习算法的系统分析，动态网络互连和监督学习切换。连续行动空间：游戏框架中的学习目前适用于在有限的选择集中学习。这个方向涉及到在连续体上学习决策，比如动态资源分配问题。问题包括新的基于反馈控制的分布式优化方法，严重受限的信息结构，以及信息结构的互补共享。具有状态进化的多智能体游戏：游戏框架中的学习使用静态游戏设置。状态变量是学习的状态，而不是子系统或环境的固有状态。这个方向考虑了具有内部状态的游戏的学习，也被称为马尔可夫或随机游戏。问题包括专家选择的时间尺度的变化，基于q学习的反馈控制，以及集成商的行动。可进化硬件的应用：可进化硬件是一个新兴领域，它使用受生物进化启发的概念来设计可重构、自组织的硬件。提出的多智能体概念可以看作是一种混合了反馈控制和多智能体学习的工程进化。这个项目将使用可进化的硬件范式来说明和激励所提议的研究。智力优势：在多人游戏领域有大量的研究。在非零和博弈的情况下，混合策略（即随机）纳什均衡的概念，尽管其核心作用，已经收到了相当多的审查，关于玩家如何，通过反复的互动，将收敛到纳什均衡。事实上，长期以来一直有相反的例子。提出的研究的创新基础是，反馈控制的简单概念可以使这种收敛成为可能。因此，本研究为如何从博弈论的角度设计分布式自适应系统开辟了许多新的可能性。更广泛的影响：建议的研究将对社会应用和本科生教育产生更广泛的影响。社会性：分布式系统概念与许多领域相关，包括工程领域和社会性领域。这些包括数据网络、分布式机器人、交通网络、分布式设计、电网基础设施和分布式计算，以及经济交换、社会交换和政治联盟动态。提出的工作解决了这些系统的数学模型的一个基本组成部分，并在各种领域具有潜在的影响。教育方面：主要开展的活动为：1)引入1单元的大一课程，介绍分布式系统和反馈控制的概念及其应用。这是通过加州大学洛杉矶分校的菲亚特Luxfreshman seminar series来实现的。2)开发一个基于可进化硬件的研究试验台，作为一个场所，通过定向学习选修课来整合本科生研究水平的参与。当然，这些活动是在对研究生研究人员进行通常的教育和培训以及向研究界传播研究成果之外进行的。

The proposed research is to develop feedback control theoretic methods for learning and adaptation inmultiagent systems and to investigate their potential as analysis and design tools for distributed adaptivesystems. The proposed work stems from recent innovations at UCLA on the role of feedback control in thegame theoretic framework of distributed learning. This recent work has overcome long standing perceivedobstacles in game theoretic learning and opens new opportunities for distributed system design.The proposed research will emphasize underlying mathematical models of multiagent learning. Theresearch directions to be explored are:Advanced analysis of feedback control based learning in games: The introduction of feedback controlmethods in learning in games is very recent, and there are many important unresolved issues. Proposedtopics of interest include strategic advantage of feedback control based methods, analysis of systems withheterogeneous learning algorithms, dynamic network interconnections, and supervised switching for learning.Continuum action spaces: The learning in games framework currently applies to learning among a finiteset of choices. This direction involves learning with decisions over a continuum, such as dynamic resourceallocation problems. Issues include new feedback control based approaches to distributed optimization,severely limited information structures, and complementary sharing of information structures.Multiagent games with state evolution: The learning in games framework uses a static game setup.The state variable is the state of learning, but not inherent states of subsystems or the environment. Thisdirection considers learning for games with internal states, also known as Markov or stochastic games. Issuesinclude changes of time-scale for expert selection, feedback control based Q-learning, and integrator action.Application to Evolvable Hardware: Evolvable hardware is an emerging area that uses notions inspiredby biological evolution for the design of reconfigurable, self-organizing hardware. The proposed concepts onmultiagent may be viewed as an engineered evolution with its blending of feedback control and multiagentlearning. This project will use the evolvable hardware paradigm to illustrate and motivate the proposedresearch.Intellectual Merit: There is an extensive body of research in the area of multiple player games. In the caseof non-zero-sum games, the concept of mixed strategy (i.e, randomized) Nash equilibrium, despite its centralrole, has received considerable scrutiny as to how players, through repeated interactions, would ever convergeto a Nash equilibrium. Indeed there are long standing examples to the contrary. The innovative basis ofthe proposed research establishes that simple notions from feedback control can enable such convergence.Consequentially, this research opens many new possibilities into how to design distributed adaptive systemsfrom a game theoretic viewpoint.Broader Impact: The proposed research will have a broader impact on societal applications and undergraduatestudent eduction.Societal: The distributed systems concept is relevant in a multitude of domains, both engineered andsocial. These include data networks, distributed robotics, traffic networks, distributed design, power gridinfrastructure, and distributed computation, as well as economic exchange, social exchange, and politicalcoalition dynamics. The proposed work addresses a fundamental component of the mathematical models ofsuch systems and has potential implications in a variety of areas.Educational: Key activities to be undertaken are 1) Introducing 1-unit freshman courses on the conceptof distributed systems and feedback control and their applications. This is through UCLA's Fiat Luxfreshman seminar series, and 2) Developing an research test-bed based on evolvable hardware as a venue toincorporate undergraduate research-level participation through directed study electives. Of course, these activitiesare in addition to the usual education and training of graduate student researchers and disseminationof results to the research community.