Distributed Adaptive Systems: Feedback Control with Evolutionary Games

本研究旨在发展多智能体系统中学习与适应的反馈控制理论方法，并探讨其作为分布式自适应系统分析与设计工具的潜力。这项拟议的工作源于加州大学洛杉矶分校最近关于反馈控制在分布式学习的博弈论框架中的作用的创新。这项最新的工作克服了博弈论学习中长期存在的障碍，为分布式系统设计打开了新的机会。拟议的研究将强调多智能体学习的基本数学模型。有待探索的研究方向是：基于游戏学习的反馈控制的高级分析：反馈控制方法在游戏学习中的引入是最近的事情，有许多重要的悬而未决的问题。提出的感兴趣的主题包括基于反馈控制的方法的战略优势，使用不同学习算法的系统分析，动态网络互联，以及用于学习的监督切换。连续动作空间：游戏中的学习框架目前适用于在有限的选择集中进行学习。这个方向包括在一个连续体中学习决策，例如动态资源分配问题。问题包括基于反馈控制的分布式优化的新方法、严重受限的信息结构和信息结构的互补共享。状态进化的多智能体博弈：博弈框架中的学习使用静态博弈设置。状态变量是学习的状态，而不是子系统或环境的固有状态。这个方向考虑了具有内部状态的博弈的学习，也称为马尔可夫博弈或随机博弈。问题包括专家选择的时间尺度的变化，基于Q学习的反馈控制，以及积分操作。应用于进化硬件：进化硬件是一个新兴的领域，它使用生物进化的概念来设计可重构的、自组织的硬件。所提出的关于多智能体的概念可以被看作是反馈控制和多智能体学习相结合的工程进化。这个项目将使用可演化的硬件范例来说明和激励所提出的研究。智力优势：在多人游戏领域有大量的研究。在非零和博弈的情况下，混合策略(即随机的)纳什均衡的概念，尽管它起着中心作用，但已经受到了相当大的审查，关于参与者如何通过反复互动收敛到纳什均衡。事实上，有一些长期存在的例子与之相反。这项研究的创新基础是，反馈控制的简单概念可以使这种收敛成为可能。因此，这项研究从博弈论的角度为如何设计分布式自适应系统开辟了许多新的可能性。广泛影响：拟议的研究将对社会应用和本科生教育产生更广泛的影响。社会：分布式系统的概念与许多领域相关，无论是工程领域还是社会领域。这些领域包括数据网络、分布式机器人技术、交通网络、分布式设计、电网基础结构和分布式计算，以及经济交换、社会交换和政治联盟动态。这项拟议的工作涉及这类系统的数学模型的一个基本组成部分，并在各种领域具有潜在的影响。教育：将开展的主要活动是：1)介绍关于分布式系统和反馈控制及其应用的概念的1个单元的新生课程。这是通过加州大学洛杉矶分校的菲亚特豪华新生研讨会系列，以及2)开发一个基于可进化硬件的研究试验台，作为通过定向学习选修课纳入本科生研究水平参与的场所。当然，这些活动是对研究生研究人员的常规教育和培训以及向研究界传播成果之外的活动。

The proposed research is to develop feedback control theoretic methods for learning and adaptation inmultiagent systems and to investigate their potential as analysis and design tools for distributed adaptivesystems. The proposed work stems from recent innovations at UCLA on the role of feedback control in thegame theoretic framework of distributed learning. This recent work has overcome long standing perceivedobstacles in game theoretic learning and opens new opportunities for distributed system design.The proposed research will emphasize underlying mathematical models of multiagent learning. Theresearch directions to be explored are:Advanced analysis of feedback control based learning in games: The introduction of feedback controlmethods in learning in games is very recent, and there are many important unresolved issues. Proposedtopics of interest include strategic advantage of feedback control based methods, analysis of systems withheterogeneous learning algorithms, dynamic network interconnections, and supervised switching for learning.Continuum action spaces: The learning in games framework currently applies to learning among a finiteset of choices. This direction involves learning with decisions over a continuum, such as dynamic resourceallocation problems. Issues include new feedback control based approaches to distributed optimization,severely limited information structures, and complementary sharing of information structures.Multiagent games with state evolution: The learning in games framework uses a static game setup.The state variable is the state of learning, but not inherent states of subsystems or the environment. Thisdirection considers learning for games with internal states, also known as Markov or stochastic games. Issuesinclude changes of time-scale for expert selection, feedback control based Q-learning, and integrator action.Application to Evolvable Hardware: Evolvable hardware is an emerging area that uses notions inspiredby biological evolution for the design of reconfigurable, self-organizing hardware. The proposed concepts onmultiagent may be viewed as an engineered evolution with its blending of feedback control and multiagentlearning. This project will use the evolvable hardware paradigm to illustrate and motivate the proposedresearch.Intellectual Merit: There is an extensive body of research in the area of multiple player games. In the caseof non-zero-sum games, the concept of mixed strategy (i.e, randomized) Nash equilibrium, despite its centralrole, has received considerable scrutiny as to how players, through repeated interactions, would ever convergeto a Nash equilibrium. Indeed there are long standing examples to the contrary. The innovative basis ofthe proposed research establishes that simple notions from feedback control can enable such convergence.Consequentially, this research opens many new possibilities into how to design distributed adaptive systemsfrom a game theoretic viewpoint.Broader Impact: The proposed research will have a broader impact on societal applications and undergraduatestudent eduction.Societal: The distributed systems concept is relevant in a multitude of domains, both engineered andsocial. These include data networks, distributed robotics, traffic networks, distributed design, power gridinfrastructure, and distributed computation, as well as economic exchange, social exchange, and politicalcoalition dynamics. The proposed work addresses a fundamental component of the mathematical models ofsuch systems and has potential implications in a variety of areas.Educational: Key activities to be undertaken are 1) Introducing 1-unit freshman courses on the conceptof distributed systems and feedback control and their applications. This is through UCLA's Fiat Luxfreshman seminar series, and 2) Developing an research test-bed based on evolvable hardware as a venue toincorporate undergraduate research-level participation through directed study electives. Of course, these activitiesare in addition to the usual education and training of graduate student researchers and disseminationof results to the research community.