Collaborative Research: CPS: Medium: Empowering prosumers in electricity markets through market design and learning

网络物理系统中大量操作和最终用户数据的可用性意味着，尽管许多人工智能代理具有截然不同的能力，但它们可以通过学习来实现监测和控制方面的范式改进。在智能电网的环境中使用这种异构的代理基础需要使用分层市场，其中终端用户通过聚合器集体参与下游市场，而聚合器又由上游市场协调。本项目的目的是对这种以市场为中介的学习和控制进行系统的研究。该项目旨在让终端用户更深入地参与其中，如屋顶太阳能发电，通过需求响应减少负荷，并提供电动汽车电池等存储能力，从而在批发市场层面转化为可靠的分布式能源（DER）。一个方法学主题是多智能体强化学习（MARL），由智能体通过不同层次的行为控制物理系统。整个项目的基础是建立良好的输配电网络的物理模型，它为问题域和具体用例提供了结构。该项目促进了电力部门更深层次的脱碳，并通过设计一个扁平的交互式电网架构，为气候变化解决方案做出了贡献，该架构允许重要的分布式电网为本地和区域电网提供电力服务。与电网级市场运营商的合作使该项目能够解决与当前电网直接相关的问题空间。该项目还包括编写关于数据分析和能源系统的教材。该计划的本质是通过基于所开发的技术的演示和讲座，努力让高中生参与进来。该项目的目标是对分层市场中介学习和控制方法进行系统和原则性的研究，电网是主要的应用领域。多智能体强化学习（MARL）作为一个共同的方法论主题贯穿整个项目，战略智能体具有不同的信息结构和理性概念，通过不同层次的行为控制物理系统。该方法不同于对一般MARL算法的研究，因为它的注意力集中在输电和配电网以及电力系统工作的可靠物理模型上。该项目分为三个相互依存的重点，即(i)学习作为批发市场的聚合者进行投标，研究聚合者在上游市场（批发级）提供供需报价和需求投标，同时从下游的DERs（零售级）采购这些服务的动态；（ii）学习激励零售用户贡献其资源；在这种情况下，有限的理性代理学习对其他代理和提供的激励的人口水平分布做出反应，以及（iii）通过将其与强化学习工具集成在一起，对全尺寸系统模拟器进行评估和实验。该项目为DERs提供了一个架构，为地方和区域电网提供电力服务，从而有助于制定应对气候变化的解决方案。与独立的系统运营商合作，可以专注于电网特定问题，确保解决方案适用于实际问题。具体的少数族裔包容活动、为扩大数据分析和能源系统背景下的参与而量身定制的计算机课程，以及利用基于项目结果的演示和讲座向高中生推广，这些都增强了项目的影响。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The availability of vast amounts of operational and end-user data in cyber-physical systems implies that paradigm improvements in monitoring and control can be attained via learning by many artificial intelligence agents despite them possessing vastly different abilities. Engaging this heterogeneous agent base in the context of the smart grid requires the use of hierarchical markets, wherein end-users participate in downstream markets collectively through aggregators, who in turn are coordinated by an upstream market. The goal of this project is to conduct a systematic study of such market-mediated learning and control. This project aims at much deeper levels of participation from end-users contributing electricity generation such as rooftop solar, shedding load via demand response, and providing storage capabilities such as electric vehicle batteries, to transform into reliable distributed energy resources (DER) at the level of wholesale markets. A methodological theme is multi-agent reinforcement learning (MARL) by agents that control physical systems via actions at different levels of the hierarchy. Underlying the whole project are well-founded physical models of the transmission and distribution grids, which provide structure to the problem domain and concrete use cases. This project facilitates a deeper level of decarbonization in the electricity sector, and contributes to climate change solutions by engineering a flat, interactive grid architecture that allows significant DERs to provide electricity services to both local and regional grids. Engagement with a grid-level market operator enables the project to address a problem space of immediate relevance to the current electricity grid. The project also includes the development of educational materials on data-analytics and energy systems. Intrinsic to the program are efforts at outreach to involve high-school students via demonstrations and lectures based on the technology developed.The goal of this project is a systematic and principled study of methods for hierarchical market-mediated learning and control, with the electric grid being the primary application domain. Multi-agent reinforcement learning (MARL) runs as a common methodological theme through the project, with strategic agents with varying information structures and concepts of rationality that control physical systems via actions at different levels of the hierarchy. The approach is different from studies on generic MARL algorithms in that attention is focused on well-founded physical models of the transmission and distribution grids, as well as the workings of the power system. The project is organized into three interdependent thrusts, namely, (i) Learning to bid as aggregators in wholesale markets, which studies dynamics of aggregators that provide supply offers and demand bids at the upstream market (wholesale level), while procuring these services from downstream DERs (retail level), (ii) Learning to incentivize retail users to contribute their resources, under which bounded rational agents learn to respond to a population-level distribution of other agents and incentives provided, and (iii) Evaluation and experimentation over a full-scale system emulator by integrating it with reinforcement learning tools. This project provides an architecture for DERs to provide electricity services to both local and regional grids, and hence contributes to developing solutions to climate change. Engagement with an independent system operator enables a focus on grid-specific issues, ensuring the applicability of the solutions to real-world problems. The impact is enhanced by specific minority inclusion activities, courses on computing tailored to broaden participation in the context of data-analytics and energy systems, and outreach to high-school students using demonstrations and lectures based on the project results.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.