Collaborative Research: CPS: Medium: Empowering prosumers in electricity markets through market design and learning

合作研究：CPS：中：通过市场设计和学习为电力市场中的产消者赋权

• 批准号: 2038416
• 负责人: Vijay Subramanian
• 金额: $ 30万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2038416&HistoricalAwards=false
• 关键词: Collaborative; Research; CPS; Medium; Empowering

The availability of vast amounts of operational and end-user data in cyber-physical systems implies that paradigm improvements in monitoring and control can be attained via learning by many artificial intelligence agents despite them possessing vastly different abilities. Engaging this heterogeneous agent base in the context of the smart grid requires the use of hierarchical markets, wherein end-users participate in downstream markets collectively through aggregators, who in turn are coordinated by an upstream market. The goal of this project is to conduct a systematic study of such market-mediated learning and control. This project aims at much deeper levels of participation from end-users contributing electricity generation such as rooftop solar, shedding load via demand response, and providing storage capabilities such as electric vehicle batteries, to transform into reliable distributed energy resources (DER) at the level of wholesale markets. A methodological theme is multi-agent reinforcement learning (MARL) by agents that control physical systems via actions at different levels of the hierarchy. Underlying the whole project are well-founded physical models of the transmission and distribution grids, which provide structure to the problem domain and concrete use cases. This project facilitates a deeper level of decarbonization in the electricity sector, and contributes to climate change solutions by engineering a flat, interactive grid architecture that allows significant DERs to provide electricity services to both local and regional grids. Engagement with a grid-level market operator enables the project to address a problem space of immediate relevance to the current electricity grid. The project also includes the development of educational materials on data-analytics and energy systems. Intrinsic to the program are efforts at outreach to involve high-school students via demonstrations and lectures based on the technology developed.The goal of this project is a systematic and principled study of methods for hierarchical market-mediated learning and control, with the electric grid being the primary application domain. Multi-agent reinforcement learning (MARL) runs as a common methodological theme through the project, with strategic agents with varying information structures and concepts of rationality that control physical systems via actions at different levels of the hierarchy. The approach is different from studies on generic MARL algorithms in that attention is focused on well-founded physical models of the transmission and distribution grids, as well as the workings of the power system. The project is organized into three interdependent thrusts, namely, (i) Learning to bid as aggregators in wholesale markets, which studies dynamics of aggregators that provide supply offers and demand bids at the upstream market (wholesale level), while procuring these services from downstream DERs (retail level), (ii) Learning to incentivize retail users to contribute their resources, under which bounded rational agents learn to respond to a population-level distribution of other agents and incentives provided, and (iii) Evaluation and experimentation over a full-scale system emulator by integrating it with reinforcement learning tools. This project provides an architecture for DERs to provide electricity services to both local and regional grids, and hence contributes to developing solutions to climate change. Engagement with an independent system operator enables a focus on grid-specific issues, ensuring the applicability of the solutions to real-world problems. The impact is enhanced by specific minority inclusion activities, courses on computing tailored to broaden participation in the context of data-analytics and energy systems, and outreach to high-school students using demonstrations and lectures based on the project results.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

网络物理系统中大量的操作和最终用户数据的可用性意味着，尽管许多人工智能代理拥有截然不同的能力，但可以通过学习来实现监测和控制方面的范式改进。 在智能电网的背景下，这种异构的代理基础需要使用分层市场，其中最终用户通过聚合器集体参与下游市场，而聚合器又由上游市场协调。 本项目的目标是对这种以市场为中介的学习和控制进行系统的研究。 该项目旨在更深层次地参与最终用户的发电，如屋顶太阳能发电，通过需求响应减少负荷，并提供电动汽车电池等存储能力，以在批发市场一级转变为可靠的分布式能源（DER）。 一个方法论的主题是多代理强化学习（MARL）的代理控制物理系统通过行动在不同层次的层次结构。 整个项目的基础是传输和配电网的基础物理模型，这些模型为问题域和具体用例提供了结构。 该项目促进了电力部门更深层次的脱碳，并通过设计一个扁平的交互式电网架构，使重要的DER能够为当地和区域电网提供电力服务，为气候变化解决方案做出贡献。 与电网级市场运营商的合作使该项目能够解决与当前电网直接相关的问题空间。 该项目还包括编写关于数据分析和能源系统的教育材料。 该项目的核心是通过基于开发的技术的演示和讲座，努力让高中生参与进来。本项目的目标是系统地、原则性地研究以电网为主要应用领域的分层市场中介学习和控制方法。 多智能体强化学习（MARL）作为一个共同的方法论主题贯穿整个项目，具有不同信息结构和合理性概念的战略智能体通过层次结构不同级别的行动控制物理系统。 该方法是不同的通用MARL算法的研究中，注意力集中在有根据的物理模型的传输和配电网，以及电力系统的工作。 该项目分为三个相互依存的重点，即：㈠学习作为批发市场的集合商投标，研究在上游市场提供供应报价和需求投标的集合商的动态（批发一级），同时从下游的DER采购这些服务（ii）学习激励零售用户贡献他们的资源，有限理性的代理人学习响应人口水平分布的其他代理人和提供的激励措施，及（iii）评估和实验，通过将其与强化学习工具集成在一个全面的系统仿真器。 该项目提供了一个架构，供能源效率机构向地方和区域电网提供电力服务，从而有助于制定应对气候变化的解决方案。 与独立的系统运营商合作，可以专注于电网特定的问题，确保解决方案适用于现实世界的问题。 通过具体的少数民族包容活动、为扩大数据分析和能源系统背景下的参与而量身定制的计算课程，以及通过基于项目结果的演示和讲座向高中生进行宣传，增强了影响力。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响力审查标准进行评估，被认为值得支持。

项目成果

Decentralized Cooperative Reinforcement Learning with Hierarchical Information Structure

• 发表时间: 2021-11
• 期刊: ArXiv
• 作者: Hsu Kao;Chen-Yu Wei;V. Subramanian

Bayesian Learning of Optimal Policies in Markov Decision Processes with Countably Infinite State-Space

可数无限状态空间马尔可夫决策过程中最优策略的贝叶斯学习

• 发表时间: 2023
• 期刊: Advances in Neural Information Processing Systems 36 (NeurIPS 2023
• 作者: Saghar Adler;Vijay Subramanian
• 通讯作者: Vijay Subramanian

Private Information Compression in Dynamic Games among Teams

团队动态博弈中的私有信息压缩

• DOI: 10.1109/cdc45484.2021.9683479
• 发表时间: 2021
• 期刊: 2021 60th IEEE Conference on Decision and Control (CDC
• 作者: Tang, Dengwang;Tavafoghi, Hamidreza;Subramanian, Vijay;Nayyar, Ashutosh;Teneketzis, Demosthenis
• 通讯作者: Teneketzis, Demosthenis

A Strong Duality Result for Cooperative Decentralized Constrained POMDPs

• DOI: 10.1109/cdc49753.2023.10383989
• 发表时间: 2023-12
• 期刊: 2023 62nd IEEE Conference on Decision and Control (CDC)
• 作者: Nouman Khan;Vijay G. Subramanian

Rarest-First with Probabilistic-Mode-Suppression (RFwPMS)

具有概率模式抑制的稀有优先 (RFwPMS)

• 发表时间: 2024
• 期刊: IEEE transactions on information theory
• 影响因子: 2.5
• 作者: Nouman Khan;Mehrdad Moharrami;Vijay G. Subramanian
• 通讯作者: Vijay G. Subramanian