An Adaptive $Q$-Learning Algorithm Developed for Agent-Based Computational Modeling of Electricity Market
An Adaptive $Q$-Learning Algorithm Developed for Agent-Based Computational Modeling of Electricity Market
复制标题
DOI：
10.1109/tsmcc.2010.2044174
复制
发表时间：
2010-09
期刊：
影响因子：
--
通讯作者：
M. Rahimiyan;H. R. Mashhadi
中科院分区：
文献类型：
--
作者：
M. Rahimiyan;H. R. Mashhadi
通过调整 Q 学习（QL）参数以适应动态不确定环境的条件来平衡探索和利用一直是强化学习背景下的一个重要课题。电力市场的特殊性提供了如此复杂的动态经济环境，从而对学习方法的先进性提出了要求。在这种经济体系中，代理人的市场势力在投标决策问题中起着至关重要的作用。为了改进QL方法，作为主要思想，提出了调整其参数以适应市场力量，以在勘探和开采之间取得良好的平衡。为了实现这种适应过程，由于人类决策过程的模糊性，设计了一个模糊系统来将每个智能体的市场力量映射到 QL 参数中。因此，开发了一种模糊QL方法来对电力供应商在计算电力市场中的战略竞价行为进行建模。在仿真框架中，QL算法根据过去的经验和参数值来选择电力供应商的出价策略，这体现了人类的风险特征。所提出的方法在多区域电力系统中的电源供应器的应用表明，与具有固定参数的 QL 相比，性能得到了改进。
Balancing between exploration and exploitation with adaptation of the Q-learning (QL) parameters to the condition of dynamic uncertain environment has always been a significant subject of interest in the context of reinforcement learning. The peculiarities of the electricity market have provided such complex dynamic economic environment, and consequently have increased the requirement for advancement of the learning methods. In this economic system, the agent's market power plays a vital role in bidding decision-making problem. In order to improve the QL method, as main idea, adaptation of its parameters to the market power is proposed for making a good balance between exploration and exploitation. To implement this adaptation process, due to the fuzzy nature of human's decision-making process, a fuzzy system is designed to map each agent's market power into the QL parameters. Therefore, a fuzzy QL method is developed to model the power supplier's strategic bidding behavior in a computational electricity market. In the simulation framework, the QL algorithm selects the power supplier's bidding strategy according to the past experiences and the values of the parameters, which show the human's risk characteristic. The application of the proposed methodology for the power supplier in a multiarea power system shows the performance improvement in comparison to the QL with fixed parameters.