Deep Inverse Reinforcement Learning for Objective Function Identification in Bidding Models
Deep Inverse Reinforcement Learning for Objective Function Identification in Bidding Models
复制标题
投标模型中目标函数识别的深度逆强化学习
DOI：
--
复制
发表时间：
2021
影响因子：
6.6
通讯作者：
C. Kang
中科院分区：
文献类型：
--
作者：
Hongye Guo;Qixin Chen;Q. Xia;C. Kang
随着世界范围内电力系统管制的放松，竞价行为的仿真研究日益受到重视。这些研究中的一个关键因素是准确定义和建模个人奖励函数（或目标函数）。考虑到市场参与者和研究者之间普遍存在的信息壁垒，通常的做法是基于理论假设来开发报酬函数，这必然会造成与真实的世界的偏差。然而，由于近年来市场数据逐渐变得透明，特别是关于历史投标行为的数据，引入数据驱动的方法来识别隐藏在原始投标数据中的个体奖励函数是可行的。因此，本文提出了一个数据驱动的投标目标函数识别框架，包括三个步骤。首先，投标决策过程的参与者制定为一个标准的马尔可夫决策过程。其次，引入基于最大熵的深度逆强化学习方法来识别个体奖励函数，其高维非线性可以保存在多层感知（MLP）中。第三，根据获得的基于MLP的目标函数，定制深度Q网络方法来模拟个人竞价行为。基于澳大利亚电力市场的真实的市场数据，验证了所提出的框架和方法的有效性和可行性。
Due to the deregulation of power systems worldwide, bidding behavior simulation research has gained prominence. One crucial element in these studies is accurately defining and modelling the individual reward function (or objective function). Considering the ubiquitous information barriers between market participants and researchers, the common way is to develop reward functions based on theoretical assumptions, which will inevitably cause deviations from the real world. However, since market data have gradually become transparent in recent years, especially data regarding historical bidding behaviors, it is feasible to introduce data-driven methods to identify the individual reward functions that are hidden in raw bidding data. Thus, this paper proposes a data-driven bidding objective function identification framework with three procedures. First, the bidding decision processes of participants are formulated as a standard Markov decision process. Second, a deep inverse reinforcement learning method that is based on maximum entropy is introduced to identify individual reward functions, whose high-dimensional nonlinearity could be saved in multilayer perceptions (MLPs). Third, a deep Q-network method is customized to simulate the individual bidding behaviors based on the obtained MLP-based objective functions. The effectiveness and feasibility of the proposed framework and methods are tested based on real market data from the Australian electricity market.