Cooperative bid and operation strategy in two-settlement energy market through dual-agent imitation learning on joint optimized policy

초록

This study proposes a reinforcement learning (RL)-based bidding and operation model for solar energy storage systems, designed to maximize electricity market profits under operational uncertainty. The model introduces a cooperative dual-agent framework within a market environment structured around a two-settlement market mechanism, employing offline learning and cooperative strategy development. To jointly derive optimal bidding and operation strategies, the problem is reformulated as a decentralized partially observable Markov decision process (Dec-POMDP), and a centralized training with decentralized execution (CTDE) architecture is adopted to enable collaborative decision-making between the bidding and battery operation agents. Notably, the proposed offline RL framework learns joint policies by imitating optimal decisions derived under perfect foresight of future market conditions, using an optimized joint policy demonstration (OJPD) approach. Experiments on real CAISO market data show that the proposed model earns about 10% more daily revenue than a single-agent RL baseline and about 6% more than an optimization-based model predictive control benchmark, with statistically significant margins over all benchmarks, while providing the most stable training and the strongest robustness under out-of-distribution market conditions.

키워드