Power Grid Cascading Failure Mitigation by Reinforcement Learning
Total Page:16
File Type:pdf, Size:1020Kb
Power Grid Cascading Failure Mitigation by Reinforcement Learning Yongli Zhu * 1 Abstract climate change, having a stable power grid is critical for renewable resources integration; This paper proposes a cascading failure mitigation strategy based on Reinforcement Learning (RL). 2) The backup generators are typically fossil-fuel (e.g., coal, The motivation of the Multi-Stage Cascading Fail- diesel) based units that emit greenhouse gases. ure (MSCF) problem and its connection with the Therefore, it is meaningful to develop nuanced strategies to challenge of climate change are introduced. The mitigate a cascading failure at its early stage with as little bottom-level corrective control of the MCSF prob- generation cost as possible. lem is formulated based on DCOPF (Direct Cur- rent Optimal Power Flow). Then, to mitigate the The cascading failure mitigation can be regarded as a MSCF issue by a high-level RL-based strategy, stochastic dynamic programming problem with unknown physics-informed reward, action, and state are de- information about the risk of failures. Previous researches vised. Besides, both shallow and deep neural net- try to tackle this problem based on either mathematical work architectures are tested. Experiments on the programming methods or heuristic methods. For example, IEEE 118-bus system by the proposed mitigation bi-level programming is used to mitigate cascading fail- strategy demonstrate a promising performance in ures when energy storages exist (Du & Lu, 2014). In (Han reducing system collapses. et al., 2018), an algorithm based on percolation theory is employed for mitigating cascading failures by using UPFC (Unified Power Flow Controller) to redistribute the system 1. Introduction power flow more evenly. In (Tootaghaj et al., 2018), a re- covery plan for load-serving from cascading failures was Increasing renewable sources (e.g., wind, solar) are inte- put forward considering the uncertainty of failure locations. grated into modern power grids to reduce emissions. How- In (Cordova-Garcia et al., 2018), a cascade control algo- ever, due to the intermittent natures of those renewable rithm using load shedding considering the communication sources, the original power grid can become fragile, i.e., network delays for power grids was proposed to reduce the more easily affected by various risks. Among those risks, failure of power lines. In (Shuvro et al., 2017), By charac- the cascading failure is one of the most challenging issues terizing the cascading-failure dynamics as a Markov chain necessary to be addressed (Sun et al., 2019). A cascading model, it is found that the impact of human-operator action failure is defined as a series of consecutive malfunctions of will have a significant impact on cascading failures. physical components (e.g., power transmission lines, power substations). A cascading failure is typically caused by an However, some of the above research share the same limita- unexpected natural disaster such as a hurricane, typhoon, or tion: unrealistic assumptions are often made, which yield impractical control strategies in terms of time or economic arXiv:2108.10424v1 [cs.LG] 23 Aug 2021 flood. Severe cascading failures can lead to a total system collapse (i.e., disintegrated into small energized pieces) or cost. Different from communication networks and social even a blackout event (i.e., loss of electricity for the entire networks, the power grid is not a pure-mathematical graph, city or country). To recover the power grid to a healthy state, but a physical grid. Its node (called “bus” in power sys- backup generators need to be turned on. The relevance of tem terminology) and edge (called “branch” in power sys- the cascading failure problem to climate change is: tem terminology) are both physical entities that can not be added or removed arbitrarily. On the other hand, most 1) Given the increase in extreme weather events due to power system lines are equipped with automatic protection- 1ECE Dept., Texas A&M University, College Station, relay devices, which can trip the line when the line cur- USA. Correspondence to: Yongli Zhu <[email protected], rent/power/temperature exceeds certain thresholds in a pre- [email protected]>. defined time window. Thus, in this paper, the main focus is on branch failures rather than node failures. “Tackling Climate Change with Machine Learning” Workshop of the 38 th International Conference on Machine Learning, July 23, Meanwhile, some emerging artificial intelligence technolo- 2021. Copyright 2021 by the author(s). Power Grid Cascading Failure Mitigation by Reinforcement Learning gies, such as reinforcement learning (RL) and deep learning (Alternative Current Power Flow) converges and all branch (DL), have nourished both the fields of power systems and flows are within secure limits or 2) become collapsed. control theory (Kiumarsi et al., 2018). In (Glavic et al., In conventional cascading failure analysis, typically only 2017), a holistic literature review is given on the recent one stage is considered (Qi et al., 2017). However, in cer- development of RL applications in the power system area, tain situations, succeeding stages might follow shortly. For though topics regarding cascading failure are not covered. example, a wind storm results in one generation, in which In (Vlachogiannis & Hatziargyriou, 2004), the RL method certain lines are lost, and the system reaches a new steady is used for reactive power control. In (Liu et al., 2018), volt- state. Then, shortly, a new stage is invoked by tripping an age restoration for an islanded microgrid is achieved via a important line due to the misoperation of human-operator distributed RL method. In (Zhu et al., 2018), an application or relay protection. As an example, Table1 and2 list the for disturbance classification is proposed based on image simulation results of the IEEE 118 system for a two-stage embedding and convolutional neural network (CNN). In MSCF problem in two independent episodes. (Yan et al., 2018), deep learning is applied in power con- sumption forecasting. However, the application of RL or A na¨ıve way to handle this complicated multi-stage scenario DL for cascading failure study are less reported. is to tackle each stage independently by existing method e.g. SCOPF (Security Constrained Optimal Power Flow). How- In this paper, a reinforcement learning approach is designed ever, merely using SCOPF may not work well due to the for the mitigation of cascading failures with the following overlook of the correlations between any two consecutive contributions: stages. For example, the previous study (Zhu et al., 2014; Chen et al., 2007) found that sequentially attacking each line • 1) Propose and formulate a novel problem called Multi- one by one can sometimes achieve a more severe effect (i.e., Stage Cascading Failure (MSCF) for the first time. more components loss) than attack multiple lines simultane- ously. Thus, the MSCF problem should be considered from • 2) Present a systematic reinforcement learning frame- a holistic perspective. work to tackle the MSCF problem. Similar to AlphaGo (Silver et al., 2016), a “two-player game“ idea is uti- 2.2. Mimicking the corrective controls by DCOPF lized in this study. When a failure event happens, the following DCOPF (Direct • 3) Unlike some previous study which treats power grid Current Optimal Power Flow) is adopted (Chen et al., 2019) purely as a graph model with no or less physical back- to mimic the bottom-level control measures, i.e., changing ground, this paper uses a professional power system generator outputs and shedding loads (if necessary). simulator as the environment in the RL framework to better reflect actual characteristics of the power system. X X In this way, the learning result is more convincing, and min cipi + dj(pj − Pdj) pi;pj the trained mitigation strategy will be more practical. i2G j2D s.t. F = Ap The remaining parts of this paper are organized as follows. n X Section 2 proposes an RL-based control framework for the pk = 0 (1) mitigation of cascading failures. Section 3 presents the case k=1 study and results analysis. Finally, conclusions and future Pdj ≤ pj ≤ 0; j 2 D directions are given in Section 4. min max Pgi ≤ pj ≤ Pgi ; i 2 G − F max ≤ F ≤ F max; l 2 L 2. Multi-Stage Cascading Failure Control L l L 2.1. Multi-Stage Cascading Failure (MSCF) Problem Firstly, the following definitions are given: Table 1. Result of Episode-1. Generation: one event of the cascading failures within one ACPF OVER-LIMIT Stage-1 stage, e.g., a line tripping (Qi et al., 2015). CONVERGE LINES Generation-1 YES 0 Stage: after an attack (e.g., one line is broken by a natural ACPF OVER-LIMIT Stage-2 disaster), the grid evolves with a series of potential gener- CONVERGE LINES ations (e.g., line tripping events if the line thermal limits Generation-2 YES 0 are reached). At the end of each stage, the power system will either 1) reach a new equilibrium point if the ACPF Result WIN Power Grid Cascading Failure Mitigation by Reinforcement Learning Where, n is the total bus number. G, D, L are respectively • Several quantities of each bus and the power flow of the generator set, load set and branch set; F = [Fl](l 2 L) each branch are chosen and packed as the state, i.e., represents the branch flow; pi(i 2 G) is the generation dis- state=[branch loading status; V1; θ1;P1;Q1; :::; Vn; patch for the i-th generator; pj(j 2 D) is the load dispatch θn;Pn;Qn], where, branch loading status are the per- for the j-th load; p = [pk], k = 1:::n represents the (net) centage values calculated by dividing each branch flow by its nodal power injections. A is a constant matrix to associate loading limit for all the branches; Vi; θi;Pi;Qi(i = 1: : : n) the (net) nodal power injections with the branch flows.