METHOD FOR MULTI-TIME SCALE VOLTAGE QUALITY CONTROL BASED ON REINFORCEMENT LEARNING IN A POWER DISTRIBUTION NETWORK

Abstract

A method for multi-time scale reactive voltage control based on reinforcement learning in a power distribution network is provided, which relates to the field of power system operation and control. The method includes: constituting an optimization model for multi-time scale reactive voltage control in a power distribution network based on a reactive voltage control object of a slow discrete device and a reactive voltage control object of a fast continuous device in the power distribution network; constructing a hierarchical interaction training framework based on a two-layer Markov decision process based on the model; setting a slow agent for the slow discrete device and setting a fast agent for the fast continuous device; and deciding action values of the controlled devices by each agent based on measurement information inputted, so as to realize the multi-time scale reactive voltage control while the slow agent and the fast agent perform continuous online learning.

Description

FIELD

The disclosure relates to the technical field of power system operation and control, and more particularly to a method for multi-time scale reactive voltage control based on reinforcement learning in a power distribution network.

BACKGROUND

In recent years, “carbon emission peak” and “carbon neutrality” have become strategic goals of our economic and social development, which put forward arduous tasks for an energy system, especially a power system. Based on rapid development of renewable energy power generation (DG) in the 21st century, a transition to clean and low-carbon energy may be further promoted and development of non-fossil energy may be accelerated, especially wind power, solar power and other new energy sources.

As the continuous increase in penetration rate of the DG in a power distribution network, the operation of the power distribution network is caused to face challenges. The refined control and optimization are thus becoming more and more important. In order to cope with a series of problems caused by the continuous increase in penetration rate of the distributed new energy, such as reverse delivery of powers, voltage violations, power quality deterioration, device disconnection, etc., a capability for controlling flexible resources is mined. A reactive voltage control system in the power distribution network has become a key component of improving a safe operation level of the power distribution network, reducing operating costs, and promoting consumption of the distributed resources. However, the current field application of reactive voltage control systems in the power distribution network often adopts a model-driven optimization paradigm, i.e., relying on accurate network models to establish optimization problems and solve control strategies. However, in engineering practice, the reliability of model parameters of the power distribution network is low, and a huge scale and a frequent change result in high maintenance costs for the model. Also, it is difficult to accurately model the influences of external networks and the device characteristics. Due to the influences of the incomplete model, conventional model-driven control system in a regional power grid faces major challenges such as inaccurate control, difficult to implement and promote. Therefore, data-driven model-free optimization methods, are important means for grid reactive voltage control, especially deep reinforcement learning methods that have developed rapidly in recent years.

However, controllable resources in the power distribution network have various types and different characteristics, especially differences in the time scale, which brings fundamental difficulties to data-driven methods and reinforcement learning methods. If a single time scale method is adopted, a waste of controllable resources will be caused and the consumption of renewable energy and may not be fully increased. For example, an installed capacity of DG as a flexible resource is often greater than its rated active power and has a fast response speed. There is a lot of adjustable space. The continuous reactive power set values may be quickly set. In contrast, controllable resources such as on-load tap changers (OLTCs) and capacitor stations have a huge impact on the power distribution network, but they can only adjust fixed gears and produce discrete control actions. At the same time, the interval between actions is long, and there are costs such as wearing. As these two types of devices have serious differences in the action nature and time scale, there is no good solution in the related art to coordinate and optimize the two types of devices under the condition of inaccurate models for the power distribution network. Generally, rough feedback methods are adopted, which is difficult to ensure the optimal operation of the power distribution network.

Therefore, it is necessary to study a method for multi-time scale reactive voltage control in a power distribution network, which can coordinate multi-time scale reactive power resources in the power distribution network for reactive voltage control, without requiring an accurate model for the power distribution network. Online learning of control process data achieves optimal reactive voltage control under the condition of incomplete models. At the same time, since the multi-time scale reactive voltage control in the power distribution network requires continuous online operation, it is necessary to ensure high safety, high efficiency, and high flexibility, so as to greatly improve the voltage quality of the power grid and reduce the network loss of the power grid.

SUMMARY

The purpose of the disclosure is to overcome the deficiencies in the related art and provide a method for multi-time scale reactive voltage control based on reinforcement learning in a power distribution network. The disclosure is specifically suitable for use of power distribution network with serious problems due to the incomplete models, which not only saves the high cost caused by repeated maintenance of accurate models, but also fully mines capabilities of controlling multi-time scale controllable resources guarantees voltage safety and economic operation of the power distribution network to the greatest extent, and be suitable for a large-scale promotion.

A method for multi-time scale reactive voltage control based on reinforcement learning in a power distribution network is provided in the disclosure. The method includes: determining a multi-time scale reactive voltage control object based on a reactive voltage control object of a slow discrete device and a reactive voltage control object of a fast continuous device in a controlled power distribution network, and establishing constraints for multi-time scale reactive voltage optimization, to constitute an optimization model for multi-time scale reactive voltage control in the power distribution network; constructing a hierarchical interaction training framework based on a two-layer Markov decision process based on the model; setting a slow agent for the slow discrete device and setting a fast agent for the fast continuous device; and performing online control with the slow agent and the fast agent, in which action values of the controlled devices are decided by each agent based on measurement information inputted, so as to realize the multi-time scale reactive voltage control while the slow agent and the fast agent perform continuous online learning and updating. The method includes the following steps.

1) determining the multi-time scale reactive voltage control object and establishing the constraints for multi-time scale reactive voltage optimization, to constitute the optimization model for multi-time scale reactive voltage control in the power distribution network comprises:

1-1) determining the multi-time scale reactive voltage control object of the controlled power distribution network:

$\begin{array}{cc}{O}_T={\min }_{T_O,T_B}\sum _{\tilde{t}=0}^{\tilde{T}-1}\left[C_O{T}_{O,\mathrm{loss}}^{\left(k\tilde{t}\right)}+C_B{T}_{B,\mathrm{loss}}^{\left(k\tilde{t}\right)}+C_P{\min }_{Q_G,Q_C}\sum _{\tau =0}^{k-1}{P}_{\mathrm{loss}}^{\left(k\tilde{t}+\tau \right)}\right]& \left(0.1\right)\end{array}$

where {tilde over (T)} is a number of control cycles of the slow discrete device in one day; k is an integer which represents a multiple of a number of control cycles of the fast continuous device to the number of control cycles of the slow discrete device in one day; T=k{tilde over (T)} is the number of control cycles of the fast continuous device in one day; {tilde over (t)} is a number of control cycles of the slow discrete device; T_O is a gear of an on-load tap changer OLTC; T_B is a gear of a capacitor station; Q_G is a reactive power output of the distributed generation DG; Q_C is a reactive power output of a static var compensator SVC; C_O,C_B,C_P respectively are an OLTC adjustment cost, a capacitor station adjustment cost and an active power network loss cost; P_loss^{(k{tilde over (t)}+τ)} is a power distribution network loss at the moment k{tilde over (t)}+τ, τ being an integer, τ=0, 1, 2, . . . , k−1; T_O,loss^{(k{tilde over (t)})} is a gear change adjusted by the OLTC at the moment k{tilde over (t)}, and T_B,loss^{(k{tilde over (t)})} a gear change adjusted by the capacitor station at the moment k{tilde over (t)}, which are respectively calculated by the following formulas:

$$\begin{gathered} \begin{array}{cc}{T}_{O,\mathrm{loss}}^{\left(k\tilde{t}\right)}=\sum _{i=1}^{n_{\mathrm{OLTC}}}❘T_{O,i}^{\left(k\tilde{t}\right)}-T_{O,i}^{\left(k\tilde{t}-k\right)}❘,\tilde{t}>0,i\in \left[1,n_{\mathrm{OLTC}}\right]\text{ \\ TB,loss(k⁢t~)=∑i=1nCB❘"[LeftBracketingBar]"TB,i(k⁢t~)-TB,i(k⁢t-k)❘"[RightBracketingBar]",t~>0,i∈[1,nC⁢B]⁢ \\ TO,loss(0)=TB,loss(0)=0}& \left(0.2\right)\end{array} \end{gathered}$$

where T_O,i^{(k{tilde over (t)})} is a gear set value of an i^th OLTC device at the moment k{tilde over (t)}, n_OLTC is a total number of OLTC devices; T_B,i^{(k{tilde over (t)})} is a gear set value of an i^th capacitor station at the moment k{tilde over (t)}, and n_CB is a total number of capacitor stations;

1-2) establishing the constraints for multi-time scale reactive voltage optimization in the controlled power distribution network:

voltage constraints and output constraints:

V≤V _i ^{(k{tilde over (t)}+τ)} ≤V,

|Q_Gi^{(k{tilde over (t)}+τ)}|≤√{square root over (S_Gi²−(P_Gi^{(k{tilde over (t)}+τ)})²)},

Q _Ci ≤Q_Ci^{(k{tilde over (t)}+τ)}≤Q_Ci,

∀i∈N,{tilde over (t)}∈[0,T),τ∈[0,k)   (0.3)

where N is a set of all nodes in the power distribution network, V_i^{(k{tilde over (t)}+τ)} is a voltage magnitude of the node i at the moment k{tilde over (t)}+τ, V,V are a lower limit and an upper limit of the node voltage respectively; Q_Gi^{(k{tilde over (t)}+τ)} is the DG reactive power output of the node i at the moment k{tilde over (t)}+τ; Q_Ci^{(k{tilde over (t)}+τ)} is the SVC reactive power output of the node i at the moment k{tilde over (t)}+τ; Q_Ci,Q_Ci are a lower limit and an upper limit of the SVC reactive power output of the node i; S_Gi is a DG installed capacity of the node i; P_Gi^{(k{tilde over (t)}+τ)} is a DG active power output at the moment k{tilde over (t)}+τ of the node i;

adjustment constraints:

1≤T_O,i^{(k{tilde over (t)})}≤T_O,i,{tilde over (t)}>0i∈[1,n_OLTC]

1≤T_B,i^{(k{tilde over (t)})}≤T_B,i,{tilde over (t)}>0i∈[1,n_CB]  (0.4)

where T_O,i is a number of gears of the i^th OLTC device, and T_B,i is a number of gears of the i^th capacitor station.

2) constructing the hierarchical interaction training framework based on the two-layer Markov decision process based on the optimization model established in step 1) and actual configuration of the power distribution network, comprises:

2-1) corresponding to system measurements of the power distribution network, constructing a state observation s at the moment t shown in the following formula:

s=(P, Q, V, T_O, T_B)_t   (0.5)

where P, Q are vectors composed of active power injections and reactive power injections at respective nodes in the power distribution network respectively; V is a vector composed of respective node voltages in the power distribution network; T_O is a vector composed of respective OLTC gears, and T_B is a vector composed of respective capacitor station gears; t is a discrete time variable of the control process, (·)_t represents a value measured at the moment t;

2-2) corresponding to the multi-time scale reactive voltage optimization object, constructing feedback variable r_f of the fast continuous device shown in the following formula:

$\begin{array}{cc}{r}_f=-C_P{P}_{\mathrm{loss}}\left(s^{\prime }\right)-C_V{V}_{\mathrm{loss}}\left(s^{\prime }\right)P_{\mathrm{loss}}\left(s^{\prime }\right)=\sum _{i\in N}{P}_i\left(s^{\prime }\right),V_{\mathrm{loss}}\left(s^{\prime }\right)=\sqrt{\sum _{i\in N}\left[{\left[V_i\left(s^{\prime }\right)-\overline{V}\right]}_{+}^2+{\left[\underset{¯}{V}-V_i\left(s^{\prime }\right)\right]}_{+}^2\right]}& \left(0.6\right)\end{array}$

where s,a,s′ are a state observation at the moment t, an action of the fast continuous device at the moment t and a state observation at the moment t+1 respectively; P_loss(s′) is a network loss at the moment t+1; V_loss(s′) is a voltage deviation rate at the moment t+1; P_i(s′) is the active power output of the node i at the moment t+1; V_i(s′) is a voltage magnitude of the node i at the moment t+1; [x]₊=max(0,x); C_V is a cost coefficient of voltage violation probability;

2-3) corresponding to the multi-time scale reactive voltage optimization object, constructing feedback variable r_s of the slow discrete device shown in the following formula:

r _s =−C _O T _O,loss({tilde over (s)},{tilde over (s)}′)−C_BT_B,loss({tilde over (s)},{tilde over (s)}′)−R_f({s_τ,a_τ|τ∈[0, k)},s_k)   (0.7)

where {tilde over (s)},{tilde over (s)}″ are a state observation at the moment k{tilde over (t)} and a state observation at the moment k{tilde over (t)}+k respectively; T_O,loss({tilde over (s)},{tilde over (s)}′) is an OLTC adjustment cost generated by actions at the moment k{tilde over (t)}; T_B,loss({tilde over (s)},{tilde over (s)}′) is a capacitor station adjustment cost generated by actions at the moment k{tilde over (t)}; R_f({s_τ,a_τ|τ∈[0,k)},s_k) is a feedback value of the fast continuous device accumulated between two actions of the slow discrete device, the calculation expression of which is as follows:

$\begin{array}{cc}{R}_f\left(\{s_{\tau },a_{\tau }❘\tau \in [0,k\right)\},s_k)=\sum _{\tau =0}^{k-1}{r}_f\left(s_{\tau },a_{\tau },s_{\tau +1}\right)& \left(0.8\right)\end{array}$

2-4) constructing an action variable a_t of the fast agent and an action variable ã_t of the slow agent at the moment t shown in the following formula:

a _t=(Q_G, Q_C)_t

ã _t=(T_O, T_B)_t   (0.9)

where Q_G, Q_C are vectors of the DG reactive power output and the SVC reactive power output in the power distribution network respectively;

3) setting the slow agent to control the slow discrete device and setting the fast agent to control the fast continuous device, comprise:

3-1) the slow agent is a deep neural network including a slow strategy network {tilde over (π)} and a slow evaluation network Q_s^{{tilde over (π)}}, wherein an input of the slow strategy network {tilde over (π)} is {tilde over (s)}, an output is probability distribution of an action ã, and a parameter of the slow strategy network {tilde over (π)} is denoted as θ_s; an input of the slow evaluation network Q_s^{{tilde over (π)}} is {tilde over (s)}, an output is an evaluation value of each action, and a parameter of the slow evaluation network Q_s^{{tilde over (π)}} are denoted as ϕ_s;

3-2) the fast agent is a deep neural network including a fast strategy network π and a fast evaluation network Q_f^π, wherein an input of the fast strategy network π is s, an output is probability distribution of the action a, and a parameter of the fast strategy network π is denoted as θ_f; an input of the fast evaluation network Q_f^π is (s,a), an output is an evaluation value of actions, and a parameter of the fast evaluation network Q_f^π is denoted as ϕ_f;

4) initializing parameters:

4-1) randomly initializing parameters of the neural networks corresponding to respective agents θ_s, θ_f, ϕ_s, ϕ_f;

4-2) inputting a maximum entropy parameter α_s of the slow agent and a maximum entropy parameter α_f of the fast agent;

4-3) initializing the discrete time variable as t=0, an actual time interval between two steps of the fast agent is Δt, and an actual time interval between two steps of the slow agent is kΔt;

4-4) initializing an action probability of the fast continuous device as p=−1;

4-5) initializing cache experience database as D_l=∅ and initializing agent experience database as D=∅;

5) executing by the slow agent and the fast agent, the following control steps at the moment t:

5-1) judging if t mod k≠0: if yes, going to step 5-5) and if no, going to step 5-2);

5-2) obtaining by the slow agent, state information from measurement devices in the power distribution network;

5-3) judging if D_l≠∅: if yes, calculating r_s, adding an experience sample to D, updating D←D∪{({tilde over (s)},ã,r_s,{tilde over (s)}′,D_l)} and going to step 5-4); if no, directly going to step 5-4);

5-4) updating {tilde over (s)} to {tilde over (s)}′;

5-5) generating the action ã of the slow discrete device with the slow strategy network {tilde over (π)} of the slow agent according to the state information {tilde over (s)};

5-6) distributing ã′ to each slow discrete device to realize the reactive voltage control of each slow discrete device at the moment t;

5-7) obtaining by the fast agent, state information s′ from measurement devices in the power distribution network;

5-8) judging if p≥0: if yes, calculating r_f, adding an experience sample to D_l, updating D_l←D_l∪{(s,a,r_f,s′,p)} and going to step 5-9); if no, directly going to step 5-9);

5-9) updating s to s′;

5-10) generating the action a of the slow discrete device with the fast strategy network {tilde over (π)} of the fast agent according to the state information s and updating p=π(a|s);

5-11) distributing a to each fast continuous device to realize the reactive voltage control of each fast continuous device at the moment t and going to step 6);

6) judging t mod k=0: if yes, going to step 6-1); if no, going to step 7);

6-1) randomly selecting a set of experiences D^B∈D from the agent experience database D, wherein a number of samples in the set of experiences is B;

6-2) calculating a loss function of the parameter ϕ_s with each sample in D^B:

$\begin{array}{cc}L\left({\varphi }_s\right)=\underset{\tilde{s},\tilde{a},r_s,{\tilde{s}}^{\prime }}{E}\left[{\left(Q_s^{\pi }\left(\tilde{s},\tilde{a}\right)-\omega y_s\right)}^2\right]\mathrm{where}& \left(0.1\right)\end{array}$ $\begin{array}{cc}{y}_s=r_s+\gamma \left[Q_s^{\pi }\left({\tilde{s}}^{\prime },{\tilde{a}}^{\prime }\right)-{\alpha }_s\log \tilde{\pi }\left({\tilde{a}}^{\prime }❘\tilde{s}\right)\right]& \left(0.11\right)\end{array}$

where ã′˜{tilde over (π)}(·|{tilde over (s)}) and γ is a conversion factor;

$\begin{array}{cc}\omega =\prod _{{i=0}^{}}^{k-1}\frac{\pi \left(a_i❘s_i\right)}{p\left(a_i❘s_i\right)}& \left(0.12\right)\end{array}$

6-3) updating the parameter ϕ_s:

ϕ_s←ϕ_s−ρ_s∇_ϕsL(ϕ_s)   (0.13)

where ρ_s is a learning step length of the slow discrete device;

6-4) calculating a loss function of the parameter θ_s;

$\begin{array}{cc}L\left({\theta }_s\right)=-\underset{\tilde{s}\in D^B}{E}\left[Q_s^{\pi }\left(\tilde{s},\tilde{a}\sim {\tilde{\pi }}_{{\theta }_s}\left(·❘\tilde{s}\right)\right)\right]& \left(0.14\right)\end{array}$

6-5) updating the parameter θ_s:

θ_s←θ_s−ρ_s∇_θsL(θ_s)   (0.15)

and going to step 7);

7) executing by the fast agent, the following learning steps at the moment t:

7-1) randomly selecting a set of experiences D^B∈D from the agent experience database D, wherein a number of samples in the set of experiences is B;

7-2) calculating a loss function of the parameter ϕ_f with each sample in D^B:

$\begin{array}{cc}L\left({\varphi }_f\right)=\underset{s,a,r_f,s^{\prime }}{E}\left[{\left(Q_f^{\pi }\left(s,a\right)-y_f\right)}^2\right]\mathrm{where}& \left(0.16\right)\end{array}$ $\begin{array}{cc}{y}_f=r_f+\gamma \left[Q_f^{\pi }\left(s^{\prime },a^{\prime }\right)-{\alpha }_f\log \pi \left(a^{\prime }❘s\right)\right]& \left(0.17\right)\end{array}$

where a′˜π(·|s);

7-3) updating the parameter ϕ_f:

ϕ_f←ϕ_f−ρ_f∇_ϕfL(ϕ_f)   (0.18)

where ρ_f is a learning step length of the fast continuous device;

7-4) calculating a loss function of the parameter θ_f;

$\begin{array}{cc}L\left({\theta }_f\right)=-\underset{\tilde{s}\in D^B}{E}\left[Q_f^{\pi }\left(\tilde{s},\tilde{a}\sim {\tilde{\pi }}_{{\theta }_f}\left(·❘\tilde{s}\right)\right)\right]& \left(0.19\right)\end{array}$

7-5) updating the parameter θ_f:

θ_f←θ_f−ρ_f∇_θfL(θ_f)   (0.20)

8) let t=t+1, returning to step 5).

The advantages and beneficial effects of the disclosure lie in:

the reactive power voltage control problem in the disclosure is established as a two-layer Markov decision process, the slow agent is set for a long-time scale device (such as an OLTC, a capacitor station, etc.), and the fast agent is set for a short-time scale device (such as a DG, a static var compensator SVC, etc.). The agents are implemented by reinforcement learning algorithms. The method for hierarchical reinforcement learning according to the present disclosure is used for training, the convergence is efficiently optimized in the interactive control, and each agent may independently decide the action value of the controlled device according to the inputted measurement information, thereby achieving the multi-time scale reactive voltage control. On the one hand, the controllable resources of the multi-time scale devices are fully used, the fast and slow devices are fully decoupled in the control phase to perform fast and slow coordinated multi-time scale reactive voltage control. On the other hand, the method for hierarchical reinforcement learning is proposed with high efficiency, in which a joint learning of fast and slow agents is realized by interaction factors, mutual interferences between the fast and slow agents are avoided in the learning, historical data may be fully used of for learning, an optimal strategy of each agent is quickly obtained, thereby ensuring the optimal operation of the system with incomplete models.

1. Compared with the conventional method for multi-time-scale reactive voltage control, the method for multi-time scale reactive voltage control based on reinforcement learning in a power distribution network according to the disclosure has model-free characteristics, that is, the optimal control strategy may be obtained through online learning without requiring accurate models of the power distribution network. Further, the disclosure may avoid control deterioration caused by model errors, thereby ensuring the effectiveness of reactive voltage control, improving the efficiency and safety of power grid operation, and being suitable for deployment in actual power systems.

2. A fast agent and a slow agent in the disclosure are separately established for the fast continuous device and the slow discrete device. In the control phase, the two agents are fully decoupled, and multi-time scale control commands may be generated. Compared with the conventional method for reactive voltage control based on reinforcement learning, the method in the disclosure may ensure that the adjustment capabilities of the fast continuous device and the slow discrete device are used to the greatest extent, thereby fully optimizing the operation states of the power distribution network and improving the consumption of renewable energy.

3. In the learning process of the fast agent and the slow agent, the joint learning of multi-time-scale agents is realized in the disclosure by interaction factors, mutual interferences between the agents are avoided in the learning phase, and the fast convergence is realized in the reinforcement learning process. In addition, full mining of massive historical data is also supported with high sample efficiency and a fully optimized control strategy may be obtained after a few iterations in the disclosure, which is suitable for scenarios of lacking samples in the power distribution network.

4. The disclosure may realize continuous online operation of the multi-time scale reactive voltage control in the power distribution network, ensure the high safety, high efficiency and high flexibility of the operation, thereby greatly improving the voltage quality of the power grid, reducing the network loss of the power grid and having very high application value.

DETAILED DESCRIPTION

A method for multi-time scale reactive voltage control based on reinforcement learning in a power distribution network is provided in the disclosure. The method includes: determining a multi-time scale reactive voltage control object based on a reactive voltage control object of a slow discrete device and a reactive voltage control object of a fast continuous device in a controlled power distribution network, and establishing constraints for multi-time scale reactive voltage optimization, to constitute an optimization model for multi-time scale reactive voltage control in the power distribution network; constructing a hierarchical interaction training framework based on a two-layer Markov decision process based on the model; setting a slow agent for the slow discrete device and setting a fast agent for the fast continuous device; and performing online control with the slow agent and the fast agent, in which action values of the controlled devices are decided by each agent based on measurement information inputted, so as to realize the multi-time scale reactive voltage control while the slow agent and the fast agent perform continuous online learning and updating. The method includes the following steps.

1) according to a reactive voltage control object of a slow discrete device (which refers to a device that performs control by adjusting gears in an hour-level action cycle, such as an OLTC, a capacitor station, etc.) and a reactive voltage control object of a fast continuous device (which refers to a device that performs control by adjusting continuous set values in minute-level action cycle, such as a distributed generation DG, a static var compensator SVC, etc.) in the controlled power distribution network, the multi-time scale reactive voltage control object is determined, and optimization constraints are established for multi-time scale reactive voltage control, to constitute the optimization model for multi-time scale reactive voltage control in the power distribution network. The specific steps are as follows.

1-1) the multi-time scale reactive voltage control object of the controlled power distribution network is determined:

$\begin{array}{cc}{O}_T={\min }_{T_o,T_B}\sum _{\tilde{t}=0}^{\tilde{T}-1}\left[C_O{T}_{O,\mathrm{loss}}^{\left(k\tilde{t}\right)}+C_B{T}_{B,\mathrm{loss}}^{\left(k\tilde{t}\right)}+C_P{\min }_{Q_G,Q_C}\sum _{\tau =0}^{k-1}{P}_{\mathrm{loss}}^{\left(k\tilde{t}+\tau \right)}\right]& \left(0.21\right)\end{array}$

where {tilde over (T)} is a number of control cycles of the slow discrete device in one day; k is an integer which represents a multiple of a number of control cycles of the fast continuous device to the number of control cycles of the slow discrete device in one day; T=k{tilde over (T)} is the number of control cycles of the fast continuous device in one day; {tilde over (t)} is a number of control cycles of the slow discrete device; T_O is a gear of an on-load tap changer OLTC; T_B is a gear of a capacitor station; Q_G is a reactive power output of the distributed generation DG; Q_C is a reactive power output of a static var compensator SVC; C_O,C_B,C_P respectively are an OLTC adjustment cost, a capacitor station adjustment cost and an active power network loss cost; P_loss^{(k{tilde over (t)}+τ)} is a power distribution network loss at the moment k{tilde over (t)}+τ, τ being an integer, τ=0, 1, 2 . . . ,k−1; T_O,loss^{(k{tilde over (t)})} is a gear change adjusted by the OLTC at the moment k{tilde over (t)}, and T_B,loss^{(k{tilde over (t)})} is a gear change adjusted by the capacitor station at the moment k{tilde over (t)}, which are respectively calculated by the following formulas:

$\begin{array}{cc}{T}_{O,\mathrm{loss}}^{\left(k\tilde{t}\right)}=\sum _{i=1}^{n_{\mathrm{OLTC}}}❘T_{O,i}^{\left(k\tilde{t}\right)}-T_{O,i}^{\langle k\tilde{t}-k)}❘,\tilde{t}>0,i\in \left[1,n_{\mathrm{OLTC}}\right]T_{B,\mathrm{loss}}^{\left(k\tilde{t}\right)}=\sum _{i=1}^{n_{\mathrm{CB}}}❘T_{B,i}^{\langle k\tilde{t})}-T_{B,i}^{\langle kt-k)}❘,\tilde{t}>0,i\in \left[1,n_{\mathrm{CB}}\right]T_{O,\mathrm{loss}}^{\langle 0)}=T_{B,\mathrm{loss}}^{\langle 0)}=0& \left(0.22\right)\end{array}$

where T_O,i^{(k{tilde over (t)})} is a gear set value of an i^th OLTC device at the moment k{tilde over (t)}, n_OLTC is a total number of OLTC devices; T_B,i^{(k{tilde over (t)})} is a gear set value of an i^th capacitor station at the moment k{tilde over (t)}, and n_CB is a total number of capacitor stations;

1-2) the constraints are established for multi-time scale reactive voltage optimization in the controlled power distribution network:

the constraints for reactive voltage optimization are established according to actual conditions of the controlled power distribution network, including voltage constraints and output constraints expressed by:

V≤V _i ^{(k{tilde over (t)}+τ)} ≤V,

|Q _Gi ^{(k{tilde over (t)}+τ)}|≤√{square root over (S_Gi²−(P_Gi^{(k{tilde over (t)}+τ)})²)},

Q _ci ≤Q_Ci^{(k{tilde over (t)}+τ)}≤Q_Ci,

∀i∈N, {tilde over (t)}∈[0, T), τ∈[0, k)   (0.23)

where N is a set of all nodes in the power distribution network, V_i^{(k{tilde over (t)}+τ)} is a voltage magnitude of the node i at the moment k{tilde over (t)}+τ, V,V are a lower limit and an upper limit (the typical values are respectively 0.9 and 1.1) of the node voltage respectively; Q_Gi^{(k{tilde over (t)}+τ)} is the DG reactive power output of the node i at the moment k{tilde over (t)}+τ; Q_Ci^{(k{tilde over (t)}+τ)} is the SVC reactive power output of the node i at the moment k{tilde over (t)}+τ; Q_Ci,Q_Ci are a lower limit and an upper limit of the SVC reactive power output of the node i; S_Gi is a DG installed capacity of the node i; P_Gi^{(k{tilde over (t)}+τ)} is a DG active power output at the moment k{tilde over (t)}+τ of the node i;

adjustment constraints are expressed by:

1≤T_O,i^{(k{tilde over (t)}+τ)}≤T_O,i, {tilde over (t)}>0, i∈[1, n_OLTC]

1≤T_B,i^{(k{tilde over (t)}+τ)}≤T_B,i, {tilde over (t)}>0, i∈[1, n_CB]  (0.24)

where T_O,i is a number of gears of the i^th OLTC device, and T_B,i is a number of gears of the i^th capacitor station.

2) in combination with the optimization model established in step 1) and actual configuration of the power distribution network, the hierarchical interaction training framework based on the two-layer Markov decision process is constructed. The specific steps are as follows:

2-1) corresponding to system measurements of the power distribution network, a state observation s at the moment t is constructed in the following formula:

s=(P, Q, V, T_O, T_B)_t   (0.25)

where P, Q are vectors composed of active power injections and reactive power injections at respective nodes in the power distribution network respectively; V is a vector composed of respective node voltages in the power distribution network; T_O is a vector composed of respective OLTC gears, and T_B is a vector composed of respective capacitor station gears; t is a discrete time variable of the control process, (·)_t represents a value measured at the moment t;

2-2) corresponding to the multi-time scale reactive voltage optimization object, feedback variable r_f of the fast continuous device is constructed in the following formula:

$\begin{array}{cc}{r}_f=-C_P{P}_{\mathrm{loss}}\left(s^{\prime }\right)-C_V{V}_{\mathrm{loss}}\left(s^{\prime }\right)P_{\mathrm{loss}}\left(s^{\prime }\right)=\sum _{i\in N}{P}_i\left(s^{\prime }\right),& \left(0.26\right)\end{array}$ $V_{\mathrm{loss}}\left(s^{\prime }\right)=\sqrt{\sum _{i\in N}\left[{\left[V_i\left(s^{\prime }\right)-\overline{V}\right]}_{+}^2+{\left[\underset{¯}{V}-V_i\left(s^{\prime }\right)\right]}_{+}^2\right]}$

where s,a,s′ are a state observation at the moment t, an action of the fast continuous device at the moment t and a state observation at the moment t+1 respectively; P_loss(s′) is a network loss at the moment t+1; V_loss(s′) is a voltage deviation rate at the moment t+1; P_i(s′) is the active power output of the node i at the moment t+1; V_i(s′) is a voltage magnitude of the node i at the moment t+1; [x]₊=max(0, x); C_V is a cost coefficient of voltage violation probability;

2-3) corresponding to the multi-time scale reactive voltage optimization object, feedback variable r_s of the slow discrete device is constructed in the following formula:

r _s =−C _O T _O,loss({tilde over (s)}, {tilde over (s)}′)−C_BT_B,loss({tilde over (s)}, {tilde over (s)}′)−R_f({s_τ, a_τ|τ∈[0, k)}, s_k)   (0.27)

where are {tilde over (s)},{tilde over (s)}′ a state observation at the moment k{tilde over (t)} and a state observation at the moment k{tilde over (t)}+k respectively; T_O,loss({tilde over (s)},{tilde over (s)}′) is an OLTC adjustment cost generated by actions at the moment k{tilde over (t)}; T_B,loss({tilde over (s)},{tilde over (s)}′) is a capacitor station adjustment cost generated by actions at the moment k{tilde over (t)}; R_f({s_τ,a_τ|τ∈[0,k)},s_k) is a feedback value of the fast continuous device accumulated between two actions of the slow discrete device, the calculation expression of which is as follows:

$\begin{array}{cc}{R}_f\left(\{s_{\tau },a_{\tau }❘\tau \in [0,k\right)\},s_k)=\sum _{\tau =0}^{k-1}{r}_f\left(s_{\tau },a_{\tau },s_{\tau +1}\right)& \left(0.28\right)\end{array}$

2-4) corresponding to each adjustable resource, an action variable a_t of the fast agent and an action variable ã_t of the slow agent at the moment t are constructed in the following formula:

a _t=(Q_G, Q_C)_t

ã _t=(T_O, T_B)_t   (0.29)

where Q_G, Q_C are vectors of the DG reactive power output and the SVC reactive power output in the power distribution network respectively;

3) the slow agent is set to control the slow discrete device and the fast agent is set to control the fast continuous device. The specific steps are as follows.

3-1) the slow agent is implemented by a deep neural network including a slow strategy network {tilde over (π)} and a slow evaluation network Q_s^{{tilde over (π)}}.

• • 3-1-1) the slow strategy network {tilde over (π)} is a deep neural network with an input being {tilde over (s)} and an output being probability distribution of an action ã, which includes several hidden layers (typically 2 hidden layers), each hidden layer having several neurons (typically 512 neurons), an activation function being the ReLU function, and a network parameter denoted as θ_s;
  • 3-1-2) the slow evaluation network Q_s^{{tilde over (π)}} is a deep neural network with an input being {tilde over (s)} and an output being an evaluation value of each action, which includes several hidden layers (typically 2 hidden layers), each hidden layer having several neurons (typically 512 neurons), an activation function being the ReLU function, and a network parameter denoted as ϕ_s;

3-2) the fast agent is implemented by a deep neural network including a fast strategy network π and a fast evaluation network Q_f^π.

• • 3-2-1) the fast strategy network π is a deep neural network with an input being s and an output being probability distribution of the action a, which includes several hidden layers (typically 2 hidden layers), each hidden layer having several neurons (typically 512 neurons), an activation function being the ReLU function, and a network parameter denoted as θ_f;
  • 3-2-2) the fast evaluation network Q_f^π is a deep neural network with an input being (s,a) and an output being an evaluation value of actions, which includes several hidden layers (typically 2 hidden layers), each hidden layer having several neurons (typically 512 neurons), an activation function being the ReLU function, and a network parameter denoted as ϕ_f;

4) the variables in the relevant control processes are initialized.

4-1) parameters of the neural networks corresponding to respective agents θ_s, θ_f, ϕ_s, ϕ_f are randomly initialized;

4-2) a maximum entropy parameter α_s of the slow agent and a maximum entropy parameter α_f of the fast agent are input, which are respectively configured to control the randomness of the slow and fast agents and a typical value of which is 0.01;

4-3) the discrete time variable is initialized as t=0, an actual time interval between two steps of the fast agent is Δt and an actual time interval between two steps of the slow agent is kΔt, which are determined according to the actual measurements of the local controller and the command control speed;

4-4) an action probability of the fast continuous device is initialized as p=−1;

4-5) experience databases are initialized, in which cache experience database is initialized as D_l=∅ and agent experience database is initialized as D=∅;

5) the slow agent and the fast agent execute the following control steps at the moment t:

5-1) it is judged whether t mod k≠0. If yes, step 5-5) is performed and if no, step 5-2) is performed;

5-2) the slow agent obtains state information {tilde over (s)}′ from measurement devices in the power distribution network;

5-3) it is judged whether D_l≠∅. If yes, r_s is calculated, an experience sample is added to D, D←D∪{{tilde over (s)},ã,r_s,{tilde over (s)}′,D_l)} is updated, and step 5-4) is performed; if no, step 5-4) is directly performed;

5-4) let {tilde over (s)}←{tilde over (s)}′;

5-5) the action ã of the slow discrete device is generated with the slow strategy network {tilde over (π)} of the slow agent according to the state information {tilde over (s)};

5-6) ã is distributed to each slow discrete device to realize the reactive voltage control of each slow discrete device at the moment t;

5-7) the fast agent obtains state information s′ from measurement devices in the power distribution network;

5-8) it is judged whether p≥0. If yes, r_f is calculated, an experience sample is added to D_l, D_l←D_l∪{(s,a,r_f,s′,p)} is updated, and step 5-9) is performed; if no, step 5-9) is directly performed;

5-9) let s←s′;

5-10) the action a of the fast continuous device is generated with the fast strategy network {tilde over (π)} of the fast agent according to the state information s and p=π(a|s) is updated;

5-11) a is distributed to each fast continuous device to realize the reactive voltage control of each fast continuous device at the moment t and step 6) is performed;

6) it is judged whether t mod k=0. If yes, step 6-1) is performed; if no, step 7) is performed;

6-1) a set of experiences D^B∈D is randomly selected from the agent experience database D, wherein a number of samples in the set of experiences is B (a typical value is 64);

6-2) a loss function of the parameter ϕ_s is calculated with each sample in D^B:

$\begin{array}{cc}L\left({\varphi }_s\right)=\underset{\tilde{s},\tilde{a},r_s,{\tilde{s}}^{\prime }}{E}\left[{\left(Q_s^{\pi }\left(\tilde{s},\tilde{a}\right)-\omega y_s\right)}^2\right]& \left(0.3\right)\end{array}$

where

$\underset{\tilde{s},\tilde{a},r_s,{\tilde{s}}^{\prime }}{E}$

is taken from D^B and y_s is determined by:

y _s =r _s+γ[Q_s^π({tilde over (s)}′, ã′)−α_s log {tilde over (π)}(ã′|{tilde over (s)})]   (0.31)

where ã′˜{tilde over (π)}(·|{tilde over (s)}) and γ is a conversion factor, a typical value of which is 0.98;

$\begin{array}{cc}\omega =\prod _{i=0}^{k-1}\frac{\pi \left(a_i❘s_i\right)}{p\left(a_i❘s_i\right)}& \left(0.32\right)\end{array}$

6-3) the parameter ϕ_s is updated:

ϕ_s←ϕ_s−ρ_s∇_ϕsL(ϕ_s)   (0.33)

where ρ_s is a learning step length of the slow discrete device, a typical value of which is 0.0001;

6-4) a loss function of the parameter θ_s is calculated:

$\begin{array}{cc}L\left({\theta }_s\right)=-\underset{\tilde{s}\in D^B}{E}\left[Q_s^{\pi }\left(\tilde{s},\tilde{a}~{\tilde{\pi }}_{{\theta }_s}\left(·❘\tilde{s}\right)\right)\right]& \left(0.34\right)\end{array}$

6-5) the parameter θ_s is updated:

θ_s←θ_s−ρ_s∇_θsL(θ_s)   (0.35)

and step 7) is then performed;

7) the fast agent executes the following learning steps at the moment t:

7-1) a set of experiences D^B∈D is randomly selected from the agent experience database D, wherein a number of samples in the set of experiences is B (a typical value is 64);

7-2) a loss function of the parameter ϕ_f is calculated with each sample in D^B:

$\begin{array}{cc}L\left({\varphi }_f\right)=\underset{s,a,r_f,s^{\prime }}{E}\left[{\left(Q_f^{\pi }\left(s,a\right)-y_f\right)}^2\right]& \left(0.36\right)\end{array}$

where

$\underset{s,a,r_f,s^{\prime }}{E}$

is taken from D^B and y_f is determined by:

y _f =r _f+γ[Q_f^π(s′,a′)−α_f log π(a′|s)]  (0.37)

where a′˜π(·|s);

7-3) the parameter ϕ_f is updated:

ϕ_f←ϕ_f−ρ_f∇_ϕfL(ϕ_f)   (0.18)

where ρ_f is a learning step length of the fast continuous device, a typical value of which is 0.00001;

7-4) a loss function of the parameter θ_f is calculated:

$\begin{array}{cc}L\left({\theta }_f\right)=-\underset{s\in D^B}{E}\left[Q_f^{\pi }\left(s,a~{\pi }_{{\theta }_f}\left(·❘s\right)\right)\right]& \left(0.39\right)\end{array}$

7-5) the parameter θ_f is updated:

θ_f←θ_f−ρ_f∇_θfL(θ_f)   (0.20)

8) let t=t+1, it returns to step 5) and repeats the steps 5) to 8). The method is an online learning control method, which continuously runs online and updates the neural network, while performing online control until the user manually stops it.

Claims

1. A method for multi-time scale reactive voltage control based on reinforcement learning in a power distribution network, comprising: determining a multi-time scale reactive voltage control object based on a reactive voltage control object of a slow discrete device and a reactive voltage control object of a fast continuous device in a controlled power distribution network, and establishing constraints for multi-time scale reactive voltage optimization, to constitute an optimization model for multi-time scale reactive voltage control in the power distribution network;constructing a hierarchical interaction training framework based on a two-layer Markov decision process based on the model;setting a slow agent for the slow discrete device and setting a fast agent for the fast continuous device; andperforming online control with the slow agent and the fast agent, in which action values of the controlled devices are decided by each agent based on measurement information inputted, so as to realize the multi-time scale reactive voltage control while the slow agent and the fast agent perform continuous online learning and updating.

2. The method as claimed in claim 1, further comprising: 1) determining the multi-time scale reactive voltage control object and establishing the constraints for multi-time scale reactive voltage optimization, to constitute the optimization model for multi-time scale reactive voltage control in the power distribution network comprises:1-1) determining the multi-time scale reactive voltage control object of the controlled power distribution network: