Optimal Charging and Discharging Control for Hybrid Energy Storage System based on Reinforcement Learning

Abstract

Systems and methods are disclosed to manage a microgrid with a hybrid energy storage system (HESS) includes deriving a dynamic model of a bidirectional-power-converter (BPC)-interfaced HESS; applying a first neural network (NN) to estimate a system dynamic; and applying a second NN to calculate an optimal control input for the HESS through online learning based on the estimated system dynamics.

Description

TECHNICAL FIELD

This invention is related to systems and methods to reinforcement-learning-based optimal control of the hybrid energy storage system in real time.

BACKGROUND

The development of microgrid technologies featuring renewable distributed energy resources (DERs) have brought new opportunities as well as challenges to conventional distribution systems. Meanwhile, hybrid energy storage system (HESS), e.g., combination of batteries and ultra-capacitors, are deployed to compensate the intermittency of renewable DERs and participate in the real-time demand-supply accommodation, which also helps to defer the extraordinary cost of updating the conventional power grids. This unveils a completely new path to alternate the traditional operation pattern of power systems, which creates significant benefits and convenience for both power suppliers and customers. Currently, HESS consisting of lithium-ion batteries (LIB) and ultra-capacitors (UC) have been widely incorporated in microgrids. However, the discrepancies of energy and power densities result in the disparate dynamic inertias between LIB and UC. Therefore, proper real-time control of HESS with promising transient performance becomes a challenging problem.

In a microgrid with a hierarchical control structure, the state of charge (SOC) of energy storage unit (ESU) can be decided by the centralized tertiary control and realized by distributed secondary/primary control. In the charging process, ESU works as a load bank, while in the discharging process, it performs as a DER. For the rest of time, ESU should be isolated from the system to avoid continuously repeated charging and discharging (C&D) caused by the self-discharging effect. Nevertheless, due to the low inertia of microgrids, the initializing and switching C&D processes for ESU may lead to nuisance disturbances, especially in fast C&D scenarios. The unexpected disturbance significantly degrades the power quality and might damage sensitive loads, such as data centers. It may even trigger false protection schemes under the worst scenarios. This disobeys the original purpose of implementing HESS. Therefore, the desired control policy should provide a smooth C&D solution for HESS in a decentralized manner with plug & play capability.

Conventionally, constant current (CC) and proportion-integration (PI)-based controls are two of the most popular methods in industrial applications. CC control has been widely used in low voltage electronic devices such as cellphone and laptop due to its implementation simplicity. However, the starting and terminating of the CC controller will introduce significant voltage disturbances, which is harmful to microgrids. From this perspective, PI-based control methods can slightly improve the transient performance, since large disturbances are introduced only during the initializing periods of C&D. Even though, PI-based methods generally require excessive parameter tuning efforts and rely on the awareness of system dynamics. Once system parameters deviate from the original set point, e.g., due to aging or heating issues, the performance of PI-based controllers will significantly degrade. Additionally, in practice, the outer voltage loop of double-loop PI controller is usually simplified as a proportional (P) controller to avoid over-C&D, which also limits the performance of PI-based controllers. A common issue is that PI-based algorithms introduce considerable disturbances to the system during C&D processes.

SUMMARY OF THE INVENTION

In one aspect, a reinforcement-learning-based online optimal (RL-OPT) control method realizes the smooth C&D control of HESS. A hybrid AC/DC microgrid involving PV, UC, LIB and DG are considered during the controller design process. Firstly, the dynamic model of modified bidirectional-power-converter (BPC)-interfaced HESS is derived. Considering that the internal impedance of each ESU is unknown, one NN is developed to estimate the system dynamic online. Then, another NN is applied to calculate the optimal control input for the HESS through online learning based on the estimated system dynamics. The control scheme also considers both grid-tied and islanded modes of the microgrids. In grid-tied mode, the main grid is considered as an infinite source which maintains the bus voltage and reactive power at the point of common coupling (PCC) through the voltage source converter (VSC). While in islanded mode, DGs are deployed to maintain the bus voltage at PCC. For either scenario, PV works under the maximum power point tracking (MPPT) mode to maximize the renewable DER utilization.

In another aspect, a method to manage a microgrid with hybrid energy storage system (HESS) includes deriving a dynamic model of a bidirectional-power-converter (BPC)-interfaced HESS; applying a first neural network (NN) to estimate a system dynamic; and applying a second NN to calculate an optimal control input for the HESS through online learning based on the estimated system dynamics.

Advantages of the preferred embodiment may include one or more of the following. The effectiveness of RL-OPT method has been tested through both software simulations and hardware-in-loop (HIL) experiments. The optimal control problem of HESS is formulated using RL method to reduce the disturbances caused by C&D of various energy storage device. The model-free method has the adaptivity for different system dynamics based on the input/output data without the system parameter information. The bidirectional converter topology is designed to avoid redundancy of C&D circuit as well as the self-discharging problem. The RL-OPT controller takes the SOC measurement of the ESU and calculates the optimal control input for the BPC in a decentralized manner. Thus, implementation of the method can be achieved with minimum communication efforts and the plug & play capability of HESS can be easily realized In addition, the RL-OPT does not need current measurement units comparing to the conventional PI-based control methods. Even though, the performance of the control method is significantly improved.

Other features and advantages of the invention will be apparent from the following description of the preferred embodiments thereof, and from the claims.

BRIEF DESCRIPTIONS OF FIGURES

The subject matter that is regarded as the invention may be best understood by reference to the following description taken in conjunction with the accompanying drawing figures in which:

FIG. 1 shows a diagram of the considered hybrid AC/DC microgrid.

FIG. 2 shows a switch-level UC interface circuit.

FIG. 3 shows a schematic of the control implementation.

FIG. 4 shows a discharging response of UC in islanded mode.

FIG. 5 shows a charging response of LIB in islanded mode.

FIG. 6 shows a DC bus voltage response in islanded mode.

FIG. 7 shows activation weights of NN in islanded mode.

FIG. 8 shows a discharging response of UC in grid-tied mode.

FIG. 9 shows a charging response of LIB in grid-tied mode.

FIG. 10 shows a DC bus Voltage response in grid-tied mode.

FIG. 11 shows activation weights of NN in grid-tied mode.

FIG. 12 shows a topology configuration of tested system.

FIG. 13 shows a laboratory setup of the HIL testbed.

FIG. 14 shows HIL experiment results: DC bus voltage, UC voltage, and UC charging current.

DETAILED DESCRIPTION

As presented in FIG. 1, a reinforcement-learning-based online optimal (RL-OPT) control method is shown for hybrid energy storage system (HESS) in AC/DC microgrids involving photovoltaic (PV) systems and diesel generators (DG). Due to the low system inertia, conventional unregulated charging and discharging (C&D) of energy storages in microgrids may introduce disturbances that degrade the power quality and system performance, especially in fast C&D situations. Secondary and tertiary control levels can optimize the state of charge (SOC) reference of HESS; however, they are lacking the direct controllability of regulating the transient performance. Additionally, the uncertainties in practical systems greatly limit the performance of conventional model-based controllers. In this study, the optimal control theory is applied to optimize the C&D profile and to suppress the disturbances caused by integrating HESS. Neural networks (NN) are devised to estimate the nonlinear dynamics of HESS based on the input/output measurements, and to learn the optimal control input for bidirectional-converter-interfaced HESS using the estimated system dynamics. Because the RL-OPT method is fully decentralized, which only requires the local measurements, the plug & play capability of HESS can be easily realized. Both islanded and grid-tied modes are considered.

The studied microgrid consists of both AC and DC buses, which are interconnected through a bidirectional DC/AC voltage source converter (VSC). The PV array (in MPPT mode) and a HESS involving both UC and LIB are connected on the DC bus. Grid-tied and islanded operation mode switching of the microgrid is realized by operating the circuit breaker (CB).

The corresponding switch-level C&D circuit with detailed ESU model is presented in FIG. 2. In the modified BPC, switch S₃ is added to avoid the self-discharging problem during the initiating period of system. Switches S₁ and S₂ are used to control the BPC in buck or boost mode to charge or discharge the ESU in HESS, respectively.

Basically, the dynamics of a standard BPC-interfaced-UC model can be represented as

$\begin{array}{cc}\{\begin{array}{c}\stackrel{.}{V_c}=-\frac{V_c}{C_{\mathrm{uc}}R_{\mathrm{pc}}}+\frac{I_c}{C_{\mathrm{uc}}}\\ U_c=V_c+R_{\mathrm{sc}}I_c\\ \stackrel{.}{I_c}=\frac{1}{L_f}\left(U_c-R_fI_c-\alpha V_t\right)\end{array}& \left(1\right)\end{array}$

in which

$\begin{array}{cc}\alpha =\{\begin{array}{cc}\frac{1}{1-D},& \forall S_1\mathrm{is}\mathrm{off}\\ \frac{1}{D},& \forall S_2\mathrm{is}\mathrm{off}\end{array}& \left(2\right)\end{array}$

and D is the duty ratio of PWM. R_f, L_f and C_f are filter resistance, inductance and capacitance, respectively. R_pc and R_sc are the internal parallel and series resistance of UC, respectively. Similarly, R_pb and R_sb are the internal parallel and series resistance of LIB, respectively. V_c and U_c are the internal and external voltages of ESU, respectively. V_t is the terminal bus voltage. Then, the dynamics in Eqn. (1) can be further written as

$\begin{array}{cc}{\stackrel{.}{U}}_c=\left(\frac{R_{\mathrm{sc}}}{L_f}-\frac{1}{C_{\mathrm{uc}}R_{\mathrm{pc}}}\right)U_c-\frac{R_{\mathrm{sc}}V_t}{L_f}\alpha +\left(\frac{R_{\mathrm{pc}}+R_{\mathrm{sc}}}{C_{\mathrm{uc}}R_p}-\frac{R_{\mathrm{sc}}R_f}{L_f}\right)I_c& \left(3\right)\end{array}$

By defining the tracking error as e(t)=U_e(t)−U*_c with U*_c being a constant SOC reference, the error dynamics of (3) can be represented in a more condensed way as

ė(t)=f(e(t))+g(t)u(t)+D(t),e(0)=e₀  (4)

where ė(t)={dot over (U)}_c(t)−{dot over (U)}*_c=U_c(t)−0={dot over (U)}_c(t); f(e)=(R_sc/L_f−1/C_ucR_pc)U_c and g(t)=R_scV_t/L_f are unknown nonlinear system dynamics. u(t)=a is the control input. D(t)=[(R_pc+R_sc)/C_ucR_p−R_scR_f/L_f]I_c is the perturbation term of the system with D(0)=0 and is bounded as ∥D(t)∥≤d_max. It should be noted that the error dynamics of LIB can be similarly represented in a form as Eqn. (4) and the derivative process is omitted here.

The performance of conventional PI-based control methods on a nonlinear uncertain system in Eqn. (4) would be very limited, especially when system dynamics are unknown. The trial & error tuning is also impractical and not reliable enough to guarantee the system performance.

Firstly, considering a nominal nonlinear system without uncertainty D, i.e., ė(t)=f(e(t))+g(t)u(t), the infinite-horizon integral cost function can be designed as

ė(t)=f(e(t))+g(t)u(t)+D(t),e(0)=e₀  (5)

where r(e,u)=Q(e)+u^TRu with R being a symmetric positive definite matrix and Q(e)=e^T Pe being a positive definite function of e.

There exists a control law u(e) that guarantees the asymptotic stability of the closed-loop nonlinear system in Eqn. (4) when the preconditions in Eqn. (5) can be satisfied with respect to a positive definite continuously differentiable function V(e), a bounded function F(e), and a feedback control law u(e).

$\begin{array}{cc}\{\begin{array}{c}{V}_{\partial e}^TD\left(t\right)\le \Gamma \left(e\right)\\ V_{\partial e}^T\left[f\left(e\right)+g\left(t\right)u\right]+\Gamma \left(e\right)+Q\left(e\right)+u^T\mathrm{Ru}=0\end{array}& \left(6\right)\end{array}$

where V_∂e is the partial derivative of the cost function V(e) with respect to e. Then, cost function Eqn. (4) satisfies

$\begin{array}{cc}\underset{D\left(t\right)\in M}{\sup }=J\left(e_0,u\right)\le J_d\left(e_0,u\right)=V\left(e_0\right)& \left(7\right)\end{array}$

where “sup” denotes the supremum operator that finds the minimal cost J_d(e₀,u) greater than or equal to J(e₀,u) for any perturbation D(t)∈M,M={D(t)|D(t)∈, ∥D(t)∥≤d_max}. J_d(e₀,u) is the modified cost function for nonlinear system with uncertainty, which can be designed as

J _d(e₀,u)=∫₀^∞[r(e,u)+Γ(e)]dt  (8)

Then, the Eqn. (8) can be further written as

J _d(e₀,u)=V(e₀)=∫₀^T[r(e,u)+Γ(e)]dt=∫_T^∞[r(e,u)+Γ(e)]dt=∫₀^T[r(e,u)+Γ(e)]d+V(e)  (9)

Since V(e) is continuously differentiable, Eqn. (9) becomes

$\begin{array}{cc}\underset{T\to 0}{\lim }\frac{V\left(e_0\right)-V\left(e\right)}{T}=\underset{T\to 0}{\lim }\frac{1}{T}\stackrel{T}{\underset{0}{\int }}\left[r\left(e,u\right)+\Gamma \left(e\right)\right]\mathrm{dt}\Rightarrow \stackrel{.}{V}\left(e\right)=V_{\partial e}^T\left[f\left(e\right)+g\left(t\right)u+D\right]=-r\left(e,u\right)-\Gamma \left(e\right)\Rightarrow 0=V_{\partial e}^T\left[f\left(e\right)+g\left(t\right)u+D\right]+r\left(e,u\right)+\Gamma \left(e\right)& \left(10\right)\end{array}$

It can be observed that Eqn. (10) is an infinitesimal version of Eqn. (9). Based on Eqn. (10), Hamiltonian of the optimal control problem can be defined as

H(e,u,V_∂e)=Q(e)+u^TRu+V_∂e^T[f(e)+g(t)u+D]+Γ(e)  (11)

Correspondingly, the optimal cost function can be designed as

V*(e)=min_u∈Ω∫₀^T[r(e,u)+Γ(e)]dt  (12)

The objective of optimal cost function in Eqn. (2) is to achieve the least tracking error using the minimal control effort with unknown disturbances, and therefore, the control process is to design the optimal controller that leads the states to track their references with a minimal tracking error. Then, Eqn. (12) can be obtained by solving the Hamilton-Jacobi-Bellman (HJB) equation as

min_u∈ΩH(e,u,V*_∂e)=0  (13)

By taking the partial derivative of HJB equation, i.e., ∂H(e,u,V*_∂e)/∂u=0, the optimal control law u* can be derived as

u*=−½R⁻¹g(t)^TV*_∂e  (14)

Accordingly, the bounded function can be designed as

Γ(e)=¼V_∂e^TV_∂e+d_max²  (15)

It can be easily proven that Γ(e) in Eqn. (15) satisfies the condition in Eqn. (6), i.e., V_∂e^TD(t)≤Γ(e). Substituting Eqn. (14) and (15) into Eqn. (13), the HJB equation in terms of V*_∂e can be represented as

0=Q(e)+¼V*_∂e^TV*_∂e+d_max²+V*_∂e^T[f(e)+D]−¼V*_∂e^Tg(t)R⁻¹g(t)^TV*_∂e  (16)

Theorem 1

(Optimal Control Policy u*): Consider any non-linear uncertain systems presented in Eqn. (4) with cost function defined in Eqn. (7) and HJB equation defined in Eqn. (16), provided any admissible control u, the cost function Eqn. (7) is smaller than a guaranteed cost bound J_b given as

J _b =V*(e₀)+∫₀^T(u−u*)^TR(u−u*)dt  (17)

If u=u*, the cost J_b is guaranteed to be minimized, i.e., J_b=V*(e₀).

Proof:

According to Eqn. (10) and the definition of V*(e), the cost function Eqn. (7) with respect to any arbitrary u can be rewritten as

J(e₀,u)=V*(e₀)+∫₀^T[r(e,u)+{dot over (V)}*(e)]dt  (18)

By Eqn. (10) and Eqn. (16), one can obtain that

r(e,u)+{dot over (V)}*(e)=Q(e)+u^TRu+V*_∂e^T[f(e)+g(t)u+D]=u^TRu+V*_∂e^Tg(t)u+¼V*_∂e^Tg(t)R⁻¹g(t)^TV*_∂e−¼V*_∂e^TV*_∂e−d_max²≤u^TRu+V*_∂e^Tg(t)u+¼V*_∂e^Tg(t)R⁻¹g(t)^TV*_∂e  (19)

Recalling Eqn. (14), Eqn. (19) can be compiled into a square form with respect to R⁻¹g(t)^TV*_∂e/2 as

r(e,u)+{dot over (V)}*(e)≤(u−u*)^TR(u−u*)  (20)

which implies that Eqn. (14) holds. Thus, if u=u*, the cost J_b is guaranteed to be minimized, i.e., J_b=V*(e₀), and the corresponding optimal control input is u*. Proof completed. ♦

The above optimal control is derived based on known nonlinear system dynamics. However, it is very difficult to numerically solve the optimal control problem of a nonlinear system, especially when system dynamics is unknown.

Generally, the optimal control input u* is calculated based on the solution of HJB (Eqn. (16)). However, in practice, the system uncertainty makes the nonlinear partial derivative function too complicated to be solved directly. Therefore, two NNs are developed in this section to realize the adaptive online learning of optimal control policy. Firstly, an NN is designed to estimate the unknown system dynamics, based on which another NN is developed to solve the optimal policy. The detailed control implementation process of the controller is elaborated below.

A. System Dynamic Identifier Design

According to the universal approximation capability of NN, the system dynamics in Eqn. (4) can be represented by a single-layer NN as

ė(t)=f(e(t))+g(t)u(t)(t)t=W*_e^Tσ₁(e)+W*_u^Tσ₁(u)+W*_d^T1+ϵ₁=W*₁^Tσ₁(e,u)+ϵ₁  (21)

where W*₁=[W*_e W*_u W*_d]^T∈^N×1 are ideal unknown weights of NN identifier approximating system dynamics f(e), g(t) and D. σ₁(e,u)=[σ₁(e) σ₁(u) 1]∈^N×1 is the activation function where 1 represents the vector of ones. N is the number of hidden-layer neurons. ϵ₁ is the NN reconstruction error. W*₁ and ϵ₁ are assumed to be bounded as ∥W*₁∥≤W_1M and ∥ϵ₁∥≤ϵ_1M, respectively. It should be mentioned that since D is a perturbation term associated with the system dynamics rather than a random external disturbance, term D satisfies the conditions to be estimated by NN. Then, the tracking error dynamics estimator can be designed as

{dot over (ê)}(t)=Ŵ₁^Tσ₁(e,u)+k₁{tilde over (e)}(t)  (22)

where Ŵ₁=[Ŵ_e Ŵ_u Ŵ_d]^T∈^N×1 are the estimated NN identifier weights and k₁ is the selected parameter to maintain the NN identifier stability. Defining {tilde over (e)}(t)=e(t)−ê(t) as the estimation error of tracking error, the dynamics of {tilde over (e)}(t) can be represented as

{dot over ({tilde over (e)})}(t)=ė(t)−{dot over (ê)}(t)=W*₁^Tσ₁(e,u)+ϵ₁−Ŵ₁^Tσ₁(e,u)−k₁{tilde over (e)}(t)={tilde over (W)}₁^Tσ₁(e,u)+ϵ₁−k₁{tilde over (e)}(t)  (23)

where the NN weight estimation error is defined as {tilde over (W)}₁(t)=W*₁−Ŵ₁(t), and furthermore, {dot over ({tilde over (W)})}₁(t)=−{dot over (Ŵ)}₁(t). To force the estimated NN identifier weight Ŵ₁(t) converging to the target weight W*₁, the updating law for Ŵ₁ can be designed as

{dot over (Ŵ)}₁=−k₂Ŵ₁^Tσ₁(e,u)+σ₁(e,u)e(t)  (24)

where k₂ is a positive tuning parameter of NN identifier.

Theorem 2

(Boundedness of NN identifier): Using the NN identifier in Eqn. (22) with updating law in Eqn. (24) and letting the activation function σ₁(e,u) satisfy the persistency of excitation (PE) condition. Given the initial NN identifier weight as Ŵ₁(0) residing in a compact set Ω, there exists a positive tuning parameter k₂ such that the identification error {tilde over (e)}(t) in Eqn. (23) and NN identifier weight estimation error {tilde over (W)}₁(t) are uniformly ultimately bounded (UUB).

Proof:

Define the following Lyapunov candidate as

L({tilde over (W)},{tilde over (e)})=½{tilde over (W)}₁²+½{tilde over (e)}(t)²  (25)

Then, taking the first derivative of Eqn. (25) and substituting Eqn. (23) and (24), one can obtain that

$\begin{array}{cc}\stackrel{.}{L}={\tilde{W}}_1^T{\stackrel{\stackrel{.}{~}}{W}}_1+{\tilde{e}}^T\stackrel{\stackrel{.}{~}}{e}=-{\tilde{W}}_1^T\left(-k_2{\hat{W}}_1+{\sigma }_1e\right)+{\tilde{e}}^T\left({\tilde{W}}_1^T{\sigma }_1+{ϵ}_1-k_1\tilde{e}\right)=k_2{\tilde{W}}_1^T\left(W_1^{*}-{\tilde{W}}_1\right)-k_1{\tilde{e}}^T\tilde{e}+{\tilde{e}}^T{ϵ}_1\le \frac{1}{2}k_2{\tilde{W}}_1^T{\tilde{W}}_1+\frac{1}{2}k_2W_1^{*T}W_1^{*}-k_2{\tilde{W}}_1^T{\tilde{W}}_1-k_1{\tilde{e}}^T\tilde{e}+{\tilde{e}}^T{ϵ}_1\le -\frac{1}{2}k_2{{\tilde{W}}_1}^2-k_1{\tilde{e}}^2+{ϵ}_b& \left(26\right)\end{array}$

where ϵ_b={tilde over (e)}^Tϵ₁+½k₂W*₁^TW*₁ is a bounded steady-state error. According to the Lyapunov synthesis, one can conclude that the identification error {tilde over (e)}(t) and NN identifier weight estimation error {tilde over (W)}₁(t) are UUB.

B. Adaptive RL-OP T Control Design

Similarly, based on the universal approximation property of NN, the cost function V*(e) can be represented by a single-layer NN on set Ω as

V*(e)=W*₂^Tσ₂(e,u)+ϵ₂  (27)

where W*₂^T ∈^1×N are ideal unknown weights of NN cost function estimator, σ₂(e,u) is the activation function, ϵ₂ is the NN reconstruction error, W*₂ and ϵ₂ are assumed to be bounded as ∥W*₂∥≤W_2M and ∥ϵ₂∥≤ϵ_2M, respectively. Thereafter, the cost function estimator can be designed as

{circumflex over (V)}(e)=Ŵ₂^Tσ₂(e,u)  (28)

Accordingly, the estimated optimal control policy can be derived based on the two NNs in Eqn. (22) and (28) as

û=−½R⁻¹Ŵ_u^T∇σ₂(e,u)^TŴ₂  (29)

where ∇σ₂(e,u)=∂σ₂(e,u)/∂e is the partial derivative of σ₂(e,u) with respect to e. Next, substituting Eqn. (22), (28) and (29) into Eqn. (11), the approximated Hamiltonian of optimal control problem becomes

Ĥ(e,û,{circumflex over (V)})=Q(e)+¼Ŵ₂^T∇σ₂Ŵ_uR⁻¹Ŵ_u^T∇σ₂^TŴ₂[Ŵ₁^T(t)σ₁(e,u)+k₁{tilde over (e)}(t)]+¼Ŵ₂^T∇σ₂∇σ₂^TŴ₂+d_max²  (30)

Because of the impact of system uncertainty and NN reconstruction error, the estimated Hamiltonian cannot hold, i.e., Ĥ(e,û,{circumflex over (V)})≠0. According to the optimal control theory, the estimated cost function can converge close to the ideal target if the approximated Hamiltonian equation approaches to the ideal Hamiltonian, i.e., Ĥ(e,û,{circumflex over (V)})→H(e,u*, V*)=0. Inspired by this, the updating law for tuning the NN weight of cost function estimator can be designed as

$\begin{array}{cc}\stackrel{\stackrel{.}{^}}{W_2}=\frac{k_3}{2}\Theta \left(e,\hat{u}\right)\nabla {\sigma }_2{\hat{W}}_uR^{-1}{\hat{W}}_u^TJ_{1\partial e}-\frac{k_4\omega \hat{H}}{{\left(1+{\omega }^T\omega \right)}^2}& \left(31\right)\end{array}$

where k₃ and k₄ are the designed control coefficients, ω=−[∇σ₂Ŵ_uR⁻¹Ŵ_u^T∇σ₂^TW₂]/2, and Θ(e,û) is an index operator given by

$\begin{array}{cc}\Theta \left(e,\hat{u}\right)=\{\begin{array}{cc}0,& \forall {\stackrel{.}{J}}_1=J_{1\partial e}^T\stackrel{.}{e}<0\\ 1,& \mathrm{otherwise}\end{array}& \left(32\right)\end{array}$

where J₁ is an unbounded Lyapunov candidate and J_1∂e is its partial derivative with respect to e. Moreover, J_1∂e can be defined as

∥ė∥≤c₁∥e∥=(c₂∥J_1∂e∥)^1/4  (33)

where c₁ and c₂ are constants. Note that ∥J_1∂e∥ can be selected to satisfy the general bound, e.g., J₁=⅕(e^Te)^5/2.

Theorem 3

(Convergence of the Optimal Control): Consider the nonlinear uncertain system in Eqn. (4) with control law in Eqn. (29) and NN weights updating law in Eqn. (24) and (31), there exists tuning parameters k₁ to k₄ such that all of signals in the closed-loop system, e.g., tracking error e, error of NN identifier weight {tilde over (W)}₁ and error of NN cost function estimator weight {tilde over (W)}₂ are guaranteed to be UUB. Moreover, the calculated control input û is proved to approximately approach the optimal control input û*.

C. Controller Implementation

The implementation process of the RL-OPT controller is shown in FIG. 3. In addition, the overall procedures of the control method are summarized in Table I. The RL-OPT controller takes the SOC measurement of the ESU and calculates the optimal control input for the BPC in a decentralized manner. Thus, implementation of the method can be achieved with minimum communication efforts and the plug & play capability of HESS can be easily realized. In addition, the RL-OPT does not need current measurement units comparing to the conventional PI-based control methods. Even though, the performance of the control method is significantly improved, which will be verified in later case studies.

TABLE I
METHODOLOGY OF PROPOSED RL-OPT CONTROLLER
1) Initialize control policy û(0) and NN weights Ŵ1(0), Ŵ2(0)
2) while e > e1,
update Ŵ1 using Eqn. (24)
3) Calculate the estimate Hamiltonian Ĥ(ê, û, {circumflex over (V)}) using Eqn. (30)
4) while Ĥ(ê, û, {circumflex over (V)}) ≠ 0,
update Ŵ2 using Eqn. (31)
5) Calculate the optimal control policy û using Eqn. (29)
6) end

The detailed switch-level model is applied in the simulation using Matlab/Simulink Simscape toolbox with a sampling frequency of 10 kHz. The MOSFET module is used for the bidirectional converter as shown in FIG. 2. The Tustin/Backward Euler method is selected for the discrete solver. Both grid-tied and islanded scenarios are tested under various C&D scenarios. The system and controller parameters are given in Table II.

TABLE II
SYSTEM PARAMETERS OF THE
SIMULATION CASE STUDY
	Parameter	Value	Parameter	Value
	Cuc	5.7	F	Rpc	6	kΩ
	Rsc	0.1	Ω	Cpb	4.7	F
	Rpb	5	kΩ	Rab	0.15Ω
	Uoc	5	V	Cf	20	μF
	Rf	0.1	Ω	Lf	4	mH
	k1	2	k2	0.1
	k3	0.8	k4	1

The performance of control method is compared to the conventional PI-based method for benchmarking studies. It should be mentioned that the conventional PI-based control methods usually take a P and PI double-loop structure to avoid the over C&D problem, which requires both voltage and current transducers. From this perspective, the control method only requires one voltage sensor with a much-improved performance, which will be demonstrated in the following case studies.

In case study I, the RL-OPT control method is tested in an islanded microgrid. One DG is used to regulate the DC bus voltage at 48V constantly. At 1s, the UC is discharged from 300% to 29%. While at time 6s, the LIB is charged from 30% to 31%. Firstly, the discharging current and SOC of UC are shown in FIG. 4. While the charging current and SOC of LIB are plotted in FIG. 5. Additionally, the response of DC bus voltage is presented in FIG. 6.

As can be observed, the conventional PI-based control method produces large disturbances to the system (blue curves). The sharply changing C&D current and SOC lead to a huge disruption to the DC bus, e.g., over 5V (>10%) overshoot. The considerable disturbances are harmful to the system, especially for the sensitive loads and power electronic devices. On the contrary, the RL-OPT control method is able to optimize the entire C&D profile (red curves). It can be seen that the C&D currents of ESU get greatly smoothed, which consequently reduces the voltage overshoot of DC bus to be less than 1V (<2%). The responses of activation weights W, is shown in FIG. 7, among which FIG. 7(a) is the weight of UC controller and FIG. 7(b) is the weight of LIB controller, respectively. As the weights converge to their desired target, the optimal control is achieved.

In case study II, the RL-NN controller is tested under the grid-tied microgrid. The simulation settings are same as that of the case I, except that the DC bus voltage is maintained by the external grid through a VSC. Similarly, the C&D current and SOC of UC and LIB are presented in FIG. 8 and FIG. 9, respectively. The responses of DC bus voltage is shown in FIG. 10. The responses of activation weights W₁ is shown in FIG. 11, among which FIG. 11(a) is the weight of UC controller and FIG. 11(b) is the weight of LIB controller, respectively. As the weights converge to their desired targets, the optimal control is achieved. As can be seen, the C&D profile as well as the activation weights of UC and LIB in grid-tied mode are almost the same as in islanded mode. However, the disturbances on DC bus voltage is significantly reduced because the main grid can provide a relatively stable voltage support compared to DG. Since the controller has already achieved the optimal control in the islanded mode, not much improvement can be observed in the grid-tied mode. Nevertheless, the performance of RL-NN controller is still better than the conventional PI-based controller in terms of C&D currents and unexpected disturbances. In addition, a detailed comparison between conventional control method and method is given in Table III.

TABLE III
SIMULATION RESULTS COMPARISON
	Islanded-mode		Grid-tied mode
	Vsurge	Isurge	Vsurge	Isurge
	Proposed RL	0.85 V	4.74 A	0.82 V	4.74 A
	method	(1.78%)		(1.70%)
	Conventional	5.16 V	20.05 A	1.35 V	20.05 A
	method	(10.7%))		(2.81%)

In case study III, the developed RL-OPT controller is fully evaluated through HIL experiments. The hardware experiment is very different from the software simulation since many practical problems may appear, e.g., communication delay and measurement noises. Therefore, it is meaningful to prove the effectiveness of the developed controller in a physical system and promote its corresponding application maturity. The topology configuration of tested system is shown in FIG. 12 and the laboratory setup of HIL testbed is presented in FIG. 13, respectively. The main system parameters are provided in Table IV. It should be mentioned that in software simulation and HIL experiments, the different components parameters and the same control parameters are used to demonstrate the adaptivity of the RL control method. Basically, a DC power supply connected with a boost converter is applied to maintain the DC bus voltage at 20V and supply the power for normal loads. A Maxwell UC (BMOD0006-E160-B02) is connected at the DC bus via a DC/DC buck converter. HIL platform dSPACE MicroLabBox (DS1202) is employed to interface the microgrid and the RL-OPT controller that is implemented in the host PC. Variables are measured by the ADC I/Os and resulted switching signals sent by DAC I/Os of the DS1202 in a real-time manner. For safety consideration, the experiment is designed to charge the UC from 9V to 11V. It is noteworthy that, except for certain necessary system settings, the controller parameters are set exactly the same as they were in the simulation case studies without further tuning effort, as this can help to evaluate the scalability of controller.

TABLE IV
SYSTEM PARAMETERS OF HIL CASE STUDIES
	Parameter	Value	Parameter	Value
	Cuc	6.0	F	Lf	2	mH
	Cf	47	μF	Rload	20Ω

The experiment results are presented in FIG. 14, where the green line denotes the DC bus voltage, blue line describes the voltage of UC, and red line describes the charging current. It can be observed from FIG. 14 that the entire charging process is smooth. Expect for the normal harmonics introduced by the switching devices and surrounding electromagnetic interference, the charging process of UC barely produces any disturbance to the common bus voltage. In addition, the system mismatch between software simulation and hardware experiment has been well resolved by the RL-based method, which is a significant merit of the method.

The instant reinforcement-learning-based online optimal (RL-OPT) control method is for hybrid energy storage system (HESS) in AC/DC microgrids involving photovoltaic (PV) systems and diesel generators (DG). Due to the low system inertia, conventional unregulated charging and discharging (C&D) of energy storages in microgrids may introduce disturbances that degrade the power quality and system performance, especially in fast C&D situations. Secondary and tertiary control levels can optimize the state of charge (SOC) reference of HESS; however, they are lacking the direct controllability of regulating the transient performance. Additionally, the uncertainties in practical systems greatly limit the performance of conventional model-based controllers. In this study, the optimal control theory is applied to optimize the C&D profile and to suppress the disturbances caused by integrating HESS. Neural networks (NN) are devised to estimate the nonlinear dynamics of HESS based on the input/output measurements, and to learn the optimal control input for bidirectional-converter-interfaced HESS using the estimated system dynamics. Because the RL-OPT method is fully decentralized, which only requires the local measurements, the plug & play capability of HESS can be easily realized. Both islanded and grid-tied modes are considered.

In sum, the RL-OPT controller is developed to provide a smooth C&D control for HESS in microgrids with unknown system parameters. First, a unified design approach for optimal control of uncertain nonlinear system is formulated. Then, one NN is designed to learn the system dynamics based on the input/output data. Next, another NN is developed to learn the optimal control input for system through online RL. The effectiveness of method is fully evaluated through extensive software simulations and HIL experiments.

Although solar energy is discussed in relation to the above implementations, the system, methods and all other implementations discussed above can also be used and applied in relation to other types of generators and for other forms of energy, such as energy harvested in wind generators and water pumps.

The operation and control features can be implemented in hardware, software or a combination of hardware and software. In the case of software, the software may be embodied in storage media or as firmware. Storage media and computer readable media for containing code, or portions of code, can include any appropriate media known or used in the art, including storage media and communication media, such as but not limited to volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage and/or transmission of information such as computer readable instructions, data structures, program modules, or other data, including RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, data signals, data transmissions, or any other medium which can be used to store or transmit the desired information and which can be accessed by the computer. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will appreciate other ways and/or methods to implement the various embodiments.

While particular embodiments have been disclosed herein in detail, this has been done by way of example for purposes of illustration only, and is not intended to be limiting with respect to the scope of the appended claims, which follow. In particular, it is contemplated that various substitutions, alterations, and modifications may be made without departing from the spirit and scope of the invention as defined by the claims. Other aspects, advantages, and modifications are considered to be within the scope of the following claims. The claims presented are representative of the embodiments and features disclosed herein. Other unclaimed embodiments and features are also contemplated. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A method to manage a microgrid with a hybrid energy storage system (HESS), comprising: deriving a dynamic model of a bidirectional-power-converter (BPC)-interfaced HESS;applying a first neural network (NN) to estimate a system dynamic; andapplying a second NN to calculate an optimal control input for the HESS through online learning based on the estimated system dynamics.

2. The method of claim 1, comprising representing the system dynamics in a single layer NN.

3. The method of claim 2, by defining the tracking error as e(t)=Uc(t)−U*c with U*c being a constant SOC reference, comprising determining ė(t)=f(e(t))+g(t)u(t)+D(t)=W*eTσ1(e)+W*uTσ1(u)+W*dT1+ϵ1=W*1Tσ1(e,u)+ϵ1 where ė(t)={dot over (U)}c(t)−{dot over (U)}*c={dot over (U)}c(t)−0={dot over (U)}c(t); f(e)=(Rsc/Lf−1/CucRpc)Uc and g(t)=RscVt/Lf are unknown nonlinear system dynamics. u(t)=a is the control input. D(t)=[(Rpc+Rsc)/CucRp−RscRf/Lf]Ic is the perturbation term of the system with D(0)=0 and is bounded as ∥D(t)∥≤dmax.

4. The method of claim 2, comprising tracking error dynamics estimation as {dot over (ê)}(t)=Ŵ1Tσ1(e,u)+k1{tilde over (e)}(t) where Ŵ1=[Ŵe Ŵu Ŵd]T∈N×1 are the estimated NN identifier weights and k1 is the selected parameter to maintain NN identifier stability.

5. The method of claim 2, comprising defining {tilde over (e)}(t)=e(t)−ê(t) as an estimation error of tracking error, determining dynamics of {tilde over (e)}(t) as {dot over ({tilde over (e)})}(t)=ė(t)−{dot over (ê)}(t)=W*1Tσ1(e,u)+ϵ1−Ŵ1Tσ1(e,u)−k1{tilde over (e)}(t)=Ŵ1Tσ1(e,u)+ϵ1−k1{tilde over (e)}(t)where the NN weight estimation error is defined as {tilde over (W)}1(t)=W*1−Ŵ1(t), and {dot over ({tilde over (W)})}1(t)=−{dot over (Ŵ)}1(t).

6. The method of claim 2, comprising forcing estimated NN identifier weight Ŵ1(t) to converge to a target weight W*1 by updating Ŵ1 as {dot over (Ŵ)}1=−k2Ŵ1Tσ1(e,u)+σ1(e,u)e(t), where k2 is a positive tuning parameter of NN identifier.

7. The method of claim 1, comprising representing a cost function V*(e) by a single-layer NN on set Ω as V*(e)=W*2Tσ2(e,u)+ϵ2 where W*2T∈1×N are ideal unknown weights of a cost function estimator, σ2(e,u) is an activation function, ϵ2 is an NN reconstruction error, W*2 and ϵ2 are bounded as ∥W*2∥≤W2M and ∥ϵ2∥≤ϵ2M with W2M and ϵ2M as two constant bounds, respectively.

8. The method of claim 7, comprising applying the cost function estimator as {circumflex over (V)}(e)=Ŵ2Tσ2(e,u).

9. The method of claim 1, comprising deriving an estimated optimal control policy based on the two NNs as û=−½R−1ŴuT∇σ2 (e,u)TŴ2 where ∇σ2(e,u)=∂σ2(e,u)/∂e is a partial derivative of σ2(e,u) with respect to e.

10. The method of claim 1, comprising determining an approximated Hamiltonian of optimal control problem as: Ĥ(e,û,{circumflex over (V)})=Q(e)+¼Ŵ2T∇σ2ŴuR−1ŴuT∇σ2TŴ2+∇σ2TŴ2[Ŵ1T(t)σ1(e,u)+k1{tilde over (e)}(t)]+¼Ŵ2T∇σ2TŴ2+dmax2 where Q(e)=eTPe is a positive definite function of e with P being a symmetric positive definite matrix.

11. The method of claim 1, comprising tuning a NN weight of cost function estimator as

12. The method of claim 11, wherein J1∂e comprises ∥ė∥≤c1∥e∥≡(c2∥J1∂e∥)1/4 where c1 and c2 are constants.

13. The method of claim 11, where ∥J1∂e∥ satisfies a general bound where J1=⅕(eTe)5/2.

14. The method of claim 1, comprising analyzing both grid-tied and islanded modes of the microgrids.

15. The method of claim 1, wherein in a grid-tied mode, the main grid is considered as an infinite source which maintains bus voltage and reactive power at a point of common coupling (PCC) through a voltage source converter (VSC).

16. The method of claim 1, wherein while in islanded mode, diesel generators (DGs) are deployed to maintain the bus voltage at PCC.

17. The method of claim 1, comprising controlling a photovoltaic (PV) array under a maximum power point tracking (MPPT) mode to maximize a renewable DER utilization.

18. The method of claim 1, comprising formulating an optimal control problem of HESS using an RL method to reduce the disturbances caused by C&D of one or more energy storage devices.

19. The method of claim 1, comprising adapting different system dynamics based on input/output data without the system parameter information and without a model.

20. The method of claim 1, wherein the microgrid consists of AC and DC buses interconnected through a bidirectional DC/AC voltage source converter (VSC) and a PV array operates in MPPT mode, and the HESS with both UC and LIB are connected on a DC bus, and grid-tied and islanded operation mode switching of the microgrid is realized by operating the circuit breaker (CB).