NONLINEAR OPTIMAL CONTROL METHOD

Abstract

A nonlinear optimal control method is provided. The nonlinear optimal control method comprises performing a policy iteration algorithm using a Lyapunov barrier function made by applying a barrier function to a control Lyapunov function and Sontag's formula.

Description

TECHNICAL FIELD

The present invention relates to a nonlinear optimal control method.

BACKGROUND ART

Recently, research on reinforcement learning technology that learns optimal policies based on artificial intelligence technology is being actively conducted in the field of computer engineering. In the case of game fields such as AlphaGo, where the algorithm is widely used, there are few concerns about stability, and the application of the algorithm has been mainly focused on optimality. However, in real systems such as chemical plants or robots, stability must be guaranteed before optimality. In the case of existing studies, an attempt was made to ensure stability by introducing an additional actor network in addition to the critic network. However, most of the existing algorithms are limited to designing update rules for actor networks for single-layer neural networks and are difficult to apply to actual systems. In addition, the actual system must be controlled so as not to exceed the constraints, but existing algorithms have limitations in not breaking the constraints.

DISCLOSURE

Technical Problem

The present invention provides a nonlinear optimal control method having good performance.

The other objects of the present invention will be clearly understood by reference to the following detailed description and the accompanying drawings.

Technical Solution

A nonlinear optimal control method according to the embodiments of the present invention comprises performing a policy iteration algorithm using a Lyapunov barrier function made by applying a barrier function to a control Lyapunov function and Sontag's formula.

The policy iteration algorithm learns an optimal controller while finding a control Lyapunov function that has the same level set form as an optimal value function among control Lyapunov functions, and ensures constraints satisfaction and stability during and after the learning using the Sontag's formula.

The barrier function may reach infinity at the boundary of the inequality constraints. The constraints of the optimal value function may be included in an objective function by the barrier function.

The policy iteration algorithm may be the following exact safe policy iteration algorithm.

[Exact safe policy iteration algorithm]
Algorithm 1 Exact Safe Policy Iteration Algorithm
1: Set an admissible control as initial control policy ψ0(x), and set k ← 0.
2: (Policy evaluation) Obtain the solution of the following LE, Vk ϵ C1:
H⁡(?,Vk,ψk)=∂Vk∂?⁢(F⁡(?)+G⁡(?)⁢ψk(?))+qaug(?,ψk(?))+ψk(?)T⁢R⁢ψk(?)=0,∀?∈?with⁢Vk(0)=0.
3: (Policy improvement) Update the control policy as
ψk+1(?)={-LF⁢Vk+??⁢R-1⁢LG⁢VkTLG⁢Vk≠00LG⁢Vk=0
4: Iterate steps 2 and 3 with k ← k + 1 until ∥Vk+1 − Vk∥∞ < ϵ.
?                                        indicates text missing or illegible when filed

The exact safe policy iteration algorithm may solve Lyapunov equation in a policy evaluation part to calculate the control Lyapunov function V_k that evaluates whether constraints are violated, and costs incurred under current stabilization control input ψ_k. The exact safe policy iteration algorithm may ensure the constraints satisfaction and stability during and after the learning using the Sontag's formula in a policy improvement part.

The policy iteration algorithm may be the following approximate safe policy iteration algorithm.

[Approximate safe policy iteration algorithm]
Algorithm 2 Proposed Approximate Safe RL
(Approximate function initialization)
1: for i = 1, . . . ,     do
2:  Initialize    (   ) = N   (   ) + LB(   ).
3:  If     satisfies the CLF condition on grid points in     then break
4:  else i ← i + 1
5:  end if
6: end for
7: Sontag's formula with the CLF    (   ),    (   ), is set as the initial controller. Set k ← 0.
(Training while restricting approximate function as a CLF)
8: for j = 1, . . . ,     do
9:   Reset the extended states of the system     that is randomly sampled in IntX. Set l ← 0.
10:  for l = 0, . . . , Tf − 1 do
11:   Apply the input     =     to the system.
12:   Obtain the next states    .
13:   Store the data set (   ) to replay buffer.
14:   if The number of replay buffer data ≥ NRB then
15:    Record Wk as Wpre. The learning rate     is set as a user-specified learning rate    .
Set c ← 0.
16:    for c = 0, . . . ,     − 1 do
17:     Train the approximate function by the Adam optimizer with minibatch data and     to
minimize,
JE,k=1NMB?12⁢BEk(?)2.
Here, BEk(   ) = LV{circumflex over (V)}k(   ) + LC{circumflex over (V)}k(   )α + qaug(   ) + αTRα, and (   ) are randomly sampled
data from the replay buffer.
18:     if The updated {circumflex over (V)}k+1(x; Wk+1) does not satisfy the CLF condition on grid points in
then
19:      Wk ← Wpre
20:          ←    /10, and c ← c + 1,
21:     else
22:      break.
23:     end if
24:    end for
(Improve control policy using Sontag's formula)
25:   Update the control policy as follows:
ψk+2(?)={-LF⁢Vk+1+??⁢R-1⁢LG⁢Vk+1TLG⁢V^k≠00LG⁢V^k=0
26:   end if
27:   k ← k + 1, l ← l + 1
28:  end for
29: end for
indicates data missing or illegible when filed

The approximate safe policy iteration algorithm may learn neural network, and the neural network may satisfy the property of the control Lyapunov function.

The approximate safe policy iteration algorithm may gather the states determined by a stabilization control input ({circumflex over (ψ)}_k) and perform weight update of a value function ({circumflex over (V)}_k) approximated by a deep neural network in the direction of reducing Bellman errors in a policy evaluation part.

Constraints may be considered through an augmented objective function including the barrier function. The approximate safe policy iteration algorithm may ensure the constraints satisfaction and stability during and after the learning using the Sontag's formula in a policy improvement part.

When the weight updated value function does not satisfy the control Lyapunov function conditions, the weight update may be performed again to satisfy the function conditions.

Advantageous Effects

The nonlinear optimal control method according to the embodiments of the present invention has good performance. For example, the nonlinear optimal control method can ensure both constraints satisfaction and stability.

DESCRIPTION OF DRAWINGS

FIG. 1 shows a four-tank configuration for illustrating a nonlinear optimal control method according to an embodiment of the present invention.

FIG. 2 shows the absolute errors between the costs of the trained controller and the model prediction controller.

BEST MODE

Hereinafter, a detailed description will be given of the present invention with reference to the following embodiments. The purposes, features, and advantages of the present invention will be easily understood through the following embodiments. The present invention is not limited to such embodiments, but may be modified in other forms. The embodiments to be described below are nothing but the ones provided to bring the disclosure of the present invention to perfection and assist those skilled in the art to completely understand the present invention. Therefore, the following embodiments are not to be construed as limiting the present invention.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. It is to be understood that the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. It will be further understood that the terms “comprises” or “has,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

A nonlinear optimal control method according to the embodiments of the present invention comprises performing a policy iteration algorithm using a Lyapunov barrier function made by applying a barrier function to a control Lyapunov function and Sontag's formula.

The policy iteration algorithm learns an optimal controller while finding a control Lyapunov function that has the same level set form as an optimal value function among control Lyapunov functions, and ensures constraints satisfaction and stability during and after the learning using the Sontag's formula.

[Barrier Function]

A barrier function (BF) is generally used in optimization solvers based on interior-point methods. The barrier function is used to consider inequality constraints into the objective function, resulting in that the inequality constrained optimization problem is converted into the equality constrained optimization problem. The barrier function reaches infinity at the boundary of the inequality constraints set and the optimization solver finds the optimal solution minimizing the sum of the original objective function and the barrier function. Thus, the optimization solver can find the solution within the feasible region.

The natural extension of the barrier function to the system with control inputs is control barrier function (CBF). To clarify the control barrier function, the control barrier function is explained with mathematical descriptions. The control barrier function is defined by the extended states x.

Definition 1: Control Barrier Function

A C function BF(x): Intχ×IntU→ is a control barrier function (CBF) for the dynamic system with the set χ×U. If there exist class functions α₁, α₂, and α₃, the following inequality follows:

$\begin{array}{cc}\frac{1}{{\alpha }_1\left(h\left(\stackrel{-}{x}\right)\right)}\le \mathrm{BF}\left(\stackrel{-}{x}\right)\le \frac{1}{{\alpha }_2\left(h\left(\stackrel{-}{x}\right)\right)}& \left[\mathrm{Inequation}1\right]\end{array}$ $\begin{array}{cc}\frac{\mathrm{dBF}\left(\stackrel{-}{x}\right)}{\mathrm{dt}}\le {\alpha }_3\left(h\left(\stackrel{-}{x}\right)\right)\forall \stackrel{-}{x}\in \mathrm{Int}𝒳\times \mathrm{Int}𝒰& \left[\mathrm{Inequation}2\right]\end{array}$

The Lyapunov-like condition (Inequation 1) implies that BF(x) behaves like

$\frac{1}{\alpha \left(h\left(\stackrel{-}{x}\right)\right)}$

with some class function α:

$\underset{x\in \mathrm{Int}\left(𝒳\right)}{\inf }\frac{1}{\alpha \left(h\left(\stackrel{-}{x}\right)\right)}\ge 0,\underset{\text{?}}{\lim }\frac{1}{\alpha \left(h\left(\stackrel{-}{x}\right)\right)}=\infty .$ $\text{?}\text{indicates text missing or illegible when filed}$

This means that BF(x) satisfies the important properties of the barrier function:

$\underset{x\in \mathrm{Int}\left(𝒳\right)}{\inf }\mathrm{BF}\left(\stackrel{-}{x}\right)\ge 0,\underset{\stackrel{-}{x}\to \partial \stackrel{-}{𝒳}}{\lim }\mathrm{BF}\left(\stackrel{-}{x}\right)=\infty$

Inequation 2 guarantees the forward control invariance of Intχ with respect to the dynamics. This is the relaxation of the original condition of

$\frac{\mathrm{dBF}}{\mathrm{dt}}\le 0.$

$\frac{\mathrm{dBF}}{\mathrm{dt}}\le 0$

makes BF(x) decrease or keep constant along with the dynamics. That is, the state x stay in the interior. The related condition (Inequation 2) allows for an increase in BF(x) when the states are far away from the constrain boundary. Even under this GC relaxed condition,

$\mathrm{BF}\left(\stackrel{-}{x}\left(t\right)\right)\le \frac{1}{\sigma \left(\frac{1}{\mathrm{BF}\left(\text{?}\right)},t-t_0\right)}$ $\text{?}\text{indicates text missing or illegible when filed}$

holds for all t and x(t₀)∈Intχ when x(t) has a unique solution for all t. The lower bound of inequation 1,

$h\left(\stackrel{-}{x}\left(t\right)\right)\ge {\alpha }_1^{-1}(\sigma \left(\frac{1}{\mathrm{BF}\left(\text{?}\right)},t-t_0\right)$ $\text{?}\text{indicates text missing or illegible when filed}$

holds for all t. This implies that h(x(t))>0 holds for all x(t₀)∈Intχ.

$\mathrm{BF}\left(x\right)=-\log \left(\frac{h\left(\stackrel{-}{x}\right)}{1+h\left(\stackrel{-}{x}\right)}\right)$

can be a control barrier function candidate with appropriate control inputs.

The control input should be designed such that the allowed increasing speed of the control barrier function value decreases near the boundary and approaches zero as the states go to the boundary. This relaxed property will be made stricter to guarantee that the control barrier function value decreases at least near the boundary in the proposed algorithm.

This is because in real applications, the data are obtained only at sampling times with a finite interval. To guarantee safety, that is, the forward control invariance under this real situation, the control barrier function value along with dynamics should decrease at least near the boundary.

[Control Lyapunov Function and Sontag's Formula]

The control Lyapunov function is and extension of the Lyapunov function for stabilization. The definition is described as follows.

Definition 2: Control Lyapunov Function

V_c(x) is the control Lyapunov function when it is a positive definite, proper, and continuous differential function satisfying the following property.

$\mathrm{for}\mathrm{all}\stackrel{-}{x}\ne 0\mathrm{of}{L}_G{V}_c\left(\stackrel{-}{x}\right)=0,L_F{V}_c\left(\stackrel{-}{x}\right)<0$

L_GV_c and L_FV_c denote

$\frac{\partial V_c}{\text{?}}G$ $\text{?}\text{indicates text missing or illegible when filed}$

and

$\frac{\partial V_c}{\text{?}}F,$ $\text{?}\text{indicates text missing or illegible when filed}$

respectively. When the property holds globally and V_c(x) is radially unbounded, then V_c is the global control Lyapunov function.

The control inputs with Sontag's formula using control Lyapunov function are as follows.

${\psi }_c\left(\stackrel{-}{x}\right)=\{\begin{array}{cc}-\frac{L_F{V}_c+\sqrt{L_F{V}_c^2+{\left(L_G{V}_c{L}_G{V}_c^{\top }\right)}^2}}{L_G{V}_c{L}_G{V}_c^{\top }}{L}_G{V}_c^{\top }& L_G{V}_c\ne 0\\ 0& L_G{V}_c=0\end{array}$

Sontag's formula input provides an asymptotic stabilizing controller because of the control Lyapunov function property. Considering the converse Lyapunov theorem and Sontag's formula, the existence of a control Lyapunov function is equivalent to the existence of a smooth controller stabilizing the system asymptotically.

As a significant property of Sontag's formula, is equivalent to the optimal controller for a user-defined cost function r(x, a)=q(x)+a^TRa when the CLF has the same level set shapes as those of the optimal value function V*.

${\psi }_c\left(\stackrel{-}{x}\right)=\{\begin{array}{cc}-\frac{L_F{V}_c+\sqrt{{L_F{V}_c^2+q\left(\stackrel{-}{x}\right)L_G{V}_c{R}^{-1}{L}_G{V}_c^{\top })}^2}}{L_G{V}_c{L}_G{V}_c^{\top }}{R}^{-1}{L}_G{V}_c^{\top }& L_G{V}_c\ne 0\\ 0& L_G{V}_c=0\end{array}$

In other words, ψ(x) is equivalent to the optimal controller a*(x) when V_c=α_c(V*) with a differentiable class function α_c. This holds because V* is the solution to the HJB (Hamilton-Jacobi-Bellman) equation.

$L_F{V}^{*}\left(\stackrel{\_}{x}\right)-\frac{1}{4}{L}_G{V}^{*}{R}^{-1}{L}_G{V}^{*T}+q\left(\stackrel{\_}{x}\right)=0\frac{\partial V_c}{\partial \stackrel{\_}{x}}=\frac{\partial {\alpha }_c\left(V^{*}\right)}{\partial V^{*}}\frac{\partial V^{*}}{\partial \stackrel{\_}{x}}=\lambda \left(\stackrel{\_}{x}\right)\frac{\partial V^{*}}{\partial \stackrel{\_}{x}}\mathrm{with}\lambda \left(\stackrel{\_}{x}\right)>0\mathrm{for}\mathrm{all}\ne 0.\mathrm{For}{L}_{G}V\ne 0,-\frac{L_F{V}_c+\sqrt{L_F{V}_c^2+q\left(\stackrel{\_}{x}\right)L_G{V}_c{R}^{-1}{L}_G{V}_c^T}}{L_G{V}_c{R}^{-1}{L}_G{V}_c^T}{R}^{-1}{L}_G{V}_c^T=-\frac{L_F{V}^{*}+\sqrt{L_F{V}^{*2}+q\left(\stackrel{\_}{x}\right)L_G{V}^{*}{R}^{-1}{L}_G{V}^{*T}}}{L_G{V}^{*}{R}^{-1}{L}_G{V}^{*T}}{R}^{-1}{L}_G{V}^{*T}=-\frac{L_F{V}^{*}+\sqrt{{\left(q\left(\stackrel{\_}{x}\right)+\frac{1}{4}{L}_G{V}^{*}{R}^{-1}{L}_G{V}^{*T}\right)}^2}}{L_G{V}^{*}{R}^{-1}{L}_G{V}^{*T}}{R}^{-1}{L}_G{V}^{*T}=-\frac{1}{2}{R}^{-1}{L}_G{V}^{*T}$

The first equality is because

$\frac{\partial V_c}{\partial \stackrel{\_}{x}}=\lambda \left(\stackrel{\_}{x}\right)\frac{\partial V^{*}}{\partial \stackrel{\_}{x}}$

with a positive scalar function λ(x). The second and third equalities are due to the HJB equation. For L_GV_c=0, both and are 0.

The similarity of the level-set shapes between two scalar functions can be represented by calculating the standard deviation of the element-wise division of their gradient vectors. If we precisely know the optimal value function, this measure can be used to demonstrate the similarity degree of the trained control Lyapunov function and the optimal value function. However, determining the optimal value function is difficult, which is why reinforcement learning (RL) is used to learn the optimal control policy along with the optimal value function.

When considering the above equation, the similarity of the level set shapes can be practically checked by comparing Sontag's formula input with the optimal formula input.

The simulation results are analyzed by investigating how much similar the Sontag's formula inputs are with the optimal formula inputs

$-\frac{1}{2}{R}^{-1}{L}_G{V}^{*}.$

For simplification, the optimal formula is called an LgV-type formula.

[Lyapunov Neural Network]

The necessary conditions for the control Lyapunov function are positive definiteness and continuous differentiability. Thus, it is needed to guarantee that the approximate function has these properties for any parameter values. For this, the Lyapunov neural network (LNN) is used.

The Lyapunov neural network {circumflex over (V)}(x) is obtained by the inner product of a feedforward neural network ϕ(x) with itself, that is, {circumflex over (V)}(x)=ϕ(x)^Tϕ(x). ϕ(x) with a finite number of parameters can approximate any continuous function on a compact set with arbitrary accuracy. Owing to the inner product, the positiveness of {circumflex over (V)}(x) is guaranteed. To ensure that {circumflex over (V)}(x) has a zero value only at x=0, the null space of ϕ(x) should be trivial. To this end, each layer of ϕ(x) must have a trivial null space. This can be obtained with the specific structure of the below equation for A_L, when the output of layer L is represented as y_L=α(A_Ly_L-1) with a weight matrix A_L and an activation function α(⋅).

$A_L=\left[\begin{array}{c}{G}_{L_1}^T{G}_{L1}+ϵI_{d_{L-1}}\\ G_{L2}\end{array}\right]$

d_L is the dimension of the layer L. G_L1∈^q1×dL-1 for some integer q_L≥1, G_L2∈^{(dL−dL-1)×dL-1}, and ϵ is a positive constant. I_dL-1 denotes the identity matrix of dimension d_L-1. The parameters to train are the elements of G_L1 and G_L2 of all layers. {circumflex over (V)} is continuously differentiable.

[Safe Reinforcement Learning for Constrained Nonlinear Systems]

Safe reinforcement learning according to the embodiments of the present invention uses a modified barrier function and Sontag's formula to guarantee constraint satisfaction. The original optimal control problem is modified by introducing a Lyapunov barrier function, LB(x) into the objective function.

$\underset{a}{\min }{V}_{\mathrm{aug}}^{\alpha }\left(\stackrel{\_}{x}\right)\mathrm{subject}\mathrm{to}\stackrel{\text{?}}{\stackrel{\_}{x}}\left(t\right)=F\left(\stackrel{\_}{x}\right)+G\left(\stackrel{\_}{x}\right)a,x\left(0\right)=x,u\left(0\right)=u{V}_{\mathrm{aug}}^{\alpha }\left(\stackrel{\_}{x}\right)={\int }_t^{\infty }{q}_{\mathrm{aug}}\left(\stackrel{\_}{x}\left(\tau \right)\right)+{a\left(\tau \right)}^T\mathrm{Ra}\left(\tau \right)d\tau$ $\text{?}\text{indicates text missing or illegible when filed}$

with q_aug(x)=q(x)+μLB(x). μ is set sufficiently small so as not to disturb the optimal performance while providing enough barrier near the boundary.

Before introducing a Lyapunov barrier function, some assumptions for the optimal control problem are necessary.

Assumption 1: Existence of an Admissible Input

For any initial extended state in Intχ, there exists a continuous control policy a(x) asymptotically stabilizing the system with a(0)=0 and its cost V_aug^a(x) is finite.

This assumption implies that the optimal control problem is feasible for the domain Intχ. If there is no admissible control policy, there is no hope of obtaining a possible control policy to keep the system in a safe region.

Assumption 2: Lyapunov Barrier Function

LB(x) is a continuously differentiable function that satisfies the following properties with class functions α₁ and α₂.

$\frac{1}{{\alpha }_1\left(h\left(\stackrel{\_}{x}\right)\right)}\le \mathrm{LB}\left(\stackrel{\_}{x}\right)\le \frac{1}{{\alpha }_2\left(h\left(\stackrel{\_}{x}\right)\right)}\forall \stackrel{\_}{x}\in \stackrel{\_}{\chi }\mathrm{LB}\left(\stackrel{\_}{x}\right)=0\mathrm{if}\mathrm{and}\mathrm{only}\mathrm{if}\stackrel{\_}{x}=0$

Lyapunov barrier function must satisfy additional property, LB(x)=0 if and only if x=0, along with the general barrier function properties. Without this, the objective function would have an infinite value. Thus, Assumption 2 cannot hold without the positive definiteness of the Lyapunov barrier function. q_aug(x) is still positive definite with LB(x). The condition of the time derivative of the control barrier function will be obtained using Sontag's formula, thus, it is not needed to assume the property here.

Assumption 3 There is a positive definite and continuously differentiable function V*: Intχ→, which is the solution of the HJB equation with the augmented objective function.

$\underset{a}{\min }\left\{\frac{\partial V^{*}}{\partial \stackrel{\_}{x}}\left(F\left(\stackrel{\_}{x}\right)+G\left(\stackrel{\_}{x}\right)a\right)+q_{\mathrm{aug}}\left(\stackrel{\_}{x}\right)+a^T\mathrm{Ra}\right\}=\frac{\partial V^{*}}{\partial \stackrel{\_}{x}}\left(F\left(\stackrel{\_}{x}\right)+G\left(\stackrel{\_}{x}\right)a^{*}\right)+q_{\mathrm{aug}}\left(\stackrel{\_}{x}\right)+a^{*T}{\mathrm{Ra}}^{*}=\frac{\partial V^{*}}{\partial \stackrel{\_}{x}}F\left(\stackrel{\_}{x}\right)-\frac{1}{4}\frac{\partial V^{*}}{\partial \stackrel{\_}{x}}G\left(\stackrel{\_}{x}\right)R^{-1}G{\left(\stackrel{\_}{x}\right)}^T\frac{\partial V^{*T}}{\partial \stackrel{\_}{x}}+q_{\mathrm{aug}}\left(\stackrel{\_}{x}\right)=0,\mathrm{for}\mathrm{all}\stackrel{\_}{x}\in \mathrm{Int} \stackrel{\_}{\chi }$

Similar to the HJB equation of the original optimal control problem, the above equation has a unique solution when V*(x) is continuously differentiable. In addition, if the value function V^ah(x)=∫_t^∞q_aug(x(τ))+a_h(x(τ))^TRa_h(x(τ))dτ is continuously differentiable, it satisfies the following Lyapunov equation.

$\frac{\partial V^{a_h}}{\partial \stackrel{\_}{x}}\left(F\left(\stackrel{\_}{x}\right)+G\left(\stackrel{\_}{x}\right)a_h\left(\stackrel{\_}{x}\right)\right)+q_{\mathrm{aug}}\left(\stackrel{\_}{x}\right)+{a_h\left(\stackrel{\_}{x}\right)}^T{\mathrm{Ra}}_h\left(\stackrel{\_}{x}\right)=0\mathrm{with}{V}^{a_h}\left(0\right)=0$

If the system is stable and q_aug(x)+a(x)^TRa(x) is zero-state observable, the solutions of HJB and Lyapunov equation are positive definite. The sufficient condition for q_aug+a^TRa to be zero-state observable is the zero-state observability of the original objective r(x, a)=q(x)+a^TRa. The general objective function for the stabilization of the tracking problem is zero-state observable because no solution can stay in S={x∥r(x, 0)=0} other than x≡0. For the augmented objective function r(x,a)=+LB(x) with the original objective function, only x≡0 can stay in S_aug={x|r(x, 0)+LB(x)=0} owing to the positive definiteness of LB(x).

With Assumptions 1-3, there exists a unique optimal control policy that guarantees safety and stabilization. Under the assumptions, the exact policy iteration algorithm with Lyapunov barrier function in Algorithm 1 guarantees the convergences to the optimal value function and optimal control policy. This can be proven easily, as in the original policy iteration proof with q_aug instead of q.

Algorithm 1 Exact Safe Policy Iteration Algorithm
1: Set an admissible control as initial control policy ψ0(x), and set k ← 0.
2: (Policy evaluation) Obtain the solution of the following LE, Vk ϵ C1:
H⁡(?,Vk,ψk)=∂Vk∂?⁢(F⁡(?)+G⁡(?)⁢ψk(?))+qaug(?,ψk(?))+ψk(?TR⁢ψk(?)=0,∀?∈?with⁢Vk(0)=0.
3: (Policy improvement) Update the control policy as
ψk+1(?)={-LF⁢Vk+??⁢R-1⁢LG⁢VkTLG⁢Vk≠00LG⁢Vk=0
4: Iterate steps 2 and 3 with k ← k + 1 until ∥Vk+1 − Vk∥∞ < ϵ.
?                                        indicates text missing or illegible when filed

Solving Lyapunov equation for nonlinear systems is difficult, thus, approximate policy iteration is used with approximate functions such as deep neural networks and up-to-date gradient-based optimization solvers such as the Adam optimizer. The approximate function {circumflex over (V)}_k is not the exact solution of Lyapunov equation and causes deviations, the Bellman error: BE(x; Ŵ_k).

$\mathrm{BE}\left(\stackrel{\_}{x},\stackrel{\text{?}}{W_k}\right)=H\left(\stackrel{\_}{x},\stackrel{\text{?}}{V_k},{\pi }_k\right)=\frac{\partial {\hat{V}}_k}{\partial \stackrel{\_}{x}}\left(F\left(\stackrel{\_}{x}\right)+G\left(\stackrel{\_}{x}\right){\pi }_k\left(\stackrel{\_}{x}\right)\right)+q_{\mathrm{aug}}\left(\stackrel{\_}{x},{\pi }_k\left(\stackrel{\_}{x}\right)\right)+{{\pi }_k\left(\stackrel{\_}{x}\right)}^{T}R{\pi }_k\left(\stackrel{\_}{x}\right)$ $\text{?}\text{indicates text missing or illegible when filed}$

Ŵ denotes the parameters of the approximate function. Because of the approximation errors, stability is not guaranteed during the training if the performance-oriented control formula is used. This can be addressed with the approximate function restricted to the control Lyapunov function and using the stability-oriented formula, Sontag's formula. Safety can be guaranteed by introducing the Lyapunov barrier function. The approximate safe reinforced learning with Lyapunov barrier function, control Lyapunov function, and Sontag's formula is proposed in Algorithm 2. The approximate function {circumflex over (V)}_k should have the Lyapunov barrier function property for constraint satisfaction. In addition, the optimal value function also has large values near the boundary when considering the augmented objective function. Thus, the sum of the Lyapunov neural network and Lyapunov barrier function is used as the approximate function as follows.

${\hat{V}}_k\left(\stackrel{\_}{x}\right)=N{N}_k\left(\stackrel{\_}{x}\right)+\mathrm{LB}\left(\stackrel{\_}{x}\right)$

The form of {circumflex over (V)}_k is the key factor along with control Lyapunov function condition and Sontag's formula when guaranteeing the forward invariance and the practically asymptotic stability of the system.

Algorithm 2 Proposed Approximate Safe RL
(Approximate function initialization)
1: for i = 1, . . . ,     do
2:  Initialize    (   ) = N   (   ) + LB(   ).
3:  If     satisfies the CLF condition on grid points in     then break
4:  else i ← i + 1
5:  end if
6: end for
7: Sontag's formula with the CLF    (   ),    (   ), is set as the initial controller. Set k ← 0.
(Training while restricting approximate function as a CLF)
8: for j = 1, . . . ,     do
9:   Reset the extended states of the system     that is randomly sampled in    . Set l ← 0.
10:  for l = 0, . . . , Tf − 1 do
11:   Apply the input     =     to the system.
12:   Obtain the next states    .
13:   Store the data set (   ) to replay buffer.
14:   if The number of replay buffer data ≥ NRB then
15:    Record Wk as Wpre. The learning rate     is set as a user-specified learning rate    .
Set c ← 0.
16:    for c = 0, . . . ,     − 1 do
17:     Train the approximate function by the Adam optimizer with minibatch data and
to minimize,
JE,k=1NMB?12⁢BEk(?)2.
Here, BEk(   ) = LV{circumflex over (V)}k(   ) + LC{circumflex over (V)}k(   )α + qaug(   ) + αTRα, and (   ) are randomly sampled
data from the replay buffer.
18:     if The updated {circumflex over (V)}k+1(x; Wk+1) does not satisfy the CLF condition on grid points in
then
19:      Wk ← Wpre
20:          ←    /10, and c ← c + 1,
21:     else
22:      break.
23:     end if
24:    end for
(Improve control policy using Sontag's formula)
25:   Update the control policy as follows:
ψk+2(?)={-LF⁢Vk+1+??⁢R-1⁢LG⁢V^k+1TLG⁢V^k+1≠00LG⁢V^k+1=0
26:   end if
27:   k ← k + 1, l ← l + 1
28:  end for
29: end for
indicates data missing or illegible when filed

N_MB and N_RB denote the sizes of the minibatch and replay buffers, respectively. The past data in the replay buffer are removed when the number of stored data exceeds N_RB. N_e is the total number of episodes with different initial states used for training. T_f is the duration of a single episode. The computational load for checking the control Lyapunov function condition on the grid points can increase as the dimension of the system increases, however, it can be addressed with multiple processors because the condition can be checked in parallel.

[Practically Asymptotic Stability]

The definitions for practically asymptotic stability are introduced by adapting it to the system of the present invention. To this end, a boundary layer Δ_δ1={x∈Intχ|x∈B_δ1(p), ∀_p∈θχ} is defined with any sufficiently small δ₁>0. The set D_m=Intχ\Δ_δx is compact, and δ_m can be set as the radius of the largest ball B_δm in D_m.

Definition 3: Asymptotic Stability with Respect to a Ball

Let δ be a positive number less than δ_m. The system is asymptotically stable with respect to B_δ on a domain D_m if there exists a class function β.

$\stackrel{\_}{x}\left(t\right)\le \delta +\beta \left({\stackrel{\_}{x}}_0,t\right),\forall {\stackrel{\_}{x}}_0\in D_m$

Definition 4 Practical Asymptotic Stability

Let P∈^np be a set of parameters. The system is said to be a practical asymptotic stability on D_m if given δ>0 and for any x₀∈D_m, there exists a P such that the system is asymptotically stable with respect to B_δ with a parameterized controller a=a(x; P).

On D_m excluding an arbitrary thin boundary layer from Intχ, the practical asymptotic stability of the system is proved under for all ψ_k in k Theorem 1. In other words, during training and at the end of training, the practical asymptotic stability is guaranteed by the algorithm of the present invention.

Suppose that with any δ<δ_m and any δ₁>0, there exists a positive definite and continuously differentiable function that satisfies the control Lyapunov function condition in the domain D_m. Then, there exists an N(δ,δ₁) such that if {circumflex over (V)}_k satisfies the control Lyapunov function condition on N(δ, δ₁) grid points on D_m\B_δ, then {circumflex over (V)}_k is a control Lyapunov function on the domain D_m.

As the constrained region χ is assumed to be compact, Intχ is precompact. The precompact set Intχ is totally bounded. Thus, for an arbitrary small δ₁, the compact set D_m=Intχ\Δ_δ1 can be set by excluding the arbitrary thin boundary layer from the Intχ. Then, there exists an N(δ,δ₁) such that if {circumflex over (V)}_k satisfies the control Lyapunov function condition on N(δ,δ₁) grid points, then {circumflex over (V)}_k is a control Lyapunov function on the domain D_m.

Theorem 1: Given a constrained set χ defined using a continuously differentiable function, the system is practically asymptotically stable on D_m=Intχ\Δ_δ1 under the controller ψ_k for all k and for an arbitrary small δ₁>0. With the largest, ρ_k, Ω_ρk={x|{circumflex over (V)}_k(x)≤ρ_k}⊂D_m is the estimate of ROA (region of attraction). Furthermore, as δ₁→0, Ω_{92 k}→Intχ.

As proven above, {circumflex over (V)}_k is a control Lyapunov function on D_m\B_δ. Thus, L_F{circumflex over (V)}_k+L_G{circumflex over (V)}_k{circumflex over (ψ)}_k+1<0 always holds for all x on D_m\B_δ a with any given positive δ<δ_m and an arbitrary small δ₁. Accordingly, Ω_ρk is the estimate of ROA.

As δ₁ goes to 0, the values of {circumflex over (V)}_k at ∂D_m goes to ∞. Accordingly, the largest estimate of ROA Ω_ρk becomes close to Intχ with ρ_k→∞ and the forward invariance is guaranteed on Ωρ_k.

As described above, the exact safe policy iteration algorithm for solving the Lyapunov equation, which learns the optimal controller by learning the optimal value function of the optimal control problem, is as follows.

[Exact safe policy iteration algorithm]
Algorithm 1 Exact Safe Policy Iteration Algorithm
1: Set an admissible control as initial control policy ψ0(x), and set k ← 0.
2: (Policy evaluation) Obtain the solution of the following LE, Vk ϵ C1:
H⁡(?,Vk,ψk)=∂Vk∂?⁢(F⁡(?)+G⁡(?)⁢ψk(?))+qaug(?,ψk(?))+ψk(?)T⁢R⁢ψk(?)=0,∀?∈?with⁢Vk(0)=0.
3: (Policy improvement) Update the control policy as
ψk+1(?)={-LF⁢Vk+??⁢R-1⁢LG⁢VkTLG⁢Vk≠00LG⁢Vk=0
4: Iterate steps 2 and 3 with k ← k + 1 until ∥Vk+1 − Vk∥∞ < ϵ.
?                                        indicates text missing or illegible when filed

The exact safe policy iteration algorithm consists of the following two main elements.

1) The exact safe policy iteration algorithm solves Lyapunov equation in a policy evaluation part to calculate the control Lyapunov function V_k that evaluates whether constraints are violated, and costs incurred under current stabilization control input ψ_k. At this time, the constraints are considered through an augmented objective function q_aug.

2) The exact safe policy iteration algorithm ensures the constraints satisfaction and stability during and after the learning using the Sontag's formula in a policy improvement part without introducing additional actor networks.

Since V_k(x), the solution to the Lyapunov equation, has a fairly large value at x near the boundary, the value function is used by applying the approximation function that can simulate these characteristics to the control Lyapunov function and the controller using Sontag's formula guarantees the constraints satisfaction and stability. On the other hand, in order to stabilize the system and satisfy the constraints by applying the previously used LgV-type optimal formula, additional conditions are needed while the value function satisfies the control Lyapunov function conditions. These facts confirms that using the Sontag's formula is superior in terms of ensuring the constraints satisfaction and stability and the use of a barrier function is essential.

Because it is very difficult to find a solution to the Lyapunov equation, an approximate policy iteration algorithm that trains a neural network is used. Here, the neural network is used that satisfies the control Lyapunov function property, which is the most important in control, and by adding a barrier function to this neural network, it prevents cases from exceeding the boundary (i.e. breaking constraints).

[Approximate safe policy iteration algorithm]
Algorithm 2 Proposed Approximate Safe RL
(Approximate function initialization)
1: for i = 1, . . . ,     do
2:  Initialize    (   ) = NN0(   ) + LB(   ).
3:  if     satisfies the CLF condition on grid points in     then break
4:  else i ← i + 1
5:  end if
6: end for
7: Sontag's formula with the CLF    (   ),    (   ), is set as the initial controller. Set k ← 0.
(Training while restricting approximate function as a CLF)
8: for j = 1, . . . ,     do
9:   Reset the extended states of the system     that is randomly sampled in    . Set l ← 0.
10:  for l = 0, . . . , Tf − 1 do
11:   Apply the input     =     to the system.
12:   Obtain the next states    .
13:   Store the data set (   ) to replay buffer.
14:   if The number of replay buffer data ≥ NRB then
15:    Record Wk as Wpre. The learning rate     is set as a user-specified learning rate    .
Set c ← 0.
16:    for c = 0, . . . ,     − 1 do
17:     Train the approximate function by the Adam optimizer with minibatch data and
to minimize,
JE,k=1NMB?12⁢BEk(?)2.
Here, BEk(   ) = LV{circumflex over (V)}k(   ) + LC{circumflex over (V)}k(   )α + qaug(   ) + αTRα, and (   ) are randomly
sampled data from the replay buffer.
18:     if The updated {circumflex over (V)}k+1(x; Wk+1) does not satisfy the CLF condition on grid points in
then
19:      Wk ← Wpre
20:          ←    /10, and c ← c + 1,
21:     else
22:      break.
23:     end if
24:    end for
(Improve control policy using Sontag's formula)
25:   Update the control policy as follows:
ψk+2(?)={-LF⁢Vk+1+??⁢R-1⁢LG⁢Vk+1TLG⁢V^k+1≠00LG⁢V^k+2=0
26:   end if
27:   k ← k + 1, l ← l + 1
28:  end for
29: end for
indicates data missing or illegible when filed

The above algorithm finally proposed consists of the following three main elements.

1) The approximate safe policy iteration algorithm gathers the states determined by a stabilization control input {circumflex over (ψ)}_k and performs weight update of a value function {circumflex over (V)}_k approximated by a deep neural network in the direction of reducing Bellman errors in a policy evaluation part. At this time, if the updated value function does not satisfy the control Lyapunov function conditions, the weight update is performed again to satisfy the function conditions. The constraints are considered through an augmented objective function q_aug including a barrier function.

2) The approximate safe policy iteration algorithm ensures the constraints satisfaction and stability during and after the learning using the Sontag's formula in a policy improvement part without introducing additional actor networks.

3) When using a deep artificial neural network, a function that has the same level set form as the optimal value function is learned, and if the function used in the Sontag's formula has the same level set form as the optimal value function, it is the same as optimal control, so the optimal controller is approximated.

FIG. 1 shows a four-tank configuration for illustrating a nonlinear optimal control method according to an embodiment of the present invention.

Referring to FIG. 1, x₁ denotes the level in each tank i, u₁ and u₂ represent the valve flow rate, and γ₁ and γ₂ represent the valve parameters. The bounds for each tank liquid level and the bounds for u₁ and u₂ corresponding to the manipulated variables are as follows.

Variable	Lower Bound	Upper bound
x1	1	28
x2	1	28
x3	1	28
x4	1	28
u1	0	60
u2	0	60

Considering the above constraints, and since then stabilization problem is to stabilize to a steady state point (subscript ss), instead of a model equation for x_i, a model equation for the deviation (subscript dev) from the setpoint must be used.

In addition, although the model equation is already in a control-affine form, in order to consider the constraints for u through the barrier function, the model equation can be finally expressed as follows.

$\frac{{\mathrm{dx}}_{1,\mathrm{dev}}}{\mathrm{dt}}=-\frac{A_{\mathrm{out},1}}{A_1}\sqrt{2{\mathrm{gx}}_1}+\frac{A_{\mathrm{out},3}}{A_1}\sqrt{2{\mathrm{gx}}_3}+\frac{{\gamma }_1}{A_1}{u}_1\frac{{\mathrm{dx}}_{2,\mathrm{dev}}}{\mathrm{dt}}=-\frac{A_{\mathrm{out},2}}{A_2}\sqrt{2{\mathrm{gx}}_2}+\frac{A_{\mathrm{out},4}}{A_2}\sqrt{2{\mathrm{gx}}_4}+\frac{{\gamma }_2}{A_2}{u}_2\frac{{\mathrm{dx}}_{3,\mathrm{dev}}}{\mathrm{dt}}=-\frac{A_{\mathrm{out},3}}{A_3}\sqrt{2{\mathrm{gx}}_3}+\frac{1-{\gamma }_2}{A_3}{u}_2\frac{{\mathrm{dx}}_{4,\mathrm{dev}}}{\mathrm{dt}}=-\frac{A_{\mathrm{out},4}}{A_4}\sqrt{2{\mathrm{gx}}_4}+\frac{1-{\gamma }_1}{A_4}{u}_1\frac{{\mathrm{du}}_{1,\mathrm{dev}}}{\mathrm{dt}}=a_1\frac{{\mathrm{du}}_{2,\mathrm{dev}}}{\mathrm{dt}}=a_2.$

This can be expressed simply using the F and G notations as follows.

$\frac{d{\stackrel{\_}{x}}_{\mathrm{dev}}}{\mathrm{dt}}=F_{\mathrm{dev}}\left({\stackrel{\_}{x}}_{\mathrm{dev}}\right)+F_{1,\mathrm{dev}}\left({\stackrel{\_}{x}}_{\mathrm{dev}}\right)a_1+G_{2,\mathrm{dev}}\left({\stackrel{\_}{x}}_{\mathrm{dev}}\right)a_2$ $F_{\mathrm{dev}}\left({\stackrel{\_}{x}}_{\mathrm{dev}}\right)=\left[\begin{array}{c}-\frac{A_{\mathrm{out},1}}{A_1}\sqrt{2{\mathrm{gx}}_1}+\frac{A_{\mathrm{out},3}}{A_1}\sqrt{2{\mathrm{gx}}_3}+\frac{{\gamma }_1}{A_1}{u}_1\\ -\frac{A_{\mathrm{out},2}}{A_2}\sqrt{2{\mathrm{gx}}_2}+\frac{A_{\mathrm{out},4}}{A_2}\sqrt{2{\mathrm{gx}}_4}+\frac{{\gamma }_2}{A_2}{u}_2\\ -\frac{A_{\mathrm{out},3}}{A_3}\sqrt{2{\mathrm{gx}}_3}+\frac{1-{\gamma }_2}{A_3}{u}_2\\ -\frac{A_{\mathrm{out},4}}{A_4}\sqrt{2{\mathrm{gx}}_4}+\frac{1-{\gamma }_1}{A_4}{u}_1\\ 0\\ 0\end{array}\right],G_{1,\mathrm{dev}}\left({\stackrel{\_}{x}}_{\mathrm{dev}}\right)=\left[\begin{array}{c}0\\ 0\\ 0\\ 0\\ 1\\ 0\end{array}\right],G_{2,\mathrm{dev}}\left({\stackrel{\_}{x}}_{\mathrm{dev}}\right)=\left[\begin{array}{c}0\\ 0\\ 0\\ 0\\ 0\\ 1\end{array}\right]$

In neural network learning, if the size difference between variables is large, learning does not work well, so the optimal value function V*(x_dev,n) with x_dev,n normalized (divided by upper bound-lower bound) as an argument is learned, at this time, since F and G used in the algorithm must also express dynamics for x_dev,n, the below equations are used.

$F=F_{\mathrm{dev}}\frac{1}{\left({\stackrel{\_}{x}}_{\mathrm{ub}}-{\stackrel{\_}{x}}_{\mathrm{lb}}\right)}G=\left[G_{1,\mathrm{dev}}⌀\frac{1}{\left({\stackrel{\_}{x}}_{\mathrm{ub}}-{\stackrel{\_}{x}}_{\mathrm{lb}}\right)},G_{2,\mathrm{dev}}⌀\frac{1}{\left({\stackrel{\_}{x}}_{\mathrm{ub}}-{\stackrel{\_}{x}}_{\mathrm{lb}}\right)}\right]$

In the above equations, represents elementary division.

The approximation function {circumflex over (V)}_k(x_dev,n) is constructed by adding the barrier function BF(x_dev,n) to the control Lyapunov function, and in order to be used with the Sontag's formula to stabilize the system to a steady-state point, the barrier function BF(x_dev,n) must also have positive definite properties.

In other words, the function value must be 0 only in x_dev,n=0 and the rest must have a value greater than 0. For this, the Lyapunov barrier function LB can be constructed as follows.

$\begin{array}{cc}{h}_1=x_{1,\mathrm{dev},n}+\frac{x_{1,\mathrm{ss}}-x_{1,\mathrm{lb}}}{x_{1,\mathrm{ub}}-x_{1,\mathrm{lb}}}& h_7=x_{4,\mathrm{dev},n}+\frac{x_{4,\mathrm{ss}}-x_{4,\mathrm{lb}}}{x_{4,\mathrm{ub}}-x_{4,\mathrm{lb}}}\\ h_2=-x_{1,\mathrm{dev},n}+\frac{x_{1,\mathrm{ub}}-x_{1,\mathrm{ss}}}{x_{1,\mathrm{ub}}-x_{1,\mathrm{lb}}}& h_8=-x_{4,\mathrm{dev},n}+\frac{x_{4,\mathrm{ub}}-x_{4,\mathrm{ss}}}{x_{4,\mathrm{ub}}-x_{4,\mathrm{lb}}}\\ h_3=x_{2,\mathrm{dev},n}+\frac{x_{2,\mathrm{ss}}-x_{2,\mathrm{lb}}}{x_{2,\mathrm{ub}}-x_{2,\mathrm{lb}}}& h_9=u_{1,\mathrm{dev},n}+\frac{u_{1,\mathrm{ss}}-u_{1,\mathrm{lb}}}{u_{1,\mathrm{ub}}-u_{1,\mathrm{lb}}}\\ h_4=-x_{2,\mathrm{dev},n}+\frac{x_{2,\mathrm{ub}}-x_{2,\mathrm{ss}}}{x_{2,\mathrm{ub}}-x_{2,\mathrm{lb}}}& h_{10}=-u_{1,\mathrm{dev},n}+\frac{u_{1,\mathrm{ub}}-u_{1,\mathrm{ss}}}{u_{1,\mathrm{ub}}-u_{1,\mathrm{lb}}}\\ h_5=x_{3,\mathrm{dev},n}+\frac{x_{3,\mathrm{ss}}-x_{3,\mathrm{lb}}}{x_{3,\mathrm{ub}}-x_{3,\mathrm{lb}}}& h_{11}=u_{2,\mathrm{dev},n}+\frac{u_{2,\mathrm{ss}}-u_{2,\mathrm{lb}}}{u_{2,\mathrm{ub}}-u_{2,\mathrm{lb}}}\\ h_6=-x_{3,\mathrm{dev},n}+\frac{x_{3,\mathrm{ub}}-x_{3,\mathrm{ss}}}{x_{3,\mathrm{ub}}-x_{3,\mathrm{lb}}}& h_{12}=-u_{2,\mathrm{dev},n}+\frac{u_{2,\mathrm{ub}}-u_{2,\mathrm{ss}}}{u_{2,\mathrm{ub}}-u_{2,\mathrm{lb}}}\end{array}$ ${\mathrm{LB}}_1\left(x_{1,\mathrm{dev},n}\right)=\left(1-s_1\right)\log \left(\frac{h_1}{h_1+1}\right)+s_1\log \left(\frac{h_2}{h_2+1}\right){\mathrm{LB}}_2\left(x_{2,\mathrm{dev},n}\right)=\left(1-s_2\right)\log \left(\frac{h_3}{h_3+1}\right)+s_2\log \left(\frac{h_4}{h_4+1}\right){\mathrm{LB}}_3\left(x_{3,\mathrm{dev},n}\right)=\left(1-s_3\right)\log \left(\frac{h_5}{h_5+1}\right)+s_3\log \left(\frac{h_6}{h_6+1}\right){\mathrm{LB}}_4\left(x_{4,\mathrm{dev},n}\right)=\left(1-s_4\right)\log \left(\frac{h_7}{h_7+1}\right)+s_4\log \left(\frac{h_8}{h_8+1}\right){\mathrm{LB}}_5\left(u_{1,\mathrm{dev},n}\right)=\left(1-s_5\right)\log \left(\frac{h_9}{h_9+1}\right)+s_5\log \left(\frac{h_{10}}{h_{10}+1}\right){\mathrm{LB}}_6\left(u_{2,\mathrm{dev},n}\right)=\left(1-s_6\right)\log \left(\frac{h_{11}}{h_{11}+1}\right)+s_6\log \left(\frac{h_{12}}{h_{12}+1}\right)$ $\mathrm{LB}\left({\stackrel{\_}{x}}_{\mathrm{dev},n}\right)=-\mu \left[{\mathrm{LB}}_1\left(x_{1,\mathrm{dev},n}\right)+{\mathrm{LB}}_2\left(x_{2,\mathrm{dev},n}\right)+{\mathrm{LB}}_3\left(x_{3,\mathrm{dev},n}\right)+{\mathrm{LB}}_4\left(x_{4,\mathrm{dev},n}\right)+{\mathrm{LB}}_5\left(u_{1,\mathrm{dev},n}\right)+{\mathrm{LB}}_6\left(u_{2,\mathrm{dev},n}\right)-{\mathrm{LB}}_1\left(x_{1,\mathrm{dev},n,\mathrm{ss}}\right)-{\mathrm{LB}}_2\left(x_{2,\mathrm{dev},n,\mathrm{ss}}\right)-{\mathrm{LB}}_3\left(x_{3,\mathrm{dev},n,\mathrm{ss}}\right)-{\mathrm{LB}}_4\left(x_{4,\mathrm{dev},n,\mathrm{ss}}\right)-{\mathrm{LB}}_5\left(u_{1,\mathrm{dev},n,\mathrm{ss}}\right)-{\mathrm{LB}}_6\left(u_{2,\mathrm{dev},n,\mathrm{ss}}\right)\right]$

A neural network to which the LB is added is learned, and the LB is also added to the objective function. Finally, several tuning parameters in the algorithm were set as follows.

Ne  100
Tf  150
NRB 450
NMB 450
lr0 0.01

FIG. 2 shows the absolute errors between the costs of the trained controller and the model prediction controller. 1000 episodes were set to start from randomly determined initial conditions within +−50% of the range around the setpoint. Of these, a total of 100 episodes were used for learning, and the performance was tested through the remaining episodes. The cost of the optimal controller was calculated using a model predictive controller with a sufficiently long prediction horizon, and the differences from the corresponding value are shown in FIG. 2.

Referring to FIG. 2, it can be confirmed that a close to optimal controller is learned through the first 100 learning episodes. Additionally, it can be confirmed that there is no episode with infinite cost values. In other words, the algorithm according to the nonlinear optimal control method of the present invention can learn an optimal controller while always satisfying the constraints.

As described above, the present invention provides an important algorithm that enables the application of artificial intelligence technology, which has been developed mainly in computer engineering, to actual systems that require stability. The algorithm utilizes the correlation between the stabilization controller and the optimal controller to ensure constraints satisfaction and stability while learning the optimal controller. The constraints satisfaction is a property that is ensured by utilizing the Sontag equation using a controlled Lyapunov function with a barrier function added. Optimality utilized the fact that the Sontag's formula and the optimal controller are exactly the same when the corresponding control Lyapunov function has the same level set form as the optimal value function. By combining this fact with a policy iteration algorithm that finds the optimal value function, an algorithm was developed to learn the optimal controller while ensuring constraints satisfaction and stability. In order to apply the above algorithm to a real system, when using the essential non-linear deep artificial neural network as an approximation function, and using weight update rules in the direction of reducing standard Bellman error and a critical-network even under the gradient descent algorithm that enables fast learning using accumulated data, it is and standard it is possible to ensure constraints satisfaction and stability. The present invention is a technology needed to expand and apply an artificial intelligence-based optimal control learning algorithm to an actual system.

Although the embodiments of the present invention have been disclosed for illustrative purposes, those skilled in the art will appreciate that the present invention may be embodied in other specific ways without changing the technical spirit or essential features thereof. Therefore, the embodiments disclosed in the present invention are not restrictive but are illustrative. The scope of the present invention is given by the claims, rather than the specification, and also contains all modifications within the meaning and range equivalent to the claims.

INDUSTRIAL APPLICABILITY

The nonlinear optimal control method according to the embodiments of the present invention has good performance. For example, the nonlinear optimal control method can ensure both constraints satisfaction and stability.

Claims

1. A nonlinear optimal control method comprising: performing a policy iteration algorithm using a Lyapunov barrier function made by applying a barrier function to a control Lyapunov function and Sontag's formula.

2. The nonlinear optimal control method of claim 1, wherein the policy iteration algorithm learns an optimal controller while finding a control Lyapunov function that has the same level set form as an optimal value function among control Lyapunov functions, and ensures constraints satisfaction and stability during and after the learning using the Sontag's formula.

3. The nonlinear optimal control method of claim 2, wherein the barrier function reaches infinity at the boundary of the inequality constraints.

4. The nonlinear optimal control method of claim 2, wherein the constraints of the optimal value function are included in an objective function by the barrier function.

5. The nonlinear optimal control method of claim 1, wherein the policy iteration algorithm is the following exact safe policy iteration algorithm.

6. The nonlinear optimal control method of claim 5, wherein the exact safe policy iteration algorithm solves Lyapunov equation in a policy evaluation part to calculate the control Lyapunov function Vk that evaluates whether constraints are violated, and costs incurred under current stabilization control input ψk, and wherein the exact safe policy iteration algorithm ensures the constraints satisfaction and stability during and after the learning using the Sontag's formula in a policy improvement part.

7. The nonlinear optimal control method of claim 1, wherein the policy iteration algorithm is the following approximate safe policy iteration algorithm.

8. The nonlinear optimal control method of claim 7, wherein the approximate safe policy iteration algorithm learns neural network, and the neural network satisfies the property of the control Lyapunov function.

9. The nonlinear optimal control method of claim 7, wherein the approximate safe policy iteration algorithm gathers the states determined by a stabilization control input ({circumflex over (ψ)}k) and performs weight update of a value function ({circumflex over (V)}k) approximated by a deep neural network in the direction of reducing Bellman errors in a policy evaluation part, constraints are considered through an augmented objective function including the barrier function, and wherein the approximate safe policy iteration algorithm ensures the constraints satisfaction and stability during and after the learning using the Sontag's formula in a policy improvement part.

10. The nonlinear optimal control method of claim 9, wherein when the weight updated value function does not satisfy the control Lyapunov function conditions, the weight update is performed again to satisfy the function conditions.