Patent Application
20160040603
The invention relates to a method for the computerized control and/or regulation of a technical system and also a corresponding computer program product.
Complex technical systems, for example, gas turbines or wind turbines, are often regulated based on computerized methods, which establish an action selection policy based on training data and corresponding optimality criteria. This action selection policy specifies which action is to be carried out on the technical system in a corresponding state of the system. In this way, for example, operation of the technical system having a high efficiency can be achieved. For gas turbines, the combustion chamber dynamics or the emissions can furthermore optionally be reduced. In the case of wind turbines, for example, the alignment of the gondola in relation to the wind can also be optimized.
To determine corresponding action selection policies for technical systems, complex regression methods, for example, neuronal networks, are usually used. However, these have the disadvantage that they generate complex action selection policies, which can no longer be interpreted or understood by human experts. Accordingly, complex action selection policies are not used in the operation of a technical system because of a lack of comprehensibility. Methods are known from the prior art, using which action selection policies having lower complexity are generated, for example, in that the technical system is represented by states having smaller dimension or simpler regression methods are used.
However, these approaches frequently result in an action selection policy which is not optimum for the control or regulation of the technical system.
The object of the invention is therefore to provide a method for the computerized control and/or regulation of a technical system, which uses an action selection policy having lower complexity, which is well suited for the technical system.
This object is achieved by the independent patent claims. Refinements of the invention are defined in the dependent claims.
The method according to the invention will be explained hereafter based on steps a) to c). The identification of these steps is only used for better referencing of the features contained therein and does not establish a sequence of the execution. In particular, specific steps can also be carried out in parallel or can be interwoven.
The method according to the invention is used for the computerized control or regulation of a technical system. According to step a), the dynamic behavior of the technical system is characterized for multiple points in time in each case by a state of the technical system and an action executed on the technical system, wherein a respective action at a respective point in time results in a new state of the technical system at the next point in time. The concept of the state or the action is to be understood broadly in this case. A state can comprise in particular a state vector having one or more state variables. A state at the respective (present) point in time can optionally comprise, in addition to a state vector for the present point in time, one or more state vectors for one or more preceding points in time, whereby the history of the state over a restricted time horizon is taken into consideration. An action can also represent a vector of multiple action variables.
In a step b) of the method according to the invention, (multiple) action selection policies are provided and/or generated, wherein a respective action selection policy specifies an action to be executed at a corresponding point in time on the technical system in dependence on at least the state of the technical system at the corresponding point in time and wherein each action selection policy is associated with a complexity measure, which describes a complexity of the respective action selection policy, which is less than or equal to a predetermined complexity threshold. The complexity measure can be defined in this case in various ways, wherein examples of such complexity measures are provided hereafter.
In a step c) of the method according to the invention, the action selection policy having the highest evaluation measure of the provided and/or generated action selection policies is ascertained from the provided and/or generated action selection policies by means of the calculation of evaluation measures, which each describe the suitability of an action selection policy for the regulation and/or control of the technical system. A higher evaluation measure describes in this case a better suitability of the action selection policy for the regulation and/or control of the technical system. In specific embodiments, steps b) and c) can be carried out in parallel or can be interwoven. For example, firstly specific action selection policies can be generated and subsequently a part of the action selection policies having poor evaluation measures can be discarded. Subsequently, new action selection policies are in turn generated and in the same manner corresponding action selection policies are again discarded. In particular in the case of the use of the genetic programming or particle swarm optimization described hereafter, steps b) and c) are interwoven.
The evaluation measure, which is calculated in the scope of step c), of a respective action selection policy can be dependent according to the invention on one or more of the following three variables:
The above-described reward measure is established in dependence on predetermined optimality criteria of the operation of the technical system, wherein a higher reward measure establishes a better control or regulation of the technical system in consideration of the optimality criteria. The determination of a quality measure based on an action selection policy evaluation method is known per se from the prior art. In particular, various types of action selection policy evaluation methods are known. In a preferred embodiment, a “fitted policy evaluation method” is used (see document [1]).
After determination of the action selection policy in step c), the technical system is finally regulated and/or controlled using this action selection policy in step d).
The method according to the invention enables the regulation or control of a technical system using an action selection policy having lesser complexity, which furthermore ensures the most optimum possible operation of the technical system by establishing a suitable evaluation measure. By reducing the complexity of the action selection policy, it is more easily comprehensible by a human, so that the computerized control or regulation of the technical system using this action selection policy is more accepted.
In a particularly preferred embodiment, the provided or generated action selection policies are each represented by a functional relationship, which supplies the action to be executed at the respective point in time based on at least the state of the technical system at the respective point in time. The concept of the functional relationship is to be understood broadly in this case and can comprise any arbitrary type of function or function composition or mathematical expression. In particular, the functional relationship can comprise settable parameters, wherein an action selection policy is defined by establishing corresponding parameter values.
The complexity measure used in the method according to the invention can be defined in various ways. Various methods for determining complexity measures are known in this case from the prior art (see, for example, document [2]). In a particularly preferred embodiment, the complexity measure is represented by a description length of the functional relationship, wherein the complexity according to the complexity measure is less the shorter the description length is. In a particularly preferred embodiment, the description length comprises the length of a binary or ASCII representation of the functional relationship and/or the number of nodes in the parsing tree represented by the functional relationship and/or the number of settable parameters of the functional relationship. In this case, the complexity according to the complexity measure is less the shorter the length of the binary or ASCII representation of the functional relationship is or the smaller the number of nodes in the parsing tree is or the smaller the number of the settable parameters is. The length of the binary or ASCII representation is represented in this case by the length of the corresponding binary code or ASCII code. The preparation of a parsing tree from a functional relationship is known per se from the prior art and therefore will not be explained in greater detail.
In a further variant of the method according to the invention, the action selection policies provided in step b) are based on expert knowledge. That is to say, the action selection policies are predefined by experts. These action selection policies are stored in a memory and are read out in step b) of the method.
In a further, particularly preferred variant of the method according to the invention, steps b) and c) are carried out by means of genetic programming and/or based on particle swarm optimization. In these methods, new action selection policies are generated and added to a population step-by-step, wherein action selection policies having a poor evaluation measure are discarded from the population again. Methods for genetic programming or particle swarm optimization are well known from the prior art and therefore will not be described in greater detail here.
In a further embodiment of the method according to the invention, the generation of the action selection policies in step b) is performed such that the action selection policies are derived from a predetermined optimum action selection policy. The predetermined optimum action selection policy can correspond in this case to the predefined optimum action selection policy from step c) of the method according to the invention. The predetermined optimum action selection policy generally has a complexity measure, the complexity of which is substantially higher than the predetermined complexity threshold. The derivation of the action selection policies is preferably performed by an approximation of the predetermined optimum action selection policy by means of a functional relationship, which represents a complexity measure having a complexity which is less than or equal to the predetermined complexity threshold.
In a further variant of the method according to the invention, the distance measure in step c) is determined such that one or more actions are generated from the predefined optimum action selection policy and one or more actions are generated from the respective action selection policy and the deviation between the action or actions which are generated from the optimum action selection policy and the action or actions which are generated from the respective action selection policy is determined. A greater deviation represents a greater distance measure in this case and therefore a lesser evaluation measure. The deviation can be, for example, a square deviation.
The method according to the invention is suitable in particular for the regulation and/or control of a technical system in the form of a gas turbine and/or wind turbine.
In the case of a gas turbine, the states of the gas turbine preferably comprise one or more of the following variables:
In contrast, the actions to be executed on the gas turbine preferably comprise a change of the setting of one or more fuel injection valves and/or a change of the position of one or more blades of the gas turbine, for example, the inlet blades.
In the case that the technical system is a wind turbine, the states of the wind turbine preferably comprise one or more of the following variables:
In contrast the actions to be executed on the wind turbine preferably comprise a change of the attack angle of the rotor blades of the rotor of the wind turbine and/or a change of the alignment of the rotor of the wind turbine in relation to the wind.
In addition to the above-described method, the invention furthermore relates to a computer program product having a program, which is stored on a machine-readable carrier, for carrying out the method according to the invention or one or more preferred variants of the method according to the invention when the program runs on a computer.
An exemplary embodiment of the invention will be described hereafter on the basis of appended
The embodiment described hereafter enables the determination of an action selection policy for the control or regulation of a technical system, which has a low complexity and can therefore be understood by the operator of the technical system or a human expert. The action selection policy furthermore has a high evaluation measure, which represents the suitability of an action selection policy for the regulation or control of the technical system. As already described above, the method is suitable in particular for the regulation and/or control of a gas turbine or a wind turbine.
The starting point of the method of
In a step S1 of the method of
Finally, in a step S2, further action selection policies PO are generated by means of particle swarm optimization (abbreviated PSO) from the provided action selection policies of step S1. In this case, new action selection policies, which are added to the population of the action selection policies, are generated by changing the free parameters. The new action selection policies are subsequently evaluated based on an evaluation measure EM.
A higher evaluation measure represents in this case a better suitability of an action selection policy for the regulation or control of the technical system. Since only the free parameters in the action selection policy predefined by the expert are optimized, it is ensured that all new action selection policies have a complexity measure CM which is less than the complexity threshold CT. In the scope of the particle swarm optimization, action selection policies having low evaluation measures are discarded and new action selection policies are added to the population again and again step-by-step, until finally, based on an abort criterion, the action selection policy PO′ which has the highest evaluation measure is determined from the population.
Steps S1 and S2 as described above can be summarized as follows:
a=sin(k*x)−g*x2
The ascertained action selection policy PO′ is finally used in a step S3 for the regulation or control of the technical system. That is to say, based on the present state and optionally additional previous states of the technical system, the action which is executed at the present point in time on the technical system is determined by means of the action selection policy PO′.
The evaluation measure EM, which is calculated in step S2, can be ascertained in various ways. In one variant, the evaluation measure represents the above-described distance measure between a respective action selection policy and a predefined optimum action selection policy. The evaluation measure can also represent the above-described reward measure or the above-described quality measure or combinations of the distance measure, the reward measure and the quality measure. Instead of a particle swarm optimization, other methods for ascertaining the action selection policy PO′ can also be used in step S2 of the above-described method. For example, genetic programming, which is known per se, can be used.
A variant of steps S1 and S2 for ascertaining an action selection policy by means of genetic programming will be explained hereafter.
In step S1, functional building blocks, for example, sin(x), exp(x), are predefined by an expert, wherein x represents the state of the technical system having corresponding state variables. By randomly joining together the functional building blocks to form valid mathematical formulae based on the state variables, initial action selection policies PO are generated. The generation process guarantees in this case that only action selection policies are generated, the complexity measure of which does not exceed the complexity threshold CT. From the provided action selection policies of step S1, further action selection policies PO are finally determined in a step S2 by means of genetic programming. In this case, new action selection policies are generated, which are added to the population of the action selection policies. The new action selection policies are subsequently evaluated based on an evaluation measure EM. A higher evaluation measure represents a better suitability of an action selection policy for regulating or controlling the technical system in this case. By way of the genetic programming, it is ensured that all new action selection policies have a complexity measure CM which is less than the complexity threshold CT. In the scope of the genetic programming, action selection policies having low evaluation measures are discarded and new action selection policies are added to the population again and again step-by-step, until finally, based on an abort criterion, the action selection policy PO′ is determined from the population which has the highest evaluation measure.
Steps S1 and S2 as explained above can be summarized as follows:
a=sin(2.704*x)+0.629*x*x.
The embodiment of the invention described above has an array of advantages. In particular, a regulation or control of a technical system using an action selection policy having lower complexity is achieved. The action selection policy is therefore comprehensible by the operator of the technical system or a human expert and results in a higher acceptance of the regulation or control of the technical system carried out by the action selection policy. Furthermore, it is also ensured that the action selection policy is very well suited for the control or regulation of the technical system in spite of its low complexity.
[1] L. Busoniu, D. Ernst, B. De Schutter, R. Babu{hacek over (s)}ka, “Approximate Reinforcement Learning: An Overview”, Adaptive Dynamic Programming And Reinforcement Learning, IEEE Conference Proceedings, 2011
[2] S. Silva, M. Castelli, L. Vanneschi, “Measuring Bloat, Overfitting and Functional Complexity in Genetic Programming”, Proceedings of the 12th annual conference on Genetic and evolutionary computation, pages 877-884, ACM New York, N.Y., 2010
| Number | Date | Country | Kind |
|---|---|---|---|
| 10 2013 205 328.2 | Mar 2013 | DE | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2014/051187 | 1/22/2014 | WO | 00 |
