Patent Application
20210026999
The present application claims the benefit under 35 U.S.C. § 119 of German Patent Application No. DE 102019211076.2 filed on Jul. 25, 2019, which is expressly incorporated herein by reference in its entirety.
The present invention relates to a method for validating a simulation of a technical system. The present invention also relates to a corresponding device, a corresponding computer program, as well as to a corresponding storage medium.
In software engineering, the use of models for automating test activities and for generating test artifacts in the test process is summarized under the generic term model-based testing, MBT. It is conventional, for example, to generate test cases from models which describe the nominal behavior of the system to be tested.
Embedded systems, in particular, are dependent on coherent input signals from sensors. They, in turn, stimulate the environment thereof by transmitting output signals to the most diverse actuators. Therefore, in the course of the verification and upstream development phases of such a system, the model thereof (model in the loop, MiL), software (software in the loop, SiL), processor (processor in the loop, PiL) or total hardware (hardware in the loop, HiL) is simulated in a control loop, together with a model of the environment. In vehicle technology, simulators, which are based on this principle and are used for testing electronic control units, are sometimes referred to as component, module or integration testers, depending on the test phase and test object.
German Patent Application No. DE 10303489 A1 describes such a method for testing software of a control unit of a vehicle, where a test system at least partially simulates a controlled system that is controllable by the control unit, by output signals, which are transmitted to first hardware modules via a first connection, being generated by the control unit, and signals from second hardware modules being transmitted as input signals to the control unit via a second connection, the output signals being provided as first control values in the software and being additionally transmitted to the test system via a communication interface in real time with respect to the controlled system.
Simulations of this kind are common in various fields of technology and are used, for example, to test the suitability of embedded systems in power tools, motor control units for drive systems, steering systems and braking systems or even in autonomous vehicles in the early stages of the development thereof. Nevertheless, the results of related art simulation models are only included to a limited extent in release decisions due to a lack of confidence in the reliability thereof.
The present invention provides a method for validating a simulation of a technical system, a corresponding device, a corresponding computer program, as well as a corresponding storage medium.
The present invention is based on the realization that validating time signals from simulation models is a question of comparing the output of the simulation model and measured values from tests. Here, the selected simulation signals or measurement signals describe a quantity of interest, QOI which may be scalar or be available in a time series.
An example embodiment of the present invention takes into account the fact that validating a simulation on the basis of a measurement is the simplest case of a validation. To this end, either two scalars or two time series are compared to one another. In practice, however, to minimize statistical uncertainties in the validation, for the most part, a comparison of a plurality of retests with a plurality of repeated simulations is sought. This statistical comparison, which may be performed using what are generally referred to as trust or confidence intervals, for example, is customary in the case of scalars. Moreover, for scalar QOIs, the validation was provided using what are generally referred to as probability boxes, p-boxes. Here, it is a comparison of two cumulative distribution functions, CDFs.
Finally, the example method of the present invention described herein is based on the insight that the validation of a time series differs from that of a scalar by the use of a validation metric. A validation metric is a mathematical function that maps two time series onto a scalar that is likewise referred to as a validation metric. The most well-known example is the mean squared error, MSE, defined as follows:
The comparison of time series of a simulation with a measurement seems, in fact, to be available in the related art, not, however, a method for comparing a plurality of measurements and simulations using a validation metric. Conventional is merely directly comparing a plurality of time series from simulation and measurements using confidence bands. However, this statistical method has the disadvantage, for example, that simulations having excessively high frequency components reside within a confidence band of the measurements and may thus be incorrectly classified as valid.
This disadvantage is eliminated by the use in accordance with the present invention of a validation metric, which generically permits a comparison of properties, such as phase errors or magnitude errors of time series, for example.
An advantage of the example embodiment of the present invention resides in attributing the validation problem to scalar methods based on confidence interval, p-box or CDF. To this end, the time signals are not directly compared, but a validation metric is initially applied. In comparison to conventional methods, the metric is not only computed for the static pairing of a single simulation with a measurement, but for all combinations of existing measurements and simulation results.
The measures described herein make possible advantageous embodiments of the basic embodiment of the present invention, and improvements thereto. Thus, a use may be provided in the field of automated driving and of other automated systems, for example, of robotics. In these fields of application, in particular, an inventive verification and validation of embedded systems may effectively enhance the functional reliability thereof.
Another aspect of the present invention may provide for the direct use of simulations for uncertainty quantification, UQ, for instance, in contrast to related art deterministic simulations.
Exemplary embodiments of the present invention are illustrated in the drawing and explained in greater detail below.
A second matrix (40) sketched in
An exemplary application programming interface, API may include the following class definition: “that StatisticSignalMetric( . . . , measurements, simulation, signal comparator).” The “measurements” and “simulation” arguments represent here the above described matrices of measurements, respectively simulations. Finally, the third argument, “signal_comparator,” represents the function of the validation metric, which may be selected from a plurality of relevant metrics.
This first step (21) of method (20) provides a vector (50—
In a second step (process 22—
In CDF plot (70), limit (εtol) may be marked on the x-axis, and the probability of limit (εtol) being observed may be read off of the y-axis. Here as well, the shape of CDF (70) shows how robust the validation is.
This method (20) may be implemented, for example, in software or hardware or in a software-hardware hybrid, for example, in a processing station (80), as illustrated by the schematic representation of
| Number | Date | Country | Kind |
|---|---|---|---|
| 102019211076.2 | Jul 2019 | DE | national |
