Patent Application
20220318620
This application claims priority to German Patent Application No. DE 10 2019 207 575.4, filed on May 23, 2019 with the German Patent and Trademark Office. The contents of the aforesaid Patent Application are incorporated herein for all purposes.
The invention relates to a method for assessing a function-specific robustness of a neural network. The invention also relates to a device for data processing, a computer program product and a computer-readable storage medium.
This background section is provided for the purpose of generally describing the context of the disclosure. Work of the presently named inventor(s), to the extent the work is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.
Machine learning, for example on the basis of neural networks, has great potential for an application in modern driver assistance systems and automated motor vehicles. In this case, functions based on deep neural networks process raw sensor data (by way of example, from cameras, radar or lidar sensors) in order to derive relevant information therefrom. This information includes, by way of example, a type and a position of objects in an environment of the motor vehicle, a behavior of the objects or a road geometry or topology. Among the neural networks, convolutional neural networks have in particular proven to be particularly suitable for applications in image processing. However, while these neural networks outperform classic approaches in terms of functional accuracy, they also have disadvantages. Thus, interference in captured sensor data or attacks based on adversarial interference can, for example, result in a misclassification or incorrect semantic segmentation taking place despite semantically unchanged content in the captured sensor data. Knowledge of a function-specific robustness of a neural network with respect to such interference is therefore desired.
A need exists to improve a method and a device for assessing a function-specific robustness of a neural network.
The need is addressed by the subject matter of the independent claims. Embodiments of the invention are described in the dependent claims, the following description, and the drawings.
The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description, drawings, and from the claims.
In the following description of embodiments of the invention, specific details are described in order to provide a thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the instant description.
In a first exemplary aspect, a method for assessing a function-specific robustness of a neural network is made available, comprising the following steps:
In a further exemplary aspect, a device for data processing is provided, comprising means for executing the steps of the method according to any one of the described embodiments.
In a further exemplary aspect, a computer program is further provided, comprising commands which, when the computer program is run by a computer, prompt the latter to execute the steps of the method according to any of the described embodiments.
In a further exemplary aspect, a computer-readable storage medium is also provided, comprising commands which, when run by a computer, prompt the latter to execute the steps of the method according to any of the described embodiments.
The method and the device make it possible to assess a robustness of a neural network, in particular of a convolutional neural network, with respect to interference. To this end, a training data set, with which the neural network is/has been trained, is changed. In this case, the changes made to the training data set do not change semantically meaningful content, but merely semantically insignificant content. In this case, semantically meaningful content denotes in particular a semantic context which is important for a function of the trained neural network. The semantically meaningful content is in particular the content which the function of the trained neural network is intended to recognize as part of a semantic segmentation or classification. In contrast to this, the semantically insignificant content is in particular content which may ideally be designed as desired without impairing a function of the trained neural network as a result. The thus changed training data set and the original training data set are subsequently applied to the trained neural network, that is to say the training data and the changed training data are in each case supplied to the trained neural network as input data. At least one activation differential between an activation produced via the training data and an activation of the neural network, which is produced via the changed training data corresponding hereto is subsequently determined. The original (i.e., undisturbed) and the changed (i.e., disturbed) training data are in this case always considered in pairs. The determined at least one activation differential is subsequently provided and constitutes a measure of a sensitivity or a robustness of the neural network with respect to a change made in each case by means of a manipulation method when the training data set is changed. In this case, the neural network may in particular be assessed all the more robustly the lower the at least one activation differential is.
A benefit of the method is that a robustness of a neural network with respect to disturbed input data may be assessed in an improved manner since an activation or an activation differential of, in particular within, the neural network is considered.
A neural network is in particular an artificial neural network, in particular a convolutional neural network. The neural network is in particular trained for a certain function, for example a perception of pedestrians in captured camera images.
The training data of the training data set may be configured to be one-dimensional or multi-dimensional, wherein the training data is marked (“labeled”) in terms of semantically meaningful content. For example, the training data may be captured camera images which are marked in terms of semantic content.
In order to change the training data of the training data set, various manipulation methods may be deployed. In this case, it is in particular provided that semantically meaningful content of the training data is not changed. This means in particular that only non-relevant context dimensions are changed. If the neural network is trained, for example, to recognize pedestrians in captured camera images, camera images used as training data are changed, when changes are made, in such a way that one or more pedestrians present in a captured camera image are not changed or are only changed in an irrelevant manner. In the example of the camera images, the following manipulation methods may be used, for example: photometric manipulation methods (e.g., a change in brightness, contrast, saturation), noise and blurring (e.g., Gaussian blur, Gaussian noise, salt-and-pepper noise) or adversarial manipulation methods (e.g., “Fast Gradient Sign Method”). More complex methods may also be applied as manipulation methods; for example, it may be provided that a summer scene is altered to a winter scene without semantically meaningful content (e.g., a depicted pedestrian) itself being removed. Furthermore, colors, textures or other properties of objects and/or surfaces of the objects may, for example, be changed; for example. a color of a motor vehicle may, for example, be changed or a reflection behavior of a surface of the motor vehicle. In particular, the following manipulations may be carried out individually or in combination with one another: added sensor noise in the training data, contrast, brightness and/or image sharpness shifts, hue shifts, color intensity shifts, color depth shifts, color changes of individual (semantic) objects, small changes to objects (e.g., dirt, a deflection, a reflection on the object, meteorological effects, stickers or graffiti on the object), a rotation and/or a shift and/or distortions in the training data, a change in the physical properties of objects (e.g., the reflection properties or the paint properties of a motor vehicle, etc.).
An activation is determined in particular on the basis of (inferred) values at the outputs of neurons of the neural network. In order to determine the activation differential, in particular the (inferred) values at the outputs of the neurons in the neural network are in each case compared with one another in pairs for the original and the changed training data.
In particular, the method is executed by means of a computing apparatus which may access a memory. The computing apparatus may be configured as a combination of hardware and software, for example as program code which is run on a microcontroller or microprocessor.
In some embodiments, it is provided that a robustness measure is derived and provided on the basis of the provided at least one activation differential. This may, for example, be a real number which makes it possible to assess the robustness and to compare a robustness of different neural networks with one another.
In some embodiments, it is provided that activation differentials are determined and provided by neurons and/or regions. This makes it possible to identify neurons and/or regions of the neural network that are particularly affected by a manipulation of the training data or are sensitive. This makes it possible to analyze sensitive neurons and/or regions of the neural network in detail, which may be taken account of, for example, during a subsequent adjustment of parameters or a construction or an architecture of the neural network. To this end, activation differentials are for example formed and provided in each case between the outputs of the neurons of the neural network, individually and/or in regions. It may for example be provided that an L2 distance (L2 standard) is formed between activation vectors which describe an activation of the neurons or regions.
If the neural network is configured as a convolutional neural network, it may be provided, for example, that an activation differential is determined and provided for each filter in the convolutional neural network.
In some embodiments, it is provided that determined activation differentials are in each case averaged over multiple neurons and/or over a region, wherein the averaged activation differentials are provided in each case. This makes it possible to analyze and evaluate an analysis of the activation differentials or a sensitivity of the neural network more efficiently. For example, an average activation differential may be calculated for multiple neurons and/or regions. The averaging may take place in particular with the aid of statistical methods, for example an expected value may be determined for averaging.
In some embodiments, it is provided that determined activation differentials are provided in a weighted manner according to a position of an associated neuron layer within the neural network. This makes it possible to take into account an influence which is to be expected on the outputs of the neural network since, as a rule, an increased sensitivity of a neuron layer in the vicinity of the input has a smaller influence on an end result supplied by the neural network than an increased sensitivity of a neuron layer in the vicinity of the output. If activation differentials of neurons and/or of regions of the neural network are averaged, the weighting may be taken into account when averaging in accordance with a position of the neuron layer in the neural network. The averaging may take place in particular with the aid of statistical methods; for example, an expected value may be determined for averaging.
In some embodiments, it is provided that activation differentials are in each case averaged over multiple inference runs, wherein in each case the averaged activation differentials are provided. In this case, it may in particular be provided that the multiple inference runs are each performed for training data changed with different manipulation methods. As a result, activation differentials of individual neurons and/or activation differentials averaged over multiple neurons and/or over regions may also be averaged and taken into account over multiple types of interference. The averaging may take place in particular with the aid of statistical methods; for example, an expected value may be determined for averaging.
In some embodiments, it is provided that determined activation differentials are provided in each case according to an associated manipulation method. For example, the respective activation differentials may be determined in each case for multiple manipulation methods for all neurons in the neural network and may in each case be provided according to the associated manipulation method. As a result, neurons and/or regions of the neural network may be analyzed in terms of a sensitivity to interference produced by determined manipulation methods.
In some embodiments, it is provided that the determined activation differentials are provided in a weighted manner according to a respective associated manipulation method. For example, an average or expected value of the activation differential may be determined for the neurons and/or regions of the neural network, wherein the respective activation differentials for the respective associated manipulation methods are taken into account in a weighted manner.
As a result, weighted activation differentials or averages or expected values of the activation differentials for individual neurons and/or activation differentials averaged over multiple neurons and/or regions are obtained in accordance with the manipulation method used in each case. This makes possible a summarizing assessment of the robustness of the neural network with respect to multiple disturbances or manipulation methods.
In some embodiments, it is provided that neurons and/or regions of the neural network are sorted according to the activation differentials determined in each case for these, and an associated ranking is provided. It may be provided, for example, that all of the (individual or averaged) activation differentials are sorted according to their amount and are provided in accordance with a ranking resulting from the sorting. This makes it possible to identify the most sensitively reacting regions, either averaged over all of the manipulation methods, or for individual manipulation methods. In an, if applicable, following step for adjusting a structure of the neural network, it may then be provided, for example, that merely the top 5% or 10% of the most sensitive neurons or regions are changed, but that the remaining neural network is left unchanged.
Reference will now be made to the drawings in which the various elements of embodiments will be given numerical designations and in which further embodiments will be discussed.
Specific references to components, process steps, and other elements are not intended to be limiting. Further, it is understood that like parts bear the same or similar reference numerals when referring to alternate FIGS. It is further noted that the FIGS. are schematic and provided for guidance to the skilled reader and are not necessarily drawn to scale. Rather, the various drawing scales, aspect ratios, and numbers of components shown in the FIGS. may be purposely distorted to make certain features or relationships easier to understand.
A schematic representation of a device 30 for executing the method is shown in
After performing the individual method steps, the determined activation differentials 7 and, if applicable, the averaged activation differentials 10 and the robustness measure 9 are output by the computing apparatus 32, for example via a suitable interface (not shown).
A schematic flow chart for illustrating an embodiment of the method for assessing a function-specific robustness of a neural network 1 is shown in
At least one changed training data set 4 is generated by manipulating the training data set 2 by means of a manipulation method 3, wherein the training data contained in the training data set 2 is changed while maintaining semantically meaningful content.
The training data set 2 and the changed training data set 4 are each applied to the neural network 1, that is to say, they are each fed to the neural network 1 as input data, wherein the input data is propagated through the neural network 1 as part of a feed-forward sequence, so that inferred results may be provided at an output of the neural network 1.
If the training data is, for example, captured camera images, an undisturbed camera image of the original training data set 2 is supplied to the neural network 1. A manipulated or disturbed camera image from the changed training data set 4 is (subsequently) also fed to the neural network 1. In this case, activations 5 are in each case determined for individual neurons and/or regions of the neural network and in each case compared with one another in pairs (undisturbed camera image/disturbed camera image), for example in a differential formation step 6. This differential formation step 6 supplies activation differentials 7 in each case for the neurons and/or regions under consideration. The determined activation differentials 7 are subsequently provided.
It may be provided that a robustness measure 9 is determined and provided on the basis of the determined activation differentials 7 in a robustness measure determination step 8. For example, a real number between 0 and 1 may be assigned to the determined activation differentials 7. Such a robustness measure 9 makes it possible to compare a robustness between various neural networks.
It may be provided that determined activation differentials 7 are averaged over multiple neurons and/or over a region, wherein the averaged activation differentials 10 are provided in each case.
It may also be provided that determined activation differentials 7 are provided in a weighted manner according to a position of an associated neuron layer within the neural network 1.
It may further be provided that activation differentials 7 are in each case averaged over multiple inference runs, wherein the averaged activation differentials 10 are provided in each case. In this case, averaging may in particular take place over inference runs which belong to changed training data 4 which has in each case been changed by means of different manipulation methods.
It may be provided that determined activation differentials 7 are in each case provided according to an associated manipulation method 3.
In some embodiments, it may be provided that the determined activation differentials are provided in a weighted manner according to a respective associated manipulation method.
It may be provided that neurons and/or regions of the neural network 1 are sorted according to the activation differentials 7 determined in each case for these, and an associated ranking is provided.
A schematic block diagram of an embodiment of the method for assessing a function-specific robustness of a neural network is shown in
A neural network is provided in a method step 100. A structure and weightings of the neural network are stored, for example, in a memory of a computer. The neural network has either already been trained on the basis of a training data set including training data or is trained as part of method step 100 on the basis of the training data set. The neural network is trained, for example, to evaluate captured camera images and to ascertain whether a pedestrian is depicted in the camera images. The input data of the neural network is therefore two-dimensional camera images. The training data of the training data set is accordingly marked (“labeled”) camera images.
In a method step 101, multiple changed training data sets are generated by manipulating the training data set, wherein the training data is changed while maintaining semantically meaningful content (e.g., pedestrians in the camera images). To this end, the camera images which form the training data of the training data set are changed with the aid of manipulation methods. In order to change the camera images, the following manipulations can, for example, be performed individually or in combination:
In a method step 102, the training data of the training data set and respective associated changed training data of the changed training data set are fed to the neural network as input data, that is to say output data is inferred by means of the trained neural network on the basis of this input data. In this case, at least one activation differential between an activation of the neural network via the training data of the original training data set and an activation via the respective corresponding changed training data of the changed training data sets is determined.
This may be averaged both over neurons and over regions of the neural network.
In the case of a neural network configured as a convolutional neural network, it may for example be provided that activation differentials are determined for the individual filters of the convolutional neural network. A metric for determining the activation differentials of the individual filters is, for example, as follows:
In this case, di is the activation differential of the filter having the index i, ^l(.,.) is an activation differential function, fi(x) is an output function of the filter having the index i, W ix Hi is a size of the output feature map of the filter having the index i, N is a number of images, xn is the original camera image (i.e., the original training datum), ^xn is the changed camera image (i.e., the changed training datum) and fi(x) is an output function of the filter having the index i. In principle, however, another metric may also be used.
An exemplary result of activation differentials for each of the filters in one convolutional neural network is shown schematically in
The determined activation differentials are provided in a method step 103. The activation differentials may be output, for example in the form of a digital data packet. In the simplest case, the activation differentials are merely output, for example as statistics in a range of 0 (no activation differential) and 1 (maximum activation differential).
It may be provided in a method step 104 that a robustness measure is derived and provided on the basis of the provided activation differentials. This may take place, for example, by deriving a key figure for all neurons and/or all regions of the neural network. In the simplest case, all (normalized) activation differentials may for example be added up and provided. It can, however, also be provided, in order to derive the robustness measure, that a function is provided, which depicts the activation differentials in a range of the real numbers between 0 (neural network is not robust with respect to the disturbances in the input data) and 1 (neural network is completely robust with respect to the disturbances in the input data).
It may be provided in method step 102 that determined activation differentials are in each case averaged over multiple neurons and/or over a region, wherein the averaged activation differentials are provided in each case.
It may also be provided in method step 103 that determined activation differentials are provided in a weighted manner according to a position of an associated neuron layer within the neural network. In particular, activation differentials of neurons or regions in neuron layers which are closer to the input of the neural network are weighted less heavily than activation differentials of neurons or regions in neuron layers which are closer to the output of the neural network. As a result, a greater influence may be given to a sensitivity of neuron layers which are closer to the output of the neural network during the assessment of the robustness.
It may further be provided in method step 102 that activation differentials are in each case averaged over multiple inference runs, wherein the averaged activation differentials are provided in each case. In particular, it is possible to average over the inference runs of changed training data which has been changed using different manipulation methods. As a result, the robustness may be assessed averaged over the individual manipulation methods. To this end, an expected value is, for example, determined for the activation differentials determined in each case on the basis of the changed training data (i.e., for a single neuron or for averaged regions).
It may further be provided in method step 102 that determined activation differentials are in each case provided according to an associated manipulation method. This is represented, by way of example, in
In some embodiments, it may be provided that the determined activation differentials are provided in a weighted manner according to a respective associated manipulation method. In the example shown in
It may also be provided that neurons and/or regions of the neural network are sorted according to the activation differentials determined in each case for these, and an associated ranking is provided. For example, the activation differentials shown in
1 Neural network
2 Training data set
3 Manipulation method
4 Changed training data set
5 Activation
6 Differential formation step
7 Activation differential
8 Robustness measure determination step
9 Robustness measure
10 Averaged activation differential
20 X-axis (filter index i)
21 Y-axis (normalized activation differential)
30 Device
31 Means
32 Computing apparatus
33 Memory
100-103 Method steps
The invention has been described in the preceding using various exemplary embodiments. Other variations to the disclosed embodiments may be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor, module or other unit or device may fulfil the functions of several items recited in the claims.
The term “exemplary” used throughout the specification means “serving as an example, instance, or exemplification” and does not mean “preferred” or “having advantages” over other embodiments. The term “in particular” used throughout the specification means “serving as an example, instance, or exemplification”.
The mere fact that certain measures are recited in mutually different dependent claims or embodiments does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10 2019 207 575.4 | May 2019 | DE | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2020/062110 | 4/30/2020 | WO |
