Patent Application
20210319267
The present application claims the benefit under 35 U.S.C. § 119 of European Patent Application No. EP 20169266.2 filed on Apr. 14, 2020, which is expressly incorporated herein by reference in its entirety.
The present invention relates to a method for training a classifier, a method for classification, a method for operating an actuator, a computer program, a machine-readable storage medium, a control system, and a training system.
German Patent Application No. DE 10 2018 200 724 A1 describes a method for generating universal adversarial perturbations.
Classifiers may be employed in various technical devices. However, they may be prone to misclassifying maliciously changed inputs known as adversarial examples.
German Patent Application No. DE 10 2018 200 724 A1 describes a method for generating universal adversarial perturbations. Here, a perturbation is obtained which can be applied to almost arbitrary input signals for a classifier and fool the classifier into misclassifying the otherwise correctly classified input signals.
The perturbations hence pose a special risk when employing classifiers, especially when using the classifier in safety-critical products. For example, a classifier may be used in an autonomous vehicle for detecting pedestrians in order to avoid collisions with the pedestrians. Here, universal adversarial examples could potentially be used to maliciously hide pedestrians from the classifier, exposing them to the risk of being hit bit the autonomous vehicle.
It is hence of great importance to safeguard classifiers against these perturbations.
A method with features in accordance with an example embodiment of the present invention is able to train a classifier to become more robust against perturbations also known as universal adversarial examples.
In a first aspect, the present invention provides a computer-implemented method for training a classifier for classifying input signals provided to the classifier, wherein the classifier is configured to obtain an output signal characterizing a classification of the input signal. In accordance with an example embodiment of the present invention, the method for training includes the following steps:
The term “classifier” may be understood to mean a device that accepts an input signal and provides an output signal characterizing a classification of the input signal. For this, the classifier may obtain the output signal by feeding the input signal to a machine learning model, especially a neural network. In addition, the classifier may adapt the input signal before feeding it into the classifier, e.g., extracting features from the input signal. Additionally, the classifier may also post-process the output signal of the machine learning model.
The classifier may accept input signals of various modalities, especially images such as, e.g., video images, RADAR images, LIDAR images and/or ultrasonic images as well as thermal camera images. For images, the classifier may preferably comprise a convolutional neural network.
Alternatively, the classifier may accept audio data as input, e.g., in the form of a quantized signal or a feature representation of an audio signal such as MFCC. For audio data, the classifier may preferably comprise a transformer network or a recurrent neural network, e.g., an LSTM.
Alternatively, an input signal may also comprise data which is a combination of multiple sensor signals and/or sensor signals of different modalities. When providing both image and audio data in the input signal, the classifier may preferably comprise different kinds of neural networks for processing the respective parts of the input signal, e.g., a combination of one or multiple convolutional neural networks and one or multiple transformer networks.
The classification characterized by the output signal may assign one or multiple class labels to the input signal. Alternatively or additionally, it can be envisioned that the output signal characterizes a classification in the form of an object detection, especially when using image data as input signals. Alternatively or additionally, it can be envisioned that the output signal characterizes a semantic segmentation of the input signal.
For audio signals, a classification may assign class labels to an entire sequence of audio data. Alternatively or additionally, the classification may also be of the form of a detection of the beginning and ending of certain audio events (e.g., sirens in a microphone recording) of an audio sequence.
The first perturbations from the set of perturbations may be used as initializations for creating second perturbations which can be applied to the input signals from the first dataset in order to obtain adversarial examples for the classifier. An adversarial example may be understood as an input signal which has been adapted to cause a misclassification by the classifier.
Applying a perturbation to an input signal can be understood as overlaying the input signal with the perturbation. For example, when using images as input signals the perturbation may be given by a small image patch that may be superimposed onto a given input image in order to form an adversarial example. In this case, the position where to place the patch in the input image may be chosen at random. Alternatively or additionally, the patch may also be transformed by an affine transformation before applying it to the input image. For audio data, the perturbations may be superimposed onto a random part of an audio sequence in order to form an adversarial example.
It can be further envisioned, that the perturbations and the input signals have the same dimensionality and that applying the perturbation to the input signal may be achieved by adding together the perturbation and the input signal. For example, when using image data, a perturbation may have the same size as the images used for training the classifier. In this case, the pixel values of the perturbation and the image may be added to form an adversarial example. Additionally, the perturbation may be scaled by a scalar value before being applied to the input image. A similar approach may be used mutatis mutandis for applying a respective perturbation to an audio signal.
If the classifier provides an output signal for an input signal, the output signal may comprise a plurality of logit values, wherein each of these logit values corresponds to a class. Also, the input signal may be assigned a desired class and the output signal may be considered as characterizing a correct classification, if the logit value corresponding to the desired class is the largest among the plurality of logit values.
For assessing whether a second perturbation is stronger, i.e., more apt to be used for fooling the classifier than a first perturbation, the second perturbation may be applied to an input signal to obtain a corresponding second adversarial example and the first perturbation may be applied to the same input signal to obtain a first adversarial example. The classifier may then obtain a second output signal using the second adversarial example as input signal and a first output signal using the first adversarial example as input signal. The second output signal hence corresponds to the second perturbation while the first output signal corresponds to the first perturbation.
The second perturbation may then be considered stronger than the first perturbation if the logit value of the desired class is smaller in the second output signal than in the first output signal. Similarly, the second perturbation may also be considered to be stronger than the first perturbation if the largest logit value of the non-desired classes in the second output signal is bigger than the largest logit value of the non-desired classes in the first output signal. Both options may be understood as the classifier being more confident in a wrong prediction using the second perturbation rather than the first perturbation.
Alternatively or additionally, an output signal may comprise a plurality of probability values, wherein each of these probability values corresponds to a class, and the first and second output signal may be computed as before. In this case, the second perturbation may be considered stronger than the first perturbation if the probability value of the desired class is smaller in the second output signal than in the first output signal.
Training the classifier with the method according to an example embodiment of the present invention may be understood as a form of meta-training, wherein the set of perturbations as well as the classifier are trained. The classifier is trained with the goal of achieving as high of a classification accuracy (also known as performance) as possible, while the method also seeks to find as strong of a set of perturbations as possible.
Training the classifier with increasingly strong perturbations hence allows for hardening the classifier against strong adversarial attacks and makes it extremely robust.
The training can be carried out in an iterative fashion. In each training step, the method may select a batch of training data, comprising one or multiple tuples of input signals and desired output signals. For each input signal, a first perturbation may then be drawn randomly with replacement from the set of perturbations. Each first perturbation drawn this way may then be adapted to its corresponding input signal in order to determine a stronger second perturbation.
The classifier may then be trained by providing it with the adversarial examples obtained from applying the obtained second perturbations to their respective input signals and minimizing a classification loss characterizing the difference between the obtained output signals to the desired output signals for the respective input signals.
In addition, each first perturbation may be replaced in the set of perturbations by a weighted sum of the first perturbation itself and the second perturbation. This advantageously also trains the perturbations to become stronger in general. The benefit for the training method is that the classifier has to learn to defend itself against increasingly stronger perturbations. In turn, this helps the classifier to become robust against adversarial attacks in general.
In another aspect of the present invention, it can be further provided that at least one perturbation from the set of perturbations is provided based on a second adversarial example of the classifier, wherein the classifier is a pretrained classifier.
The process of training the classifier benefits from initializing the perturbations to be diverse. The advantage of this is that diverse perturbations lead to the classifier to become robust against a diverse plurality of adversarial examples and hence a diverse plurality of perturbations.
In order to obtain an initial set of strong perturbations, the classifier may be pretrained without a defense method against adversarial examples. Afterwards, one or multiple second adversarial examples can be obtained from the classifier, e.g., by adapting the images in the training dataset with known adversarial attack methods. The perturbations necessary to cause the one or multiple second adversarial examples may then be used as initial set of perturbations for the training.
An advantage of this approach is, that the perturbations used for initializing the training can already be used to form adversarial examples. The step of forming a second perturbation is hence greatly speed up as, e.g., gradient based approaches such as the iterative fast gradient sign method (I-FGSM) or projected gradient descent (PGD) require considerably fewer steps for obtaining the second perturbation from the first perturbation. In turn, this leads to a faster training, which subsequently leads to the classifier being able to extract information from more training samples during training. This has the effect that the classifier's performance and robustness is improved even further.
In another aspect of the present invention, it can be further envisioned that the second adversarial example is provided based on applying random noise at a random location of an input signal from the first dataset.
When using image data as input, the noise may be applied across an image. Alternatively, the noise may be applied to a smaller region of the image in the form of a patch. It can be further envisioned that the patch undergoes some form of transformation such as rotation, translation or scaling before being applied to the image. The random noise may further be used to obstruct certain pixels. Alternatively, the noise may be blended over the image.
Similarly, when using audio signals, the noise may be transformed by, e.g., translation, scaling of the amplitudes or scaling of individual frequencies before being applied to the audio signals. For application, the noise may be blended over the original audio signal. Alternatively, certain frequencies of the original signal could be replaced by certain frequencies of the noise.
Using random noise as initial perturbations allows for obtaining highly diverse initial perturbations. This diversity allows for defending the classifier against a wide range of adversarial examples. In turn, this improves the performance and robustness of the classifier.
In accordance with an example embodiment of the present invention, it can be further provided that one or multiple perturbations from the set of perturbations are provided according to the following steps:
For image data, scaling may be considered the scaling of intensity or pixel values. For audio data, the amplitude of an audio signal may be scaled to obtain an adapted input signal.
In the step of applying the adapted input signals as perturbations, it can be envisioned that each adapted input signal is applied to each input signal from the first dataset.
A first value for a corresponding adapted input signal may for example be determined by dividing the number of incorrectly classified new input signals corresponding to the adapted input signal by the number of all new input signals corresponding to the adapted input signal.
An advantage of this approach is that the adapted input signals show a diverse content and may hence serve as good initialization for obtaining diverse perturbations. In turn, this initialization reduces the time necessary to obtain the second perturbations. This leads to the training being speed up which allows the classifier to process more data, which leads to a better performance of the classifier.
In accordance with an example embodiment of the present invention, it can be further provide that the method for training further comprises the step of:
An advantage of this approach is that the classifier will not only be trained on adversarial examples but unperturbed input signals as well. This enables the classifier to not only focus on defending against the perturbations but also to achieve as high of a performance as possible on unperturbed images.
Example embodiments of the present invention will be discussed with reference to the following FIGS. in more detail below.
Shown in
Thereby, the control system (40) receives a stream of sensor signals (S). It then computes a series of actuator control commands (A) depending on the stream of sensor signals (S), which are then transmitted to the actuator (10).
The control system (40) receives the stream of sensor signals (S) of the sensor (30) in an optional receiving unit (50). The receiving unit (50) transforms the sensor signals (S) into input signals (x). Alternatively, in case of no receiving unit (50), each sensor signal (S) may directly be taken as an input signal (x). The input signal (x) may, for example, be given as an excerpt from the sensor signal (S). Alternatively, the sensor signal (S) may be processed to yield the input signal (x).
The input signal (x) is then passed on to a classifier (60).
The classifier (60) is parametrized by a set (Φ) of parameters, which are stored in and provided by a parameter storage (St1).
The classifier (60) determines an output signal (y) from the input signals (x). The output signal (y) comprises information that assigns one or more labels to the input signal (x). The output signal (y) is transmitted to an optional conversion unit (80), which converts the output signal (y) into the control commands (A). The actuator control commands (A) are then transmitted to the actuator (10) for controlling the actuator (10) accordingly. Alternatively, the output signal (y) may directly be taken as actuator control commands (A).
The actuator (10) receives actuator control commands (A), is controlled accordingly and carries out an action corresponding to the actuator control commands (A). The actuator (10) may comprise a control logic which transforms an actuator control command (A) into a further control command, which is then used to control actuator (10).
In further embodiments, the control system (40) may comprise a sensor (30). In even further embodiments, the control system (40) alternatively or additionally may comprise the actuator (10).
In one embodiment, the classifier (60) may be designed to identify lanes on a road ahead, e.g., by classifying a surface of the road and markings on the road, and identifying lanes as patches of road surface between the markings. Based on an output of a navigation system, a suitable lane for pursuing a chosen path can then be selected, and depending on a present lane and said target lane, it may then be decided whether a vehicle (100) is to switch lanes or stay in the present lane. The actuator control command (A) may then be computed by, e.g., retrieving a predefined motion pattern from a database corresponding to the identified action.
Additionally or alternatively, the classifier (60) may also detect road signs and traffic lights in the environment (20). Upon identifying road signs or traffic lights, depending on an identified type of road sign or an identified state of said traffic lights, corresponding constraints on possible motion patterns of the vehicle (100) may then be retrieved from, e.g., a database, a planned trajectory of the vehicle (100) may be computed according to the constraints, and the actuator control command (A) may be computed to steer the vehicle (100) such as to execute the planned trajectory.
Additionally or alternatively, the classifier may also detect pedestrians and/or vehicles in the environment (20). Upon identifying pedestrians and/or vehicles, a projected future behavior of the pedestrians and/or vehicles may be estimated, and based on the estimated future behavior, a trajectory may then be selected such as to avoid collision with the identified pedestrians and/or vehicles, and the actuator control command (A) may be computed to steer the vehicle (100) such as to execute the trajectory.
In further embodiments, it can be envisioned that the control system (40) controls a display (10a) instead of or in addition to the actuator (10).
Furthermore, the control system (40) may comprise a processor (45) (or a plurality of processors) and at least one machine-readable storage medium (46) on which instructions are stored, which, if carried out, cause the control system (40) to carry out a method according an aspect of the invention.
The sensor (30) may comprise one or more video sensors and/or one or more radar sensors and/or one or more ultrasonic sensors and/or one or more LiDAR sensors and or one or more position sensors (like e.g. GPS). Some or all of these sensors are preferably but not necessarily integrated in the vehicle (100).
Alternatively or additionally, the sensor (30) may comprise an information system for determining a state of the actuator system. One example for such an information system is a weather information system which determines a present or future state of the weather in the environment (20).
For example, using the input signal (x), the classifier (60) may, for example, detect objects in the vicinity of the at least partially autonomous robot. The output signal (y) may comprise an information which characterizes where objects are located in the vicinity of the at least partially autonomous robot. The actuator control command (A) may then be determined in accordance with this information, for example to avoid collisions with the detected objects.
The actuator (10), which is preferably integrated in the vehicle (100), may be given by a brake, a propulsion system, an engine, a drivetrain, or a steering of the vehicle (100). The actuator control commands (A) may be determined such that the actuator (or actuators) (10) is/are controlled such that vehicle (100) avoids collisions with the detected objects. Detected objects may also be classified according to what the classifier (60) deems them most likely to be, e.g., pedestrians or trees, and the actuator control commands (A) may be determined depending on the classification.
In further embodiments, the at least partially autonomous robot may be given by another mobile robot (not shown), which may, for example, move by flying, swimming, diving or stepping. The mobile robot may, inter alia, be an at least partially autonomous lawn mower, or an at least partially autonomous cleaning robot. In all of the above embodiments, the actuator command control (A) may be determined such that propulsion unit and/or steering and/or brake of the mobile robot are controlled such that the mobile robot may avoid collisions with the identified objects.
In a further embodiment, the at least partially autonomous robot may be given by a gardening robot (not shown), which uses the sensor (30), preferably an optical sensor, to determine a state of plants in the environment (20). The actuator (10) may control a nozzle for spraying liquids and/or a cutting device, e.g., a blade. Depending on an identified species and/or an identified state of the plants, an actuator control command (A) may be determined to cause the actuator (10) to spray the plants with a suitable quantity of suitable liquids and/or cut the plants.
Shown in
The control system (40) then determines actuator control commands (A) for controlling the automated personal assistant (250). The actuator control commands (A) are determined in accordance with the sensor signal (S) of the sensor (30). The sensor signal (S) is transmitted to the control system (40). For example, the classifier (60) may be configured to, e.g., carry out a gesture recognition algorithm to identify a gesture made by the user (249). The control system (40) may then determine an actuator control command (A) for transmission to the automated personal assistant (250). It then transmits the actuator control command (A) to the automated personal assistant (250).
For example, the actuator control command (A) may be determined in accordance with the identified user gesture recognized by the classifier (60). It may then comprise information that causes the automated personal assistant (250) to retrieve information from a database and output this retrieved information in a form suitable for reception by the user (249).
Shown in
Shown in
Shown in
In a first step (601) a plurality of perturbations is initialized. For initialization, the input signals from the training dataset can each be scaled with a random number to form perturbations. Each perturbation is then be applied to each input signal of the training dataset in order to obtain a plurality of new input signals. For each perturbation, a first value is then obtained by computing the fraction of new input signals corresponding to an adapted input signal for which the classifier provides an incorrect classification with respect to all new input signals corresponding to the adapted input signal.
Alternatively, the input signals may be downsampled and/or cropped in order to obtain perturbations. These perturbations can be applied to parts of the input signals of the training dataset in order to obtain the plurality of new input signals. For example, when using images as input signals, the perturbations may be applied as patches to the input signals of the training dataset.
Alternatively, the perturbations may be initialized as noise which is applied to the input signal, either to the entire input signal or parts thereof, wherein each of the perturbations is then further adapted to an individual input signal from the training dataset to fool the classifier. This can be achieved using, e.g., the I-FGSM or the PGD attack algorithms for creating adversarial examples.
In a second step (602) a subset of the training dataset is then selected to serve as training batch. For this, a desired amount of input signals and corresponding desired output signals is chosen at random form the training dataset.
In a third step (603) a perturbation from the plurality of perturbations is drawn randomly with replacement for each pair of input signal and corresponding desired output signal from the training batch.
In a fourth step (604) each of the drawn perturbations is adapted to its corresponding input signal and desired output signal such that when the perturbation is applied to the respective input signal an adversarial example for the classifier (60) is created. This can be achieved by, e g, miming the I-FGSM or PGD attack algorithms. For each perturbation, this step returns a second perturbation which has been adapted to the respective input signal.
In a fifth step (605) each second perturbation is applied to its respective input signal in order to obtain a plurality of adversarial examples for the classifier (60).
In a sixth step (606), the classifier may be trained such that it predicts the desired output signal for an adversarial example obtained in the previous step. For this, a classifier can be trained using (stochastic) gradient descent or an adapted form thereof, e.g., Adam. When training a classifier (60) this way, the classifier predicts an output signal based on the adversarial example. The predicted output signal is then compared to the desired output signal using a loss function. The classifier is then adapted such that a loss value obtained from the loss function becomes smaller when again processing the adversarial example by the classifier (60) and comparing the then predicted output signal to the desired output signal. For example, when using, e.g., a neural network as classifier (60), a plurality of parameters of the neural network are adapted according to the negative gradient of the parameters with respect to the loss value.
Additionally, the classifier may also be trained using the input signal without altering it and the corresponding desired output signal.
In a seventh step (607) each randomly drawn perturbation is replaced in the plurality of perturbations by a linear combination of the perturbation itself and its corresponding second perturbation. The replacement may, e.g., be obtained according to the formula:
l=(1−σ)·a1+σ·a2,
wherein l is the linear combination, a1 is the perturbation, a2 is the second perturbation and a is a predefined value. This formulation can be understood as training the perturbations through meta-learning, wherein a is akin to a learning rate in gradient-based learning algorithms.
In an eight step (608) the steps two to seven are repeated. The steps may for example be repeated until the loss value falls below a predefined threshold. Alternatively, training may be carried out for a predefined amount of iterations. Having completed the desired number of iterations, the classifier (60) may then be used in the control system (40).
Shown in
The obtained adapted input signal (xi) is then provided to the classifier (60) which determines an output signal (ŷ) from the adapted input signal (xi). The output signal (ŷ) is then passed back to the training unit (150), where the selected perturbation is adapted using projected gradient descent such that it becomes stronger. The procedure of applying the stronger perturbation to the input signal to obtain an adapted input signal (xi), determining an output signal (yi) for the adapted input signal (xi) and adapting the perturbation to become stronger is then repeated for a desired amount of iterations.
After the desired amount of iterations, the output signal (ŷ) and desired output signal are passed to an update unit (180). Additionally, the training unit (150) stores an updated set (ξ′) of perturbations in the perturbation database by replacing the selected perturbation with the adapted perturbation.
Based on the output signal (ŷ) and the desired output signal (yi), the update unit then determines an updated set (Φ′) of parameters for the classifier (60) using, e.g., stochastic gradient descent. The updated set (Φ′) of parameters is then stored in the parameter storage (St1).
In further embodiments, the training process is then repeated for a desired amount of iterations, wherein the updated set (Φ′) of parameters is provided as set (Φ) of parameters by the parameter storage (St1) in each iteration and the updated set (ξ′) of perturbations is provided as set (ξ) of perturbations by the perturbation database (St3).
Furthermore, the training system (140) may comprise a processor (145) (or a plurality of processors) and at least one machine-readable storage medium (146) on which instructions are stored, which, if carried out, cause the training system (140) to carry out the training method according an aspect of the invention.
In even further embodiments, the training unit (150) selects randomly for each iteration whether a perturbation is to be applied to an input signal or not. If it decides that no perturbation is to be applied to the input signal, the classifier (60) is provided with the input signal as adapted input signal (xi) and no perturbation updates are computed.
| Number | Date | Country | Kind |
|---|---|---|---|
| 20169266.2 | Apr 2020 | EP | regional |
