Patent Application
20220327429
This application claims priority to German Patent Application No. DE 10 2019 213 061.5, filed on Aug. 29, 2019 with the German Patent and Trademark Office. The contents of the aforesaid Patent Application are incorporated herein for all purposes.
The present invention relates to a method, a computer program with instructions, and a device for providing a classifier for an AI module for processing input data provided by a sensor system of a motor vehicle, and also relates to a classifier provided using such a method. The invention also relates to a method, a computer program with instructions, and a device for configuring a control system of a motor vehicle with a library of AI modules for processing input data provided by a sensor system of the motor vehicle, and also relates to a motor vehicle which uses such a method or such a device.
This background section is provided for the purpose of generally describing the context of the disclosure. Work of the presently named inventor(s), to the extent the work is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.
(Highly) automated driving cannot be realized at this point in time without artificial intelligence (AI)-based methods and, in particular, image data processing based on deep neural networks. However, AI models, even if designed to solve the same task, are very versatile and also differ greatly in their functional quality. In this context, the term functional quality describes the goodness or quality of the AI model with respect to the intended function or task. In current approaches, AI models are mainly differentiated based on their architecture and the data used in training. As a rule, attempts are made to compensate for insufficient functional quality by adding training data or increasing the complexity of the architecture.
Simple approaches to the use of artificial intelligence in motor vehicles are limited to the use of a single AI module or an ensemble of AI modules and its optimization. An AI module is understood here as a software component through which an AI model is implemented.
In another approach, the focus is not on developing a function that works well and safely in all situations. Rather, a set of good functions is combined, i.e., different AI modules are dynamically combined on the vehicle side.
A need exists for providing solutions that support a dynamic combination of available AI modules.
The need is addressed by a method, by a computer program with instructions, by a device, and by a classifier according to the independent claims. Embodiments of the invention are described in the dependent claims, the following description, and the drawings.
The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description, drawings, and from the claims.
In the following description of embodiments of the invention, specific details are described in order to provide a thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the instant description.
In some embodiments, a method for providing a classifier for an AI module for processing input data provided by a sensor system of a motor vehicle comprises the steps of:
In some embodiments, a computer program includes instructions that, when executed by a computer, cause the computer to perform the following steps for providing a classifier for an AI module for processing input data provided by a sensor system of a motor vehicle:
The term computer is to be understood broadly. In particular, it also includes workstations, distributed systems and other processor-based data processing devices.
For example, the computer program may be provided for electronic retrieval or stored on a computer-readable storage medium.
In some embodiments, a device for providing a classifier for an AI module for processing input data provided by a sensor system of a motor vehicle comprises:
When using AI modules or the AI models realized by these AI modules, it should be noted that there is a dependency between the functional quality of an AI model and the data it processes. This dependence ensures that AI models are not good or bad per se, but have an environment-dependent functional quality. By using the solution according to the teachings herein, a meaningful description of AI models and their capabilities can be created in a test phase. In doing so, a list of the most meaningful properties can be determined as a metric for the performance of an AI module. The AI models can thus not only be better understood, but also be used in more diverse ways, for example as ensembles of expert models.
In the testing phase, a classification system based on a set of contextual dimensions, a set of AI modules, and a set of test data evaluates the AI modules with respect to their expected functional quality relative to individual contextual parameters. The contextual parameters may comprise, for example, properties in the context of the data points or properties of an architecture of the AI modules. Here, the term data point refers to the input data for a given situation. For the test data, the ground truth is known, i.e., the correct results are available for the respective input data. In addition, the contextual parameters for the evaluation of the data with regard to the context dimensions are known for the test data. The contextual assignment makes the resulting classification comprehensible, testable and validatable. The resulting classifier is set up to classify the AI modules based on contextual parameters of input data with respect to the expected functional quality.
In some embodiments, the AI module realizes an AI model or a family of AI models in the sense of an ensemble. Usually, a single AI model is realized in an AI module, e.g., a neural network, which is classified in the context of the testing phase. However, the solution according to the teachings herein may also be used to determine an empirically based weighting function for the joint inference of a family of AI models in the sense of an ensemble, i.e., a collective of AI models that process the same input data.
In some embodiments, for determining the functional quality for a data point, an output of the AI module for the data point is compared with the associated ground truth. For this purpose, for example, an IoU metric (IoU: Intersection over Union; ratio between intersection and union, also referred to as Jaccard coefficient) may be used. Through the comparison to the ground truth, the functional quality can be determined in a simple way. In this context, the use of an IoU metric has proven useful, especially for AI modules for object recognition.
In some embodiments, the classifier is formed by a neural network. This has the benefit that the classifier may be trained in the test phase without having to know a relevance of the contextual parameters in advance. However, the classifier may also be realized by other functions.
For example, a classifier for an AI module is provided by means of a method according to the teachings herein. By executing such a classifier on given input data, a situation-dependent evaluation of the AI modules available for processing the input data can be performed.
In some embodiments, a method for configuring a control system of a motor vehicle with a library of AI modules for processing input data provided by a sensor system of the motor vehicle comprises the steps of:
In some embodiments, a computer program comprises instructions that, when executed by a computer, cause the computer to perform the following steps for configuring a control system of a motor vehicle with a library of AI modules for processing input data provided by a sensor system of the motor vehicle:
The term computer is to be understood broadly. In particular, it also comprises control devices and other processor-based data processing devices.
For example, the computer program may be provided for electronic retrieval or stored on a computer-readable storage medium.
In some embodiments, a device for configuring a control system of a motor vehicle with a library of AI modules for processing input data provided by a sensor system of the motor vehicle has:
The solution according to the teachings herein determines in a test phase those conditions or contextual parameters that have the most relevant influence on the functional properties of the available AI modules, for example deep neural networks for use in automatic driving. The classifier now uses these previously determined conditions to determine a particularly effective combination of several AI modules for the given input data in order to increase the effectiveness of the overall system.
It is beneficial to use a method or device according to the teachings herein in a motor vehicle. Use of the described solution is particularly useful if automatic driving of autonomy levels or level 4 or 5 is to be implemented. In this context, the AI modules may be set up in particular to perform environment recognition for an automatic driving function of a motor vehicle.
For example, the different AI modules may be adapted to different lighting conditions, different speeds, different vehicle environments, different driving situations, different environmental conditions, different driving conditions, or different objectives.
Adaptation to different lighting conditions is particularly beneficial for environment detection for an automatic driving function in changing lighting conditions. For example, the lighting conditions may change due to a change in the weather, driving into a tunnel or driving out of a tunnel, or a very short twilight, such as occurs near the equator, for example. An AI module can be provided as an expert system for each of the different lighting conditions.
Adaptation to different speeds is useful for 3D object recognition, for example. Here, it may be useful to provide AI modules as expert systems for different speeds of the vehicle, e.g., in case the vehicle drives onto a road with a speed limit that differs significantly from the speed limit of the previously traveled road.
When adapting to different vehicle environments, a distinction can be made, for example, between urban and rural environments, but also whether the vehicle is located in the city center or near a school or hospital.
When adapting to different driving situations, AI modules can, for example, be provided for driving on the highway, in parking garages, in traffic jams, or for complex intersections with a specific Car2X infrastructure.
With regard to adaptation to different environmental conditions, AI modules may be provided, for example, for specific weather conditions, lighting conditions, traffic densities, road types, times of day, or geographies. In this context, pedestrian density can also be considered. In particular, expert systems may be provided for roads with a high pedestrian density and expert systems for detecting pedestrians at different distances. When pedestrian density is high, the expert system must be able to detect the intent of pedestrians in close proximity to the vehicle. When pedestrian density is low, it is usually possible to drive faster. Here, again, it is important to detect pedestrians farther away early on. The intention of these pedestrians, on the other hand, is less important.
The driving behavior of the automatic driving function can be adapted for different driving conditions by using customized AI modules, e.g., the speed, the vehicle type, the presence of a trailer, parameters of the trip, or preferences of the vehicle occupants.
Different objectives can result from legal or environmental constraints. For example, AI modules may be provided for noise-reduced or low-emission driving. Other AI modules may be adapted to specific behavioral rules.
In all of these examples, it is useful to determine, based on contextual parameters, the AI modules to be used and the associated weights for processing the input data.
Further features of the present invention will be apparent from the appended claims and the following description in conjunction with the FIGS.
For a better understanding of the principles of the present invention, further embodiments will be discussed in more detail below with reference to the FIGS. It is understood that the invention is not limited to these embodiments and that the features described may also be combined or modified without departing from the scope of the invention as defined in the appended claims. Specific references to components, process steps, and other elements are not intended to be limiting. Further, it is understood that like parts bear the same or similar reference numerals when referring to alternate FIGS.
The test module 23 and the evaluation module 24 can be controlled by a control unit 25. If necessary, settings of the test module 23, of the evaluation module 24 or of the control unit 25 can be changed via a user interface 28. The data arising in the device 20 can be stored in a memory 26 if required, for example for later evaluation or for use by the components of the device 20. The test module 23, the evaluation module 24 and the control unit 25 can be implemented as dedicated hardware, for example as integrated circuits. Of course, they may also be partially or fully combined or implemented as software running on a suitable processor, for example a GPU or a CPU. The input 21 and the output 27 may be implemented as separate interfaces or as a combined bidirectional interface.
The processor 32 may comprise one or more processing units, such as microprocessors, digital signal processors, or combinations thereof.
The memories 26, 31 of the described embodiments may have both volatile and non-volatile memory regions and may comprise a wide variety of storage devices and storage media, for example, hard disks, optical storage media, or semiconductor memories.
The data module 52 and the evaluation module 53 can be controlled by a control unit 54. If necessary, settings of the data module 52, of the evaluation module 53 or of the control unit 54 can be changed via a user interface 57. The data arising in the device 50 may be stored in a memory 55 if required, for example for later evaluation or for use by the components of the device 50. The data module 52, the evaluation module 53 and the control unit 54 can be realized as dedicated hardware, for example as integrated circuits. Of course, they may also be partially or fully combined or implemented as software that runs on a suitable processor, for example a GPU or a CPU. The input 51 and the output 56 may be implemented as separate interfaces or as a combined bidirectional interface.
The processor 62 may comprise one or more processing units, such as microprocessors, digital signal processors, or combinations thereof.
The memories 55, 61 of the described embodiments may have both volatile and non-volatile memory regions and may comprise a wide variety of storage devices and storage media, for example, hard disks, optical storage media, or semiconductor memories.
In the following, another embodiment is discussed using
Distinctive properties in the context of the input data to be processed influence to a certain extent the functional quality of the processing AI modules. Such properties can be very diverse and are not necessarily intuitive for humans, such as the distribution of special color values, contrasts, or specific frequencies. Additionally, architectural properties of the AI modules also play a role. For example, specific features in the composition of a neural network have an impact on its performance. For example, if in addition to a learned neural network, a rule-based knowledge base for allowable road signs is also present, the resulting AI model will have better sign recognition performance than a comparable AI model without this knowledge base. Properties in the context of the input data to be processed include all those influences on the data that affect, or at least can affect, the functional behavior of the AI module. Such can be semantic, such as weather, traffic, environment, etc., as well as non-intuitive as described above.
During a test phase, all given AI modules NNi are now used for inferences for all data points dn from a test data set D. Here, the term inference refers to the process of using the trained model for conclusions. By comparing the outputs of the AI modules NNi with the respective ground truth Gn, in this example using an IoU metric, the functional quality FGi_n for each data point dn with the given contextual parameters Pn is determined. This information can now be used to train, for example, a neural network that learns during training the relevance of the different contextual parameters Pn as well as the dependence of the functional quality FGi of the different AI modules NNi on the contextual parameters Pn. In this way, a classifier K is created that outputs a functional quality FGi or weights Wi corresponding to the functional quality FGi for given contextual parameters P for all AI modules NNi independent of data points. Alternatively, the classifier K can output an empirically based weighting function for the joint inference of a family of AI modules NNi in the sense of an ensemble.
10 Selecting an AI module
11 Selecting a test data set
12 Applying the AI module to data points of the test data set
13 Determining a functional quality for the data points
14 Creating a classifier for the AI module
20 Device
21 Input
22 Database
23 Test module
24 Evaluation module
25 Control unit
26 Memory
27 Output
28 User interface
30 Device
31 Memory
32 Processor
33 Input
34 Output
40 Capturing of input data to be processed by AI modules
41 Evaluating the AI modules based on contextual parameters
42 Determining AI modules and weights to be used
50 Device
51 Input
52 Data module
53 Evaluation module
54 Control unit
55 Memory
56 Output
57 User interface
60 Device
61 Memory
62 Processor
63 Input
64 Output
70 Motor vehicle
71 Control system
72 Environmental sensor system
73 Control device
74 Data transmission unit
75 Memory
76 Network
80 Fusion module
81 Sensor system
dn Data point
D Test data set
E Entry date
FGi, FGi_n Functional quality
Gn Ground truth
K Classifier
NNi AI module
P, Pn Contextual parameter
Wi Weight
The invention has been described in the preceding using various exemplary embodiments. Other variations to the disclosed embodiments may be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor, module or other unit or device may fulfil the functions of several items recited in the claims.
The term “exemplary” used throughout the specification means “serving as an example, instance, or exemplification” and does not mean “preferred” or “having advantages” over other embodiments. The term “in particular” used throughout the specification means “for example” or “for instance”.
The mere fact that certain measures are recited in mutually different dependent claims or embodiments does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10 2019 213 061.5 | Aug 2019 | DE | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2020/073851 | 8/26/2020 | WO |
