Patent Application
20240043022
This application claims priority on German Patent Application No 10 2022 119 715.8 filed Aug. 5, 2022, the entire disclosure of which is incorporated herein by reference.
The invention relates to a method, system, and computer program product for objective assessment of the performance of an ADAS/ADS system.
Modern vehicles are equipped with a variety of driver assistance systems or automated driving assistance functions to assist the driver in driving and to increase safety. For example, driver assistance systems support speed and distance control and contain lane-keeping and lane-changing functions. A certain maximum speed can be set that is not exceeded as long as the speed limiting function is enabled. Distance controls use radar sensor and/or camera systems to monitor and maintain a certain distance from a vehicle ahead and vehicles in the side region. This leads to improved driving convenience and safety, especially when driving on a highway and during overtaking maneuvers.
This trend towards advanced driver assistance systems (ADAS) and automated driving systems (ADS) for motor vehicles, aircraft, marine vehicles and other moving objects requires extensive safeguarding strategies, because the responsibility for vehicle management is no longer entirely with the driver, but rather active functions are taken over by computer units in the vehicle. These systems employed in fully or semi-autonomously moving objects must have a very low error rate in their driving behavior. The detection and classification of other objects and the interpretation of traffic scenarios near a vehicle are important prerequisites to ensure a safe functionality of an ADAS/ADS system. For this purpose, the targeted testing and training of the advanced driver assistance systems and automated driving systems are required for both extreme and exceptional situations (corner cases) and also for day-to-day situations. Such extreme situations result from a particular combination of factors. Examples of these factors are infrastructural particularities, such as the type of road, the edge construction on a road and the quality of the markings as well as ambient conditions, such as weather, the time of day and the time of year. Furthermore, the behavior of the other road users, the geographic topography and the weather conditions play a major role.
Key performance indicators (KPI) are used to evaluate the performance and functionality of one or more functions or the overall performance of an ADAS/ADS system. KPIs are primarily numeric values, for example on a scale of 1 to 100. KPIs are used to describe the performance of an ADAS/ADS system, and KPIs can be established for various evaluation categories such as comfort, safety, naturalness of travel and efficiency. Additionally, further KPIs can be implemented to verify the correct functionality of an ADAS/ADS system. One example of a KPI is the evaluation of a minimum distance from another vehicle or a mean acceleration in a deceleration scenario.
ADAS/ADS systems typically are developed independently by various manufacturers and thus differ from one another, for example in terms of the components used, the software versions, and the application datasets. The calculated KPIs are therefore context-dependent, because they are dependent on both the performance of a specific ADAS/ADS system and the scenario under consideration. Therefore, KPIs for variously configured ADAS/ADS systems can only be compared to one another in part, because it is not always clear whether the ADAS/ADS systems have also been tested with the same scenarios and from which data sources the scenarios have been generated. The meaningful value of a KPI metric therefore often only relates to a specific ADAS/ADS system, so that the comparability of KPIs is difficult for variously formed ADAS/ADS systems and results in inaccurate outcomes. This results in the performance of an ADAS/ADS system often not being objectively evaluated because a consistent metric for assessment is lacking.
DE 10 2019 209554 A1 discloses a method for testing the robustness of a technical system that can be simulated, wherein the technical system is simulated as a function of a test scenario and a test configuration, and the simulation result is evaluated.
DE 10 2017 221971 A1 discloses a method for adjusting a vehicle control system to an altered vehicle state, wherein a control quality of the vehicle control system is assessed, taking into account at least one control criterion. The control criterion is determined for a specific driving scenario by means of a simulation.
DE 10 2020 120141 A1 discloses a method for optimizing tests of control systems for automated driving dynamic systems, wherein relevant parameters and relevant system responses are defined and quasi-random parameter combinations of the relevant parameters and system responses of the test system are generated. A probabilistic system model is created and trained as a function of the generated parameter combinations and the system responses. System responses of the model generated by the system model then are assessed.
DE 10 2018 215351 A1 discloses a method for generating an information collection of driving scenarios. Data of the driving scenarios is generated by sensor data from at least one vehicle.
An object of the invention is to specify possibilities for analyzing and comparing assessment indicators (KPIs) for various test drives and variously trained advanced driver assistance systems (ADAS) and/or automated driving systems (ADS), which differ from one another with regard to the software versions used and application datasets to create the conditions for objective comparability of the performance of variously configured ADAS/ADS systems and/or driving functions.
In accordance with one aspect of the invention, test results from variously configured ADAS/ADS systems can be evaluated objectively and compared to one another by generating assessment indicators for scenarios. The scenarios can be real-world scenarios that were measured in real-world time or created from historical data, as well as simulated scenarios generated by simulation software. According to the invention, a calculated assessment indicator is set for a considered scenario in relation to a scenario parameter value of a selected scenario parameter of the scenario. This results in an objective metric for a calculated assessment indicator so that, based on the objective metric, the assessment indicators from various ADAS/ADS systems can be compared to one another. In a further development, the simulated scenarios generated by simulation and the scenarios created from measurements are compared to one another, and the quality of a simulation method can be derived from this comparison. The invention thus allows the performance of an ADAS/ADS system to be assessed objectively, precisely, and comprehensibly so that a comparison of variously configured ADAS/ADS systems is made possible, and possible measures for improvement can be developed. Overall, the estimation of the safety and functionality of one or more driving functions of an ADAS/ADS system for a particular driving task can thus be adjusted and improved significantly.
A first aspect of the invention provides a method for objective assessment of the performance of an ADAS/ADS system of a vehicle for a defined driving task in at least one selected scenario, in particular for testing and training at least one driving function of the advanced driver assistance system (ADAS) and/or the automated driving system (ADS). A scenario represents a traffic event in a time sequence and is defined by a selection of scenario parameters and associated scenario parameter values. The method comprises the method steps:
In a further development, the scenario parameter value of the selected scenario parameter of the respective real-world scenario and/or the respective simulated scenario is extracted by the evaluation module from the dataset of the real-world scenario and/or the simulated scenario.
In one embodiment, the assessment indicators comprise key performance indicators (KPIs).
In a further embodiment, the evaluation results are configured as KPI plots, and the values of a KPI in a KPI plot are represented as a function of the parameter values of a scenario parameter for a particular scenario, and each plot point includes a real-world scenario or a simulated scenario.
In particular, the KPI plots for variously formed ADAS/ADS systems and/or the KPI plots for various real-world scenarios and/or for simulated scenarios and real-world scenarios for an ADAS/ADS system are compared to one another. The KPI plots may be configured as histograms with segments for cluster analysis.
In a another development, the scenario identification module comprises a software application that uses the computational methods and/or algorithms of artificial intelligence; the simulation module comprises a software application that uses the computational methods and/or algorithms of artificial intelligence; the assessment module comprises a software application that uses the computational methods and/or algorithms of artificial intelligence; and the evaluation module comprises a software application that uses the computational methods and/or algorithms of artificial intelligence.
The computational methods and/or algorithms of artificial intelligence may be configured as mean values, minimum and maximum values, lookup tables, expected value models, linear regression methods, Gaussian processes, fast Fourier transforms, integral and differential calculations, Markov methods, probability methods, such as Monte Carlo methods, temporal difference learning, extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, recurrent neural networks, and/or folded neural networks.
A scenario parameter may comprise: a physical variable, a chemical variable, a torque, a speed, a voltage, a current strength, a speed, an acceleration, a lurch, a braking value, a direction, an angle, a radius, a location, a number, a movable object such as a motor vehicle, a person or a cyclist, a stationary object such as a building or tree, a road configuration such as a highway, a road sign, a traffic light, a tunnel, a roundabout, a turn-off lane, a traffic volume, a topographical structure such as an incline, a time, a temperature, a precipitation value, a weather condition and/or a time of year.
According to a second aspect, the invention provides a system for objective assessment of the performance of an ADAS/ADS system of a vehicle for a defined driving task in at least one selected scenario, in particular for testing and training at least one driving function of the advanced driver assistance system (ADAS) and/or the automated driving system (ADS). A scenario represents a traffic event in a time sequence and is defined by a selection of scenario parameters and associated scenario parameter values. The system comprises sensors connected to the vehicle, a scenario identification module, a simulation module, an assessment module, and an evaluation module. The sensors are configured to record data in real-time while traveling with the vehicle on a test path. The scenario identification module is configured to identify real-world scenarios from the real-time captured data of the sensors or from stored data. The simulation module is configured to generate simulated scenarios. The assessment module is configured to calculate an assessment indicator for at least one real-world scenario and/or an assessment indicator for at least one simulated scenario. The assessment indicator represents the performance of the ADAS/ADS system for the defined driving task. The evaluation module is configured to generate evaluation results. An evaluation result for a real-world scenario and/or a simulated scenario includes mapping between the determined assessment indicator and the scenario parameter value of a selected scenario parameter of the respective real-world scenario and/or the respective simulated scenario.
In a further development that the assessment indicators are configured as key performance indicators (KPIs) and the evaluation results are configured as KPI plots.
In a KPI plot, the values of a KPI are represented as a function of the parameter values of a scenario parameter for a particular scenario, and each plot point includes a real-world scenario or a simulated scenario.
In particular, the KPI plots for variously formed ADAS/ADS systems and/or the KPI plots for various real-world scenarios and/or for simulated scenarios and real-world scenarios for an ADAS/ADS system are compared to one another.
According to a third aspect, the invention provides an ADAS/ADS system for a vehicle that uses the method according to the first aspect for testing and training and/or for calibration and validation of the ADAS/ADS system.
According to a fourth aspect, the invention relates to a computer program product comprising an executable program code configured to carry out the method according to the first aspect when it is executed.
The invention is explained in further detail below on the basis of an exemplary embodiment shown in the drawings.
Real-world and also increasingly simulated traffic scenarios created by programming are used for testing, training, and securing driver assistance systems (ADAS) and automated driving systems (ADS). A scenario in the context of the invention is defined as a traffic event in a temporal sequence. An example of a scenario is driving on a highway bridge, turning off on a turn-off lane, passing through a tunnel, turning onto a roundabout, or stopping in front of a pedestrian crossing. Moreover, specific visibility conditions, for example due to twilight or bright sunlight, as well as environmental conditions such as the weather and the time of year, traffic levels, and certain geographic topographical conditions, can affect a scenario. For example, an overtaking maneuver can be described as a scenario in which a first vehicle is initially behind another vehicle, then makes a lane change to the other roadway, and increases speed to overtake the other vehicle. Such a scenario is also referred to as a cut-in scenario.
A “module” can be understood, for example, in connection with this disclosure to mean a processor and/or a processor unit and/or a memory unit for storing program instructions. The processor is configured to execute the program instructions for implementing or realizing the method disclosed herein or a step of the method. In particular, a module can be integrated into a cloud computing infrastructure 800.
A “processor” is understood in connection with the invention to mean, for example, a machine or an electronic circuit. In particular, a processor can be a master processor (central processing unit (CPU)), a microprocessor, or a microcontroller, for example an application-specific integrated circuit or a digital signal processor, optionally in combination with a memory unit for storing program instructions, etc. A processor also can mean a virtualized processor, a virtual machine, or a soft CPU. For example, it can also be a programmable processor equipped with configuration steps for carrying out the method or configured with configuration steps in such a way that the programmable processor realizes the features of the method, the system, the modules, or other aspects and/or partial aspects of the invention. In particular, the processor can include highly parallelized computing units and high-performance graphics modules.
In connection with the invention, a “storage module” may be a volatile memory in the form of a working memory (random access memory, RAM), or a permanent memory such as a hard drive or a data carrier or, for example, a replaceable memory module. However, the storage module can also be a cloud-based storage solution.
“Data base” means both a memory algorithm and hardware in the form of a memory unit. In particular, a database can be part of a cloud computing infrastructure.
“Data” in connection with the invention should be understood as raw data as well as data already prepared from measurement results from sensors as well as from other data sources.
The objective assessment of the performance of an ADS/ADS system 210 and/or a travel function is based according to this disclosure on real-world scenarios SZri that occur during testing of the ADAS/ADS system 210 along a travel path and/or simulated scenarios SZsi. Both a real-world scenario SZri and a simulated scenario SZsi are defined by various scenario parameters P1, P2, . . . , Pn from a large number of possible scenario parameters Pi and associated scenario parameter values P1, P2, . . . , Pn from a large number of possible scenario parameter values PVi, wherein scenario parameter values PVi define the range of values of a scenario parameter Pi. For example, a scenario parameter Pi sets a physical variable, a chemical variable, a torque, a rotary speed, a voltage, a current strength, an acceleration, a speed a braking value, a direction, an angle, a radius, a location, a number, a movable object such as a motor vehicle, a person or a cyclist, a stationary object such as a building or a tree, a road configuration such as a highway, a road sign, a traffic light, a tunnel, a roundabout, a turn-off lane, a traffic volume, a topographical structure such as an incline, a time, a temperature, a precipitation value, a weather condition and/or a time of year. Scenario parameters Pi thus denote characteristics and features of a scenario SZi in the context of the present invention. An example of a scenario parameter Pi of a scenario SZi is the speed of an ego vehicle. For this scenario parameter “speed,” the range of values of the associated scenario parameter value PVi can comprise the range of 100 km/h to 180 km/h. By contrast, for another scenario SZk, the range of values of the parameter value PVi can range from 40 km/h to 70 km/h for the scenario parameter “speed.”
The real-world scenarios SZri based on real-world testing and measurements that arise when the vehicle 200 equipped with the ADAS/ADS system 210 is traveling along a test path are identified by the scenario identification module 300 using the software application 320 by means of the data captured by the sensors 220 in real-time and/or by means of stored data in the database 240 and optionally extended with additional requirement specifications that can be input using the input module 250.
The sensors 220 can be configured as radar systems with one or more radar sensors, LIDAR systems for optical distance and speed measurement, 2D/3D image recording cameras in the visible, IR and/or UV range, and/or GPS systems. Furthermore, accelerometers, speed sensors, capacitive sensors, inductive sensors, voltage sensors, torque sensors, engine speed sensors, precipitation sensors, and/or temperature sensors, etc. can be provided.
Data can be stored in the database 240 in the form of images, graphics, time series, characteristic values, etc. For example, target variables and target values that define a safety standard can be stored in the database 240. Furthermore, road network data can be provided with road specifications such as lanes and bridges, road infrastructure such as road surface, edge construction, roadway guidance, country-specific characteristics, weather conditions, etc. The database 240 can also be integrated into a cloud computing infrastructure 800.
A suitable real-world scenario SZri is derived from the data captured by the sensors 220 in real-time at a particular geographic location and/or from the data stored in the database 240 by the software application 320 of the scenario identification module 300. In particular, the software application 320 uses artificial intelligence algorithms to identify the real-world scenarios SZri. The artificial intelligence algorithms can be encoders and decoders with neural networks.
A neural network has neurons arranged in multiple layers and interconnected to one another variously. A neuron is able to receive information at its input from outside or from another neuron, evaluate the information in a particular manner, and pass it on in altered form at the neuron output to another neuron, or output it as a final result. Hidden neurons are located between the input neurons and the output neurons. Depending on the type of network, multiple layers of hidden neurons can be present and provide the forwarding and processing of the information. Output neurons deliver a result and issue it to the outside world. Neural networks can be trained through unsupervised or supervised learning.
The arrangement and linking of the neurons result in various types of neural networks, such as a feedforward neural network (FNN), a recurrent neural network (RNN) or a convolutional neural network (CNN). A convolutional neural network has multiple fold layers and is well suited to machine learning, pattern recognition and image recognition applications. Because a large proportion of the data captured by the sensors is present as images, particularly convolutional neural networks (CNN) are used.
The identified real-world scenarios SZri can be stored in a scenario database 340. The data for an identified real-world scenario SZri can include the designation for the selected real-world scenario SZri. In particular, the important scenario parameters Pi of the scenario SZri are saved.
The simulation module 400 also uses a software application 420 to generate simulated scenarios SZsi. The software application 420 can also use algorithms of artificial intelligence for this purpose. In particular, the simulation module 400 can be associated with a test scenario database 410 in which test scenarios SZti, scenario parameters Pi, scenario parameter values PVi, and other information generated from various data sources are stored. The test scenario database 410 can also be integrated into a cloud computing infrastructure 800. In addition, the simulation module 400 can be associated with further databases, in particular the database 240, in which additional information for performing a simulation is stored.
The software application 420 of the simulation module 400 can create test cases SZti for one or more driving tasks of the ADAS/ADS system 210 to be tested by selecting appropriate test scenarios Pi and scenario parameters PVi as well as scenario parameter values Ti and optionally further information. The particular driving task is formulated prior to starting the simulation, for example, by an expert such as an engineer. However, it can also be provided that the driving task is specified by a software application. An example of a driving task is to perform a passing maneuver on a highway, also referred to as a cut-in maneuver.
Such a cut-in scenario can be described by the speed vego of the first-person vehicle 200, the speed vtarget of a target vehicle, the spatial distance dcutin between the two vehicles, and the spatial length lcutin of the passing maneuver. The speed of one or both vehicles can also change during the passing maneuver through an acceleration or braking operation. This also applies to the spatial distance dcutin between the two vehicles.
The following table shows possible scenario parameters Pi and scenario parameter values PVi of a cut-in scenario:
In particular, a test strategy can be provided for the selection and design of the test cases Ti in the simulation, which specifies how the test cases Ti are created. Various computational methods and algorithms, in particular algorithms of artificial intelligence, can be provided for establishing the test strategy.
After selecting a test case Ti, the software application 420 creates a simulated scenario SZsi for this test case Ti that reflects the simulation of the behavior of the ADAS/ADS system 210 or a travel function of the ADAS/ADS system 210 for a defined driving task.
To assess the performance of an ADAS/ADS system 210, the simulated scenarios SZsi created by the simulation are forwarded to the assessment module 500 for calculation of assessment indicators 570, in particular in the form of KPIs. For this purpose, the assessment module 500 comprises a software application 520 that evaluates the simulated scenarios SZsi with regard to the performance and functionality of one or more functions or the overall performance of the considered ADAS/ADS system 210, in particular in the form of performance indicators (KPIs). For this purpose, the software application 520 uses calculation methods and/or algorithms of artificial intelligence, such as neural networks for the creation of the assessment indicators 570.
KPIs are used to describe the performance of the ADAS/ADS system 210 to be tested, and various KPIs are established for evaluation categories, such as comfort, safety, naturalness of travel and efficiency. Further KPIs can be implemented to verify the correct functionality of the ADAS/ADS system 210 being tested. One example of a KPI is the evaluation of a minimum distance from another vehicle or a mean acceleration in a deceleration scenario. The KPIs can be represented by numeric values or Boolean values. For example, a scale of 1 to 100 can be provided for the KPI metrics.
Real-world scenarios SZri identified by the scenario identification module 300 are forwarded to the assessment module 500 for determining assessment indicators 570 in the form of KPIs to assess the quality of the ADAS/ADS system 210 used.
To compare the defined KPIs of various real-world scenarios SZri or the KPIs of simulated scenarios SZsi with real-world scenarios SZri , the scenario parameters Pi and the scenario parameter values PVi of the respective scenario SZi are used as assessment metrics. For this purpose, the simulated scenarios SZsi, the real-world scenarios SZri and the defined KPIs are forwarded to the evaluation module 700. The evaluation module 700 comprises a software application 720 that generates evaluation results 750 from the transmitted information in the form of KPI plots.
For the example of the cut-in scenario already considered, a first parameter P1 can be the relative speed vrel between the first-person vehicle 200 and a target vehicle, and the KPI can be the brake lurch, i.e., the temporal rate of change of acceleration of the vehicle 200 during the cut-in scenario. Such a KPI plot is shown in
From the KPI plot shown in
In particular, the KPI plots from variously configured ADAS/ADS systems 210 are compared to one another for a driving task to be able to objectively, precisely, and comprehensibly assess the performance of an ADAS/ADS system 210 in this way.
In
In step S10, real-world scenarios SZri are identified from real-time captured data from sensors 220 while traveling on a test path with the vehicle 200 or from stored data from a scenario identification module 300.
In step S20, simulated scenarios are generated SZsi by a simulation module 400.
In step S30, the real-world scenarios SZri and/or the simulated scenarios SZsi are communicated to an assessment module 500.
In step S40, the assessment module 500 calculates an assessment indicator 570 for at least one real-world scenario SZri and/or an assessment indicator 570 for at least one simulated scenario SZsi. The assessment indicator 570 represents the performance of the ADAS/ADS system 210 for the defined driving task.
In step S50, evaluation results 750 are generated by an evaluation module 700. An evaluation result 750 for a real-world scenario SZri and/or a simulated scenario SZsi includes a mapping between the respective determined assessment indicator 570 and the scenario parameter value PVci of a selected scenario parameter Pci of the respective real-world scenario SZsri and/or the respective simulated scenario SZsi.
The invention enables test results from variously configured ADAS/ADS systems to be evaluated and compared to one another objectively by calculating assessment indicators for scenarios and/or simulated scenarios generated from measurements. For this purpose, a calculated assessment indicator for a real-world scenario and/or a simulated scenario is set in relation to a scenario parameter value of a selected scenario parameter, in particular in the form of KPI plots. This results in an objective metric that enables the assessment indicators of variously formed ADAS/ADS systems to be compared to one another. For example, KPI plots generated from at least two ADAS/ADS systems may be displayed side-by-side on a display device, e.g., the display device being coupled to system 210, cloud computing infrastructure 800, etc., for comparison and analysis. In another embodiment, the KPI plots generated from at least two ADAS/ADS systems may be overlaid upon each other on the display device for comparison and analysis, for example, to align like color-coded segments to facilitate analysis. In a further embodiment, a processor, e.g., a processor disposed in the evaluation module 700 or any other module, may perform the comparison of the KPI plots via an image analysis algorithm or function. In this embodiment, the processor may determine a deviation between the KPI plots that exceeds a predetermined, adjustable threshold and generates a recommendation to reduce the deviation. In this example, the recommendation may be human readable text that describes the deviation and how to reduce the deviation. In another example, the recommendation may be a calibration of the ADAS/ADS system, e.g., the recommendation includes newly generated calibration parameters. In a further example, the recommendation may be generated as an executable program code to modify the operation of at least one ADAS/ADS system to reduce the deviation, where a processor of the at least one ADAS/ADS system is automatically updated with the newly generated executable program code.
The present invention thus allows the performance of an ADAS/ADS system to be assessed objectively, precisely, and comprehensibly so that possible measures for improvement can be developed on this basis. Overall, the estimation of the safety and functionality of one or more driving functions of an ADAS/ADS system for a particular driving task can be improved significantly. In this way, resources can be conserved, since actual departures from test routes both with standard traffic situations as well as with specific corner cases can be reduced.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10 2022 119 715.8 | Aug 2022 | DE | national |
