Patent Application
20230036840
The present application claims the benefit under 35 U.S.C. § 119 of German Patent Application No. DE 10 2021 208 192.4 filed on Jul. 29, 2021, which is expressly incorporated herein by reference in its entirety.
The present invention relates to a method for the control of a vehicle by an operator. The present invention additionally relates to a corresponding system for the control of a vehicle by an operator, a corresponding computer program, and a corresponding memory medium.
In teleoperated driving (ToD), a vehicle is remotely controlled from a distance using a wireless connection by an operator or an algorithm. Sensor data of the vehicle to be remote-controlled are sent to a remote control center and corresponding driving commands are sent back to the vehicle to be carried out. The use of teleoperated driving functions frequently takes place in conjunction with overcoming predicted or unpredicted situations or to overcome inadequacies in automated driving.
U.S. Patent Application Publication No. US 2019/0385460 A1 describes a method, using which teleoperated or automated vehicles are excluded from specific areas, localities, or routes, and a visualization of these areas in a map. German Patent Application No. DE 10 2016 001 264 A1 describes a similar system, which permits autonomous vehicles to drive exclusively in a limited region.
German Patent Application No. DE 10 2016 222 219 A1 describes a method, in which problem cases of autonomous vehicles, for example, (near) collisions or hazardous areas are sent together with the occurring geo-position to a cloud, in order to warn other drivers/vehicles of potential problems in this way.
U.S. Patent Application Publication No. US 2019/0271548 A1 describes the possibility of superimposing data from camera sensor and LIDAR sensor from a vehicle with object data from a map in order, among other things, to extend the vision, in that map data are displayed behind actual concealing objects. This technology not only takes place for static situations, but also during travel, in that a video is played back.
Methods for fusion of various sensors are described in the literature. U.S. Pat. No. 10,592,784 B1 describes, for example, a method for detection by fusion of multiple sensors; for example, the detection of a rumble strip by fusion of the data of the following three sensors: camera, microphone, and IMU (inertial measurement unit). Document DE 10 2018 008 442 A1 describes a method for weather and/or visual range recognition, which evaluates data of the LIDAR sensor with the aid of machine learning and fuses them with other types of surroundings detection sensors of the vehicle.
U.S. Patent Application No. US 2020/0160559 A1 describes a system and a method for carrying out a multitask and/or multisensor fusion for three-dimensional object recognition, for example, to promote the perception and control of autonomous vehicles.
China Patent Application No. CN 107976182 A describes a multisensor fusion mapping system, which includes a route planning module, a data detection module, a data fusion module, and a mapping module.
An object of the present invention described here is to enable operators optimized control of teleoperated vehicles in all regions and situations. It is central for this goal that an adequate representation of the vehicle surroundings, based on a variety of different sensors, is always available to the operator. A camera sensor is the standard sensor here. However, there is an array of situations in which the camera sensor does not supply an adequate surroundings model. For example, in bad weather or in darkness, the fusion of the camera image with the data from the radar or LIDAR sensor is capable of better displaying the vehicle surroundings to the operator. In addition, pieces of information from the infrastructure, for example, intersection sensors, or from other vehicles via V2X, may also be incorporated into the surroundings model.
In accordance with an example embodiment of the present invention, situations in which data of multiple sensors are to be fused with one another are defined not only by bad weather or temporal aspects, but are frequently also dependent on the local conditions. Therefore, a so-called predictive map is central for the present invention, which describes for all locations of the traveled route which sensors are to be fused with one another in which form and intensity. If an operator controls a teleoperated vehicle, with the aid of the predictive map, it is possible to predict and initiate data of which sensors are to be transmitted from the vehicle to a control center and how they are to be fused with one another.
A method for the control of a vehicle by an operator is described. The vehicle may be an automated or semiautomated vehicle which also permits teleoperation. The term “vehicle” is understood as mobile means of transportation which are used for the transport of people, goods, or tools. These include, for example, passenger vehicles, trucks, semitrailer trucks without cab, vehicles from the agricultural field such as robot mowers, general transportation robots, construction machines, drones, aircraft in which teleoperation is used as a backup, trains, delivery vehicles/robots, container transporters, tugboats, and cranes.
When carrying out the method described according to an example embodiment of the present invention, the predictive map is used to control the vehicle in order to carry out optimal fusing of sensor data, which supplies the best possible representation to the operator.
The predictive map may be viewed in the case of a local reference as a type of further layer of a roadmap or navigation map. For example, it is indicated at an intersection of two roads in the roadmap in the predictive map that, when an operator steers a vehicle in, for example, the northerly direction, a first sensor, a second sensor, and additionally an optional third sensor are to be used to display the surroundings for the operator. In the case of specific situations, such as bad weather, darkness, or dazzling due to the sun, the predictive map does not represent a “classic” static map layer, but rather a more dynamic layer which always engages when the particular dependent situation occurs. This may take place independently of the specific location. For example, it may be shown for the situation “dazzling by sun” in the predictive map that the camera sensor is additionally fused with the LIDAR sensor for object recognition in order to display the surroundings in the best possible manner to the operator.
If the predictive map is used, the possible combinations of various sensors, namely switching over and changing the weighting of the sensors, are read out as a function of a spatial position or situation. Initially a situation and/or location reference of the vehicle is detected. Subsequently, data of a defined set of sensors are transferred. The data of the defined set of sensors are then fused and processed. The fused and processed data are subsequently displayed to the operator.
The term “sensor” in the meaning of the present invention is understood to include devices and systems which enable a comprehensive surroundings model of the vehicle to be controlled for the operator. The sensors are primarily situated at the vehicle, but may also be situated elsewhere, for example, in the road infrastructure or in parking garages. These include, for example, camera, LIDAR, radar, and ultrasonic sensors, V2X, IMU, microphone or microphone array, thermometer, rain sensor, infrared camera, and all sensors of the surrounding traffic and all other sensors of the vehicle to be controlled.
A digital roadmap may also be used as a sensor within the scope of the present invention, in that, on the one hand, sensor data are fused with the roadmap (expanded map matching) and, on the other hand, the digital roadmap may be the trigger for a fusion adaptation. If, for example, curves, turnoffs, tunnels, or slope changes are recognizable in the digital roadmap, relevant sensors and their fusing may be adapted to the upcoming or present situation. The digital roadmap is advantageously provided in a backend of a system for the control of the vehicle by an operator and does not first have to be transmitted from the vehicle. The digital roadmap may be stored for this purpose in any part of the system, but it is sufficient if parts of the map are acquired from the backend, for example, as an electronic horizon.
Fusion or fusing of data is understood as a priority-based or weighted combination of data of different sources with the goal of bringing about a shared result. The result may be a numeric result, an interpretation, or a representation of data here.
In accordance with an example embodiment of the present invention, the fusion may take place at arbitrary points of the system for the control of the vehicle, for example, in the vehicle, in a backend between the vehicle to be controlled and the operator, or also on the computer of the operator. The fusion may also be allocated onto multiple partial fusions at different points of the system, for example, to preprocess and reduce data prior to the wireless transmission and to process and finally fuse data after the wireless transmission. A fusion also does not have to take place uniformly for all sensor data from the surroundings, but rather may be applied adapted to partial areas of the surroundings.
The predictive map may be created/updated while the operator controls the vehicle. For the present position or for the present situation, for example, fog, the matching fusion parameters for this purpose are progressively loaded from the predictive map and the data of the defined set of sensors are transmitted according to these parameters, subsequently fused, and finally displayed to the operator.
The creation/update of the predictive map takes place in that a problematic situation and/or a problematic location is recognized by observation of the operator and/or marking by the operator. The problematic situation and/or the problematic location are subsequently stored in a first database for storing problematic situations and locations.
In accordance with an example embodiment of the present invention, a model, for example, a neural network, is trained to select the defined set of sensors and fuse the data of the defined set of sensors, by whose use the predictive map may be automatically created or updated. Recorded situations, including all sensor data and features which are incorporated in the predictive map, are simulated to teach a trained operator and this operator selects the sensors and priorities in such a way that he/she achieves the best view. This process may also take place iteratively (jumping forward and back multiple times in the scenario, driving through multiple times, various (trainer) operators) to obtain an optimized result. The model thus trained selects the prioritization and selection of the sensors upon application.
In accordance with an example embodiment of the present invention, to select a weighting with respect to the sensors from the detected situations, the following strategies may be followed:
In accordance with an example embodiment of the present invention, the transmission of sensor data takes place through a wireless network, for example, a mobile network or a WLAN. The wireless network also permits teleoperation from a distance. The sensor data to be transmitted from the vehicle to the backend cannot exceed the available data rate. Priorities may be assigned to the data streams which are based on parameters such as driving speed, traffic density, or weather. The parameters for determining the priorities may be detected not only at the point in time of the occurrence, but may also be stored in the predictive map. A method which enables a prediction of connection quality or quality of service (QoS) may be used within the scope of the present invention to check whether the sensor data may be transmitted at all at a specific location at a specific time.
Under certain circumstances it is not possible or reasonable to transmit the data of all sensors to the backend at any time, for example, if the transmission channel does not permit this or excessively high costs result. The sensors may therefore be differentiated into required sensors, whose data are to be transmitted in any case to the backend and displayed to the operator, for example, the camera image, and optional sensors, whose data the operator may explicitly request, or which are also to be transmitted automatically in special driving situations, for example, a siren in the surroundings. Whether and how the fusion of the data already takes place in the vehicle, in the backend, or only in the control center is decided statically on the basis of requests by the operator and/or dynamically on the basis of the existing situation by the system, as described above. An adapted reduction of the amount of sensor data by compression or selection of relevant details is also possible, as long as the requests of the operator are fulfilled.
In accordance with an example embodiment of the present invention, the predictive map may include a matrix which indicates whether and how data of individual sensors are fused with one another. A priority and/or a weighting of individual sensors are calculated here. For example, for a specific location or a specific situation, the data of a first sensor are to be fused using a weighting of 0.6, a second sensor using a weighting of 0.25, and a third sensor using a weighting of 0.15. In the case that the third sensor is optional and is therefore not also to be transmitted in the case of a mobile communication connection which is not absolutely perfect, it could also be recorded in the matrix how the proportion of the optional sensor is distributed to the required sensors, for example, ⅔ to the first sensor and ⅓ to the second sensor, so that the following distribution results for this case: the first sensor using a weighting of 0.7 and the second sensor using a weighting of 0.3.
In the fusion of sensor data, the different positioning of the sensors in the vehicle is to be known in order to equalize them in the fusion. Thus, for example, a transformation of pieces of 3D LIDAR information to the coordinates of a camera image taking into consideration the various sensor positions is to take place.
The transmission of data of acoustic sensors such as microphones may be initiated directly by the vehicle in case of the occurrence, for example, siren or driving over rumble strips. In the simplest case, the fusion may mean that the noises are played to the operator via headphones. However, it is also possible to attempt to recognize the noises in the vehicle or downstream components of the system and then have them incorporated in the fusion with the other sensors. In addition, it is also possible to attempt to determine the direction of the noise, for example, via a microphone array or via V2X, for example, ETSI CPM or CAM messages. This direction could then also be shown to the operator in addition in the visual surroundings model and additionally be used to employ the required sensor data quality for the area in which the source of the noise is suspected.
The observation of the operator is preferably carried out by detecting the stress level of the operator. If, for example, a high stress level of the operator is often observed at a location, this location may be marked as a potentially hazardous location and other operators are warned of this. A second operator could also be switched in for monitoring in order to provide more safety to the actively driving operator.
Additionally or alternatively, the observation of the operator may be carried out by the detection of the viewing direction of the operator, for example, by eye or head tracking. For example, rapid eye movements or a frequent look next to the road indicate a potentially hazardous situation.
However, it is also possible that the observation of the operator is carried out by detecting the behavior of the operator. For example, locations at which strong braking frequently takes place are potentially hazardous.
Problematic locations and/or situations may also be recognized by marking of locations and/or situations by the operator as critical, for example, using an extra button. For example, the operator may also add labels after the end of the trip about what was problematic there, and may possibly make optimization suggestions, for how an operator may proceed in future at the location or how the sensor data fusion is to be adapted.
An individualization by the operator is also possible, so that he/she has the option of switching on optional sensors always or depending on a specific situation.
The location of the occurrence of the above-described situations is then stored together with a description of the situation in the first database for storing problematic situations and locations. The data recorded there are aggregated with one another at a later point in time, from which the predictive map is created or updated, which is represented by a second database for storing situation-related and/or location-related detection, fusion, and display parameters.
The method provided according to the present invention preferably furthermore includes the following steps:
The fusion of the data of sensors is preferably allocated onto multiple partial fusions.
The method provided according to the present invention preferably furthermore includes the following steps:
After the search, an evaluation of identical situations and/or locations may be compiled. It is then checked whether the recognized situation and/or the recognized location is permanently critical. If the recognized situation and/or the recognized location is permanently critical, it is furthermore checked whether parameters are already present. If the parameters are already present, they are retrieved from the second database and taken into consideration when evaluating the recognized situation and/or the recognized location.
A further aspect of the present invention is to provide a system for the control of a vehicle by an operator according to the method provided according to the present invention.
The method according to the present invention is preferably carried out using one of the systems described hereinafter. Features described within the context of one of the systems accordingly apply to the method and, vice versa, features described within the context of the method apply to the systems.
The system provided according to an example embodiment of the present invention includes the following:
The system provided according to an example embodiment of the present invention preferably furthermore includes a backend, in which data of sensors between the wireless network and the control center are processed. The backend may be a data processing center. The backend above all fulfills the purpose here of carrying out computing-intensive fusions of data of different sources, for example, different vehicles or infrastructure units, for example, traffic signs or traffic signal systems, and then transferring the data preprocessed to the control center.
The backend may be a part of the control center or may be separate from the control center.
The fusion of the data of individual sensors preferably takes place at arbitrary points of the system provided according to the present invention. For example, the fusion may take place in the vehicle, in the backend between the vehicle to be controlled and the operator, or also on the computer of the operator.
Furthermore, a computer program, which is configured to carry out the method according to the present invention, and a machine-readable memory medium, on which the computer program according to the present invention is stored, are provided.
The present invention enables the safe teleoperated control of a vehicle by an operator, in that he/she is always supplied with a comprehensive surroundings model of the vehicle, made up of the adapted fusion of data of various sensors.
Specific embodiments of the present invention are explained in greater detail on the basis of the figures and the following description.
In the following description of the specific embodiments of the present invention, identical or similar elements are identified by identical reference numerals, a repeated description of these elements being omitted in individual cases. The figures only schematically represent the subject matter of the present invention.
It may be seen from
Situations in which data of multiple sensors 12 have to be fused with one another are defined not only by bad weather or temporal aspects, but are often also dependent on the local conditions.
It is apparent from the representation according to
Backend 32 is designed here as a data processing center, in which the data of sensors 12 are processed between wireless network 40 and control center 34. In the present case in
The fusion may take place at arbitrary points of system 100. The fusion may also be allocated onto multiple partial fusions at different points of system 100, for example, to preprocess and reduce data prior to the wireless transmission and to process and finally fuse data after the wireless transmission.
Thus, for example, an in-vehicle fusion of data of LIDAR sensor 14 and camera sensor 16 of vehicle 10 may be carried out. The fusion result is transmitted via wireless network 40 to backend 32.
Optionally, infrastructure unit 20, in the present case traffic sign 22, may be configured to fuse data of various sensors 12. The fused data are also transmitted to backend 32.
Backend 32 may be configured to receive the data sent from vehicle 10 and infrastructure unit 20, fuse them, and transmit the data fused there further to control center 34.
Control center 34 may also be configured to fuse the received data. The data thus fused are provided directly to operator 36, for example, via audiovisual or haptic devices.
Initially, data of a LIDAR sensor 14 are detected in a first step 201 and data of a camera sensor 16 are detected in a second step 202. Subsequently, the data of LIDAR sensor 14 and camera sensor 16 are brought together in a third step 203.
The data of LIDAR sensor 14 and camera sensor 16 are then fused with one another. There is not only one fusion possibility for a combination of two sensors 12. Two possibilities 210, 220 for fusing data of LIDAR sensor 14 and camera sensor 16 are shown in
Finally, in a seventh step 207, the fused data of LIDAR sensor 14 and camera sensor 16 are displayed to operator 36.
In this situation, the data of camera sensor 16 are fused with the data of a LIDAR sensor 14 of vehicle 10. The fusion is carried out on the basis of a weighting of particular sensors 12. In the present case, a weighting of camera sensor 16 of 0.5 and a weighting of LIDAR sensor 14 of 0.5 are selected for an area 304 problematic for camera sensor 16. A weighting of camera sensor 16 of 1 [sic] is selected outside area 304.
Initially an object detection is carried out from the LIDAR data. The detected objects are subsequently shown to operator 36 as a bounding box 306 in first camera image 300.
In a first method step 601, the method according to the present invention is started. Vehicle 10 is controlled by operator 36. In a second method step 602, a situation and/or location reference of vehicle 10 is detected. Subsequently, data of a defined set of sensors 12 are transmitted in a third method step 603. The transmitted data are then fused in a fourth method step 604. The fused and processed data are then displayed to operator 36 in a fifth method step 605.
With the aid of method steps 602 through 605, a predictive map is used which may be updated during the control by operator 36. It is checked whether a problematic situation and/or a problematic location was recognized. A problematic situation and/or a problematic location may be recognized in a sixth method step 606 by observation of operator 36. A problematic situation and/or a problematic location may also, however, be recognized by marking by operator 36 in a seventh method step 607.
If a problematic situation and/or a problematic location are recognized, it and/or these are stored in an eighth method step 608 in a first database 630 for storing problematic situations and locations.
In a ninth method step 609, it is checked whether the trip is ended. If the trip is ended, the method is ended in a tenth method step 610. If vehicle 10 drives further, method steps 602 through 609 repeat.
In the creation of the predictive map, the detection, fusion, and display parameters are adapted if parameters are already present for the situation and/or location reference detected in second method step 602.
In an eleventh method step 611, the parameters for upcoming routes and/or areas are retrieved from a second database 640 for storing situation-related and/or location-related detection, fusion, and display parameters, which the predictive map displays. Subsequently, the defined set of sensors 12, whose data are transmitted, is adapted in a twelfth method step 612. The fusing of the data of sensors 12 is adapted in a thirteenth method step 613 and the display for operator 36 is adapted in a fourteenth method step 614.
If a problematic situation and/or a problematic location are recognized, an aggregation of data is carried out in parallel. The aggregation is started in a fifteenth method step 615 if a problematic situation and/or a problematic location are recognized.
A search is made for the recognized situations and/or locations in a sixteenth method step 616. Subsequently, in a seventeenth method step 617, an evaluation of identical situations and/or locations is compiled. In an eighteenth method step 618, it is then checked whether the recognized situation and/or the recognized location are permanently critical. Method steps 616 through 618 are repeated. If the recognized situation and/or the recognized location are permanently critical, it is furthermore checked in a nineteenth method step 619 whether parameters are already present. If the parameters are already present, they are retrieved in a twentieth method step 620 from second database 640 and taken into consideration when evaluating the recognized situation and/or the recognized location in a twenty-first method step 621. Subsequently, situation-adapted and/or location-adapted detection, fusion, and display parameters are generated in a twenty-second step 622, which are stored in a twenty-third method step 623 in second database 640. After the storage of the adapted parameters, the aggregation of data is ended in a twenty-fourth method step 624.
However, as already stated above in general, this selected sequence for carrying out the method according to the present invention in
The present invention is not restricted to the exemplary embodiments described here and the aspects highlighted therein. Rather, a variety of modifications are possible within the scope of the present invention, which are within the expertise of those skilled in the art.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10 2021 208 192.4 | Jul 2021 | DE | national |
