Patent Application
20240053744
This application claims priority to European Patent Application No. EP 21159108.6, filed on Feb. 24, 2021 with the European Patent Office. The contents of the aforesaid Patent Application are incorporated herein for all purposes.
The present disclosure is related to methods, computer programs, and apparatus for performing a tele-operated driving session for a vehicle equipped with an automated driving function. The disclosure is further related to a vehicle equipped with an automated driving function, which makes use of such a method or apparatus.
This background section is provided for the purpose of generally describing the context of the disclosure. Work of the presently named inventor(s), to the extent the work is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.
Tele-operated driving is gathering more and more interest. Tele-operated driving in the present context means that an external operator controls a vehicle remotely. The external operator is located in a control center. There may be a large distance between the control center and the vehicle. The control center and the vehicle are connected via a radio communication system and its backhaul. Primarily, the radio communication system is part of a public mobile communication system such as LTE (Long Term Evolution) or 5G.
Tele-operated driving belongs to safety-related time-critical applications. Main requirements for the exchange of information are low latency, high data rate, and high reliability.
Autonomous driving, also referred to as automatic driving, automated driving, or piloted driving, is the movement of vehicles, mobile robots and driverless transport systems that are largely autonomous. There are different degrees of autonomous driving. In Europe, various transport ministries, for example the Federal Institute for Road Systems (Bundesanstalt für Straßenwesen) in Germany, have defined the following autonomous stages:
A slightly different definition of levels is known from the Society of Automotive Engineers (SAE). In this regard, reference is made to the SAE J3016 standard. Such definitions could also be used instead of the above given definition.
Tele-operated driving might become a key technology in order to solve issues with Level 4 and Level 5 driven vehicles. A vehicle driving autonomously makes its decisions based on the perception of its environment as well as on predefined traffic regulations. However, it may happen that an autonomous vehicle is no longer able to continue its planned route, e.g., due to an incorrect interpretation of the environment, sensor failures, poor road conditions, or unclear traffic conditions, e.g., an accident or a construction site. In such situations, the vehicle needs external instructions from someone else to solve the situation, e.g., the external operator located in the control center. The vehicle will be driven remotely by the external operator during a tele-operated driving session until the vehicle can resume its autonomous driving operation.
To operate this remote control, data is exchanged through a cellular network. The quality of the uplink connection for the transmission of perception data and the downlink connection for the transmission of control data, also named quality of service (QoS), has a dramatic impact on the quality of application (QoA). The most important key performance indicators for tele-operations are latency and data rate of the communication network, e.g. a 4G network.
Two modes of tele-operated driving co-exist. A first mode is direct control, in which the control center controls the steering and throttle of the car. A second mode is indirect control, in which the control center gives high-level commands, such as waypoints or modifications of the environmental model. Direct control has higher requirements on the communication service, as the direct commands need to arrive as a real time stream at the vehicle under control.
During a tele-operated driving session with direct control, the quality of service may drop below and not meet the quality of service requirements anymore. In this case, the vehicle under control has to stop to put itself in a safe state. If the quality of service meets the requirements again at a later stage, the tele-operated driving session may continue. In some instances, it may even be necessary to start a new tele-operated driving session.
A need exists to provide improved solutions for performing a tele-operated driving session for a vehicle, which avoids an inefficient interruption of the tele-operated driving session. The need is addressed by the subject matter of the independent claims. Embodiments of the invention are described in the dependent claims, the following description, and the drawings.
The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description, drawings, and from the claims.
In the following description of embodiments of the invention, specific details are described in order to provide a thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the instant description.
According to some embodiments, a method for performing a tele-operated driving session for a vehicle equipped with an automated driving function comprises:
Accordingly and in some embodiments, a computer program comprises instructions, which, when executed by at least one processor, cause the at least one processor to perform the following for performing a tele-operated driving session for a vehicle equipped with an automated driving function:
The term computer has to be understood broadly. In particular, it also includes workstations, distributed systems and other processor-based data processing devices.
The computer program code can, for example, be made available for electronic retrieval or stored on a computer-readable storage medium.
According to some embodiments, an apparatus for performing a tele-operated driving session for a vehicle equipped with an automated driving function comprises:
From the point of view of the control center, use is made of a recent concept in communications, namely predictive quality of service. According to this concept, the future quality of service is estimated and made available as a basis for decisions or adaptations. The predictive quality of service is now used to determine a problematic drop in the quality of service. The quality of service can drop for multiple reasons, such as white spots in the coverage map, temporary channel overload, temporary network congestion, etc. These factors can be predicted by the communication service and provided within the predictive quality of service. Using this information, the control center can know in advance that during a specific part of the route driven during the tele-operated driving session direct control will not be possible. Therefore, the control mode, i.e., the mode of tele-operated driving, is temporarily changed from direct control to indirect control in order to enable the vehicle to pass through this unfavorable part of the route. To avoid any interruption of the tele-operated driving session, control data to be used by the vehicle in this section is provided to the vehicle before the section is reached.
In some embodiments, the section of the route for which an indirect control shall be used is determined by detecting a drop of the predictive quality of service below a threshold. For a tele-operated driving session with direct control, a defined minimum quality of service is needed. By detecting a drop in the quality of service below this defined minimum, a section that requires indirect control can easily be determined.
In some embodiments, the predictive quality of service is provided by the vehicle or by a communication network. The estimation of the quality of service can be performed at the vehicle or be provided by the communication network. The latter case has the benefit that the communication network has access to a larger amount of information, such as radio maps, instant radio resource utilization, historical utilization, etc.
In some embodiments, the control data comprises a set of waypoints or one or more trajectories. When the tele-operated driving session is switched to indirect control, the automated driving function of the vehicle has to make use of the control data to continue along the route. One solution for enabling the automated driving function to continue operation is to provide a trajectory that the vehicle shall follow. Another solution is to provide waypoints, which are located on a trajectory. The automated driving function then sequentially approach the various locations.
In some embodiments, trigger and stop times, trigger and stop positions, or switching signals are sent to the vehicle. Various possibilities exist for switching the control mode from direct control to indirect control and later back to direct control. For example, trigger and stop times may be used. The trigger time corresponds to the temporal beginning of the quality of service drop period. The stop time corresponds to the temporal end of the quality of service drop period. Alternatively, trigger and stop positions may be used. The trigger position corresponds to the spatial beginning of the quality of service drop section. The stop position corresponds to the spatial end of the quality of service drop section. A further possibility is to provide dedicated switching signals at the beginning and the end of the indirect control section.
According to some embodiments, a method for performing a tele-operated driving session for a vehicle equipped with an automated driving function comprises:
Accordingly and in some embodiments, a computer program comprises instructions, which, when executed by at least one processor, cause the at least one processor to perform the following for performing a tele-operated driving session for a vehicle equipped with an automated driving function:
The term computer has to be understood broadly. In particular, it also includes electronic control units, embedded devices and other processor-based data processing devices.
The computer program code can, for example, be made available for electronic retrieval or stored on a computer-readable storage medium.
According to some embodiments, an apparatus for performing a tele-operated driving session for a vehicle equipped with an automated driving function comprises:
From the point of view of the vehicle to be controlled, the vehicle receives information that during a specific part of the route driven during a tele-operated driving session direct control will not be possible. Therefore, the control mode, i.e., the mode of tele-operated driving, is temporarily changed from direct control to indirect control in order to enable the vehicle to pass through this unfavorable part of the route. To avoid any interruption of the tele-operated driving session, control data to be used by the vehicle in this section is received by the vehicle before the section is reached.
In some embodiments, the section of the route with indirect control is a section with a drop of a predictive quality of service below a threshold. For a tele-operated driving session with direct control, a defined minimum quality of service is needed. By detecting a drop in the quality of service below this defined minimum, a section that requires indirect control can easily be determined.
In some embodiments, the predictive quality of service is provided by the vehicle or by a communication network. The estimation of the quality of service can be performed at the vehicle or be provided by the communication network. The latter case has the benefit that the communication network has access to a larger amount of information, such as radio maps, instant radio resource utilization, historical utilization, etc.
In some embodiments, the control data comprises a set of waypoints or one or more trajectories. When the tele-operated driving session is switched to indirect control, the automated driving function of the vehicle makes use of the control data to continue along the route. One solution for enabling the automated driving function to continue operation is to provide a trajectory that the vehicle shall follow. Another solution is to provide waypoints, which are located on a trajectory. The automated driving function then sequentially approach the various locations.
In some embodiments, trigger and stop times, trigger and stop positions, or switching signals are received. Various possibilities exist for switching the control mode from direct control to indirect control and later back to direct control. For example, trigger and stop times may be used. The trigger time corresponds to the temporal beginning of the quality of service drop period. The stop time corresponds to the temporal end of the quality of service drop period. Alternatively, trigger and stop positions may be used. The trigger position corresponds to the spatial beginning of the quality of service drop section. The stop position corresponds to the spatial end of the quality of service drop section. A further possibility is to provide dedicated switching signals at the beginning and the end of the indirect control section.
In some embodiments, an autonomous or semi-autonomous vehicle comprises an apparatus according to the teachings herein or is configured to perform a method according to the teachings herein for performing a tele-operated driving session. In this way, the vehicle shows an improved behavior when a tele-operated driving session is subject to a drop in the quality of service. The vehicle may be any type of vehicle, e.g. a car, a bus, a motorcycle, a commercial vehicle, in particular a truck, an agricultural machinery, a construction machinery, a rail vehicle, etc. More generally, the teachings herein can be used in land vehicles, rail vehicles, watercrafts, and aircrafts. This expressively includes robots and drones.
Further features of the present invention will become apparent from the following description and the appended claims in conjunction with the FIGS.
The present description illustrates the principles of the present disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the disclosure.
All examples and conditional language recited herein are intended for educational purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions.
Specific references to components, process steps, and other elements are not intended to be limiting. Further, it is understood that like parts bear the same or similar reference numerals when referring to alternate FIGS.
Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.
Thus, for example, it will be appreciated by those skilled in the art that the diagrams presented herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure.
The functions of the various elements shown in the FIGS. may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, read only memory (ROM) for storing software, random access memory (RAM), and nonvolatile storage.
Other hardware, conventional and/or custom, may also be included. Similarly, any switches shown in the FIGS. are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.
In the claims hereof, any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a combination of circuit elements that performs that function or software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to perform the function. The disclosure as defined by such claims resides in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which the claims call for. It is thus regarded that any means that can provide those functionalities are equivalent to those shown herein.
The analyzing module 22, the processing module 23, and the communication module 24 may be controlled by a controller 25. A user interface 27 may be provided for enabling a user to modify settings of the analyzing module 22, the processing module 23, the communication module 24, or the controller 25. The various modules 22-25 can be embodied as dedicated hardware units. Of course, they may likewise be fully or partially combined into a single unit or implemented as software running on a processor, e.g. a CPU or a GPU.
A block diagram of a second embodiment of an apparatus 30 for use in a control center for performing a tele-operated driving session for a vehicle equipped with an automated driving function is illustrated in
The processing device 31 as used herein may include one or more processing units, such as microprocessors, digital signal processors, or a combination thereof.
The local storage unit 26 and the memory device 32 may include volatile and/or non-volatile memory regions and storage devices such as hard disk drives, optical drives, and/or solid-state memories.
The communication module 52 and the control module 53 may be controlled by a controller 54. A user interface 56 may be provided for enabling a user to modify settings of the communication module 52, the control module 53, or the controller 54. The various modules 52-54 can be embodied as dedicated hardware units. Of course, they may likewise be fully or partially combined into a single unit or implemented as software running on a processor, e.g. a CPU or a GPU.
A block diagram of a second embodiment of an apparatus 60 for use in a vehicle equipped with an automated driving function for performing a tele-operated driving session is illustrated in
The processing device 61 as used herein may include one or more processing units, such as microprocessors, digital signal processors, or a combination thereof.
The local storage unit 55 and the memory device 62 may include volatile and/or non-volatile memory regions and storage devices such as hard disk drives, optical drives, and/or solid-state memories.
In the following, another embodiment shall be explained in more detail with reference to
Such base station 210 may be an eNodeB (Evolved Node B) base station of an LTE mobile communication service provider or a gNB (Next Generation Node B) base station of a 5G mobile communication provider. The base station 210 and the corresponding equipment are part of a mobile communication network with a plurality of network cells, where each cell is served by one base station 210.
The base station 210 in
In terms of an LTE mobile communication system, the Evolved-UTRAN consists of a plurality of eNodeBs, providing the E-UTRA user plane protocol terminations, i.e., PDCP (Packet Data Convergence Protocol), RLC (Radio Link Control), MAC, (Medium Access Control), and PHY (Physical Layer), and the control plane protocol termination, i.e., RRC (Radio Resource Control) towards the user equipment. The eNodeBs are interconnected by means of the so-called X2 interface. The eNodeBs are also connected by means of the so-called S1 interface to an EPC (Evolved Packet Core) 200, more specifically to an MME (Mobility Management Entity) by means of an S1-MME interface and to a serving gateway by means of an S1-U interface.
In relation to this general architecture,
The various interfaces of the LTE network architecture are standardized. In this regard, reference is made to the various LTE specifications, which are publicly available for the sake of sufficiently disclosing further implementation details.
The vehicles 1, 2 may also be equipped with means for observing their surroundings. Their sensor systems, which are used to capture the environmental objects, are based on different measuring methods, depending on the application. Widespread technologies are, among others, RADAR, LIDAR, cameras for 2D and 3D image acquisition, and ultrasonic sensors.
Since automated driving is on the rise, a lot more data needs to be exchanged among the vehicles 1, 2, e.g. using V2V communication links PC5, and also between the vehicles 1, 2 and the network. The communication systems for V2V and V2X communication need to be adapted correspondingly. The 3GPP standard setting organization has been and is releasing features for the new generation of the 5G cellular mobile communication system, including V2X features. A large panel of vehicular use cases have been designed, ranging from infotainment to cooperative driving. Depending on the application, the requirement on the access link Uu in the scope of V2N communication drastically changes. When it comes to safety-related time-critical applications such as tele-operated driving, in which a command center takes over certain driving functions of the vehicle, these requirements are the exchange of information with low latency, high data rate and high reliability.
The memory device 80 is connected to the computing device 67 via a data line 85. In the memory device 80, a pictogram directory and/or symbol directory is deposited with pictograms and/or symbols for possible overlays of additional information.
The other parts of the infotainment system, such as a camera 150, radio 140, navigation device 130, telephone 120 and instrument cluster 110 are connected via a data bus 100 with the computing device 60. As data bus 100, the high-speed variant of the CAN (Controller Area Network) bus according to ISO standard 11898-2 may be used. Alternatively, an Ethernet-based bus system such as IEEE 802.03cg can be used. Bus systems implementing the data transmission via optical fibers are also usable. Examples are the MOST Bus (Media Oriented System Transport) or the D2B Bus (Domestic Digital Bus). For inbound and outbound wireless communication, the vehicle is equipped with an on-board connectivity module 160. It can be used for mobile communication, e.g., mobile communication according to the 5G standard.
Reference numeral 172 denotes an engine control unit. Reference numeral 174 denotes an ESC (electronic stability control) unit, whereas reference numeral 176 denotes a transmission control unit. The networking of such control units, all of which are allocated to the category of the drive train, typically occurs with a CAN bus 104. Since various sensors are installed in the motor vehicle and these are no longer only connected to individual control units, such sensor data are also distributed via the bus system 104 to the individual control devices.
Modern vehicles may comprise additional components, such as further sensors for scanning the surroundings, like a LIDAR sensor 186 or a RADAR sensor 182 and additional video cameras 151, e.g., a front camera, a rear camera or side cameras. Such sensors are increasingly used in vehicles for observation of the environment. Further control devices, such as an ADC (automatic driving control) unit 184, etc., may be provided in the vehicle. The RADAR and LIDAR sensors 182, 186 may have a scanning range of up to 250 m, whereas the cameras 150, 151 may cover a range from 30 m to 120 m. The components 182 to 186 are connected to another communication bus 102, e.g. an Ethernet-Bus due to its higher bandwidth for data transport. One Ethernet-bus adapted to the special needs of car communication is standardized in the IEEE 802.1Q specification. Moreover, a lot of information about the environment may be received via V2V communication from other vehicles. Particularly for those vehicles that are not in line of sight to the observing vehicle, it is very beneficial to receive the information about their position and motion via V2V communication.
Reference numeral 190 denotes an on-board diagnosis interface, which is connected to another communication bus 106.
For the purpose of transmitting the vehicle-relevant sensor data via the an on-board connectivity module 160 to another vehicle or to a control center computer, a gateway 90 is provided. This gateway 90 is connected to the different bus systems 100, 102, 104 and 106. The gateway 90 is adapted to convert the data it receives via one bus to the transmission format of another bus so that it can be distributed using the packets specified for the respective other bus. For forwarding this data to the outside, i.e. to another vehicle or to the control central computer, the an on-board connectivity module 160 is equipped with a communication interface to receive these data packets and, in turn, to convert them into the transmission format of the appropriate mobile radio standard.
The invention has been described in the preceding using various exemplary embodiments. Other variations to the disclosed embodiments may be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor, module, or other unit or device may fulfill the functions of several items recited in the claims.
The term “exemplary” used throughout the specification means “serving as an example, instance, or exemplification” and does not mean “preferred” or “having advantages” over other embodiments. The terms “in particular” and “particularly” used throughout the specification means “for example” or “for instance”.
The mere fact that certain measures are recited in mutually different dependent claims or embodiments does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.
| Number | Date | Country | Kind |
|---|---|---|---|
| 21159108.6 | Feb 2021 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2022/054388 | 2/22/2022 | WO |
