This application claims priority to EP App. No. 22 199 861 filed Oct. 5, 2022, the entire disclosure of which is incorporated by reference.
The present disclosure relates to camera-based mechanisms contributing to safety and automated control in a vehicle and more particularly to systems for vehicle exit assistance.
Vehicle exit situations may involve danger that approaching other vehicles such as motorcycles or bicycles may collide with the door, such collisions potentially causing damage to persons and the vehicles.
Such, generally, safety and assistance systems are sought which help to reduce such risks.
EP 3 594 035 (B1) describes a method and vehicle including at least one door, moveable between an open and closed position and including at least one inside door handle for opening the door, and a door opening warning system including at least one object sensor configured to detect an object outside the vehicle in a risk zone in which there is a risk that the object collides with the door if it is moved from the closed position to the open position, a touchless sensor configured to detect if a part of a user is positioned in an opening zone at the inside door handle and a touch sensor, such as an capacitive sensor, configured to detect if the user touches the inside door handle, and a control unit connected to the at least one object sensor, the touchless sensor and the touch sensor, wherein the control unit of the door opening warning system is configured to: trigger a first alarm if the object sensor detects an object in the risk zone and the touchless sensor detects a part of the user in the opening zone at the inside door handle, and trigger a second alarm if the object sensor detects an object in the risk zone and the touch sensor detects that the user is touching the inside door handle.
The background description provided here is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.
The present disclosure seeks to provide improvements to such existing recognition mechanisms. The present methodologies are generally directed to improve systems for vehicle exit assistance. The main aspects are set out by the independent claims.
In this respect, according to a first aspect a computer-implemented method for vehicle exit assistance is provided. The method includes analyzing interior cabin data originating from a vehicle sensing system to detect a position and/or movement of a person in a cabin of the vehicle. The method further includes determining, based on the detected position and/or movement of the person, a door opening act indicative of an intention of the person to exit the vehicle, determining, based on exterior environment data collected from the vehicle sensing system, an object in a given vicinity of the vehicle, determining a vehicle exit score based on the object and the door opening act, and, based on the determined vehicle exit score, outputting a control signal to a vehicle assistance system to control vehicle exit of the person.
In another aspect, a system for vehicle exit assistance is provided, including a vehicle sensing system, a data processing system, interfaces for outputting a control signal, configured to perform the computer-implemented methods as described herein.
In another aspect, a vehicle is provided including a system for vehicle exit assistance as described herein.
Finally, a computer program is presented that includes instructions which, when the program is executed by a computer, cause the computer to carry out the methods described herein.
Further refinements are set forth by the dependent claims.
Further areas of applicability of the present disclosure will become apparent from the detailed description, the claims, and the drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.
The present disclosure will become more fully understood from the detailed description and the accompanying drawings.
Aspects and examples of the present disclosure are described with reference to the following figures, in which:
In the drawings, reference numbers may be reused to identify similar and/or identical elements.
The present disclosure relates to an improvement of systems for vehicle exit assistance, especially to prevent door accidents, also commonly referred to as dooring, when a person departs a vehicle.
As an example,
One standard behavior to mitigate such risk of a door accident is for the person in first vehicle 1 to look out of the windows of the first vehicle 1 and check if any vehicles or pedestrians approach vehicle 1. This can be facilitated by using mirrors, such as a side mirror 5 or a rear mirror. However, this relies on the discipline of the person to actually perform these actions with all due care every time he/she intends to open the door 2 of the vehicle 1. Human nature, however, is prone to negligence, especially when for a certain amount of time no accident has happened, or the persons are in a hurry or occupied with other thoughts than traffic safety when intending to open the vehicle door to depart the vehicle 1.
In this way, the danger is reduced that the vehicle door 2 is opened in moments when a hazardous situation may occur, in a way by providing an assistance for checking the situation around the vehicle when the person intents to exit the vehicle. It does not depend on the person alone anymore to carefully check the vicinity of the vehicle and the traffic situation around the vehicle before opening the door. The vehicle, as an additional instance, also performs this task of checking its vicinity and the traffic situation. If the person does not check the situations around the vehicle carefully, such as due to being in a hurry, due to negligence or just by forgetting it, there is still the vehicle sensing system as the second instance to perform these checks.
The vehicle exit score may indicate a risk level of the vehicle exit. For example, a higher vehicle exit score may indicate a higher risk level of the vehicle exit and a lower vehicle exit score may indicate a lower risk level of the vehicle exit. In some embodiments, the vehicle exit score may be a number, e.g. a normalized number within a given range, e.g. between 0 and 100, with 0 indicating no risk of the vehicle exit (e.g. because no relevant object outside of the vehicle at all has been detected) and 100 indicating maximum risk. In some embodiments, the vehicle exit score may be a more complex figure, e.g. a feature vector, a vector indicating multiple characteristics incorporated by the vehicle exit score (e.g. characteristics of the determined object as explained in more detail below), a vector indicating multiple weighted characteristics incorporated by the vehicle exit score, an aggregated number plus the more detailed vector, and so on.
In addition, the method as described in the present disclosure idea ensures sufficient response times. The response times are short enough, so that it is possible to give e. g. a warning signal to the person intending to open a door of the vehicle 1 and/or locking the corresponding door before an object approaching the vehicle 1 reaches the door and collides with that door. At the same time, the present methodologies render the entire system user-friendly as the is not overly intrusive and output warning signals or takes safety measures if the risk of the vehicle exit is determined to be significant enough.
In some embodiments, the vehicle sensing system includes a vehicle interior sensing system and a vehicle exterior sensing system and wherein the interior cabin data originates from the vehicle-interior sensing system and the exterior environment data originates from the vehicle-exterior sensing system. The sensors of the vehicle sensing system begin to operate continuously immediately after the electric power supply system of the vehicle 1 starts to operate. The sensors detecting a position and/or movement of a person in a cabin of the vehicle may include 2D and/or 3D sensors. The sensors for the exterior environment of the vehicle 1 may sense a range up to at least 30 meters or even up to 100 meters.
As an example,
In some embodiments, the vehicle interior sensing system 6 and/or the vehicle exterior sensing system includes one or more camera systems. In some embodiments, the one or more camera systems include one or more infrared cameras or one or more cameras operating in the visible light spectrum. In some embodiments, the vehicle interior sensing system 6 includes sensors, such as cameras located at different points in the cabin of vehicle 1. In some embodiments, the vehicle exterior sensing system includes sensors, such as cameras, at different exterior positions of the vehicle 1. In some embodiments, the vehicle exterior sensing system include a radar system. In some embodiments, the vehicle interior system 6 includes a radar system, such as a 60 GHz MIMO radar system for the detection of the presence of persons inside the cabin of the vehicle 1 and the movements of the persons.
In some embodiments, a control signal is outputted in response to determining that the door opening act is about to occur and that an object in the given vicinity of the vehicle is detected.
The determination whether of a door opening act may include analyzing the current velocity of the vehicle 1. In the case the vehicle is proceeding at a usual driving speed, e. g. 50 km/h, it is unlikely that a person in the vehicle 1 intents to leave the vehicle and therefore to open a vehicle door. In some embodiments, including outputting the control signal in response to that data on the current velocity of the vehicle indicates that the current velocity of the vehicle is under a given threshold. As an example,
In some embodiments, the determination of a door opening act includes the determination of a number of body keypoints indicative of the location of defined body portions of the person.
As an example,
Hence,
In some embodiments, the determination of a door opening act includes the determination whether the person has unfastened his/her seatbelt, and/or based on the interior cabin data, whether the head and/or the eyes of the person gaze towards the door.
As an example,
In some embodiments, determining of a door opening act may take into account the geometry and functionality of the seat belt and the way the seat belt is fastened and unfastened, especially the type of fasteners which fasten the seat belt and the types of operations a person has to perform e. g. with his hand and fingers to operate a specific type of fastener. The vehicle sensing system therefore also detects the types of the seat belts, especially their types of fasteners and the movement of the hands and the fingers of the person when operating the seat belt and its fastener. In some embodiments, the vehicle sensing system detects seatbelt-fastener for the front seats of vehicle 1 and/or seatbelt-fastener for the back seats of vehicle 1. In some embodiments, the vehicle sensing system detects seatbelt fastener for child seats.
As an example,
In some embodiments, when the head and/or the eyes of the person gazes towards the door of the vehicle, the tracking of the movement of the person is continued until a determined probability of a door opening act to occur exceeds a given probability threshold. This reduces the probability to erroneously detect an intention of a door opening and, accordingly, that e. g. a false alarm is triggered.
In some embodiments the determination of a door opening act includes tracking of one or more shoulder keypoints and/or hand keypoints and/or elbow keypoints.
The tracking of body keypoints and/or the determination whether the person has unfastened his/her seatbelt enables the system to more reliably determine whether a door opening act to about to occur.
As example,
In some embodiments, the determination of a door opening act includes tracking of any body keypoints of any body portion of the persons of vehicle 1.
In some embodiments, the determination of the probability of a door opening act includes detecting areas surrounding a hand of the person and a handle of the door 2 of vehicle 1 and calculating the distance and/or the overlap between the hand and the handle. In further embodiments, any shape for the detecting areas are included. In some further embodiments, the areas include a volume of any spatial dimension. In some embodiments, the distance includes any line connection a point within the detecting area surrounding the hand and a point within the detecting area surrounding the handle. In some embodiments the overlay includes any area or volume with are part of the detecting area surrounding the hand and the detecting area surrounding the handle. This further enables the system to more reliably determine a door opening act.
In some embodiments, the determination 12 of the vehicle exit score is based on the characteristics of the vehicle 1 and/or the characteristics of the object in the vicinity of the vehicle 1. The characteristics of the vehicle 1 may include the shape and/or the size of the vehicle, such as the vehicle 1 being a car, a bus, a delivery car or a truck. For example, a bus 1 has usually doors 2 only on one side, normally the side pointing towards the pedestrian line. When a bus 1 stops at a bus stop and the bus driver operates the door opening system, causing the bus doors 2 to open, usually there are always persons, normally the passengers wishing to board the bus 1, standing directly in front of the opening doors 2. This situation is different compared to a car 1, where normally there should not any person stand in front of door 2 when a person inside the cabin intend to open that door 2. These passengers should not influence the determination of the vehicle exit score. Therefore, for a bus 1, the determination 12 of the vehicle exits score may be carried out in a different way than for a passenger car 1.
The characteristics of the object in the vicinity of the vehicle 1 may also influence the determination 12 of the vehicle exit score. More specifically, the vehicle exit score may be determined considering particular recognized details of the object such as the geographically distance between the vehicle and the object, the velocity of the object, the movement direction (if any) of the object, the size and/or shape of the object, an identified type of the object such as the object being a human, a vehicle, a static object (e.g. a garbage bin standing on the footway). For example, the object may be a pedestrian moving at a typical walking speed, a bicycle or a motorbike or another car. The walking pedestrian walking at around 5 km/h usually has sufficient time to react on a door opening act and will avoid the opening door in time. Even a collision with that opening door 2 usually happens at low speed, so it will be unlikely that the pedestrian will suffer serious injuries. Riders of bicycles or motorbikes are, on the contrast much more vulnerable to a dooring accident, since they proceed at a much larger speed and their bodies are usually directly involved in a collision. The risk of severe injuries is therefore much higher for riders of bicycles and motor bikes. The determination 12 of the vehicle exit score may therefore different for a pedestrian and a rider of a bicycle or motorbike, e.g. lower if the object is determined to be pedestrian and higher if the object is determined to be a bicycle. The vehicle exit score may be determined to be lower if the object is farther away and to be higher if the object is closer to the vehicle. The vehicle exit score may be determined to be lower if the object is determined to be moving away from the vehicle and higher if the object is determined to be moving towards the vehicle, etc. Generally, the higher the determined object characteristics and/or vehicle characteristics indicate the risk of vehicle and/person damages potentially caused by the vehicle exit, the higher the vehicle exit score may be set.
The determination 12 of the vehicle exit score may include aggregated weights of the components included in the determination 12, such as the positions and/or movements of the persons inside the cabin of the vehicle 1, the detections of objects in the vicinity of the vehicle 1, the characteristics of the vehicle 1 and/or the objects. The components may be include sensor input form the vehicle sensing system such a inputs from cameras, radars etc.
The vehicle exit score may be compared to one or multiple thresholds in order to determine if and when a control signal is outputted. In some embodiments the vehicle exit score may include several thresholds defining e. g. the intensity of a warning signal the control signal causes to output. As an example, there may be a lower threshold for the vehicle exit score, at which a low impact warning signal is outputted, such in a case when a walking pedestrian is detected in the vicinity of the vehicle. A middle and a high threshold may be additionally defined for the outputting of higher impact warning signals such as for the detection of bicycle or motorbike riders or another vehicles. Hence, a lower vehicle exit score may exceed the low impact control signal threshold, causing outputting a low impact warning signal, causing, in turn, a visual and/or audio warning signal to the person. On the other hand, a higher vehicle exit score may not only exceed the low impact control signal threshold, but also the higher impact control signal threshold, causing the output of a more critical control signal which causes, in turn, temporarily locking the door until, e.g. an updated vehicle exit score is determined in a subsequent iteration of the present procedure indicating that the damage risk of the vehicle exit has decreased (e.g. because the bicycle which has previously approached the vehicle has then passed the vehicle moves away from the vehicle).
In some embodiments, the outputted control signal may be a raw control signal indicating the vehicle exit score, and further processing such as comparing the raw control signal against the one or more thresholds and determining a current criticality level of the vehicle exit may be performed by a further module such as a vehicle control system. In some embodiments, the outputted control signal is a comprehensive signal also indicating a determined criticality level of the vehicle exit. Combinations of both, i.e. the control signal indicating the vehicle exit score and a determined criticality level of the vehicle exit are envisaged as well.
The determination 12 of vehicle exit score may be carried out using deterministic mathematical models taking into account the various components to be included in the determination 12. The model may be defined by the various sensor of the vehicle sensing system, such as the cameras and/or the radars inside the vehicle cabin and the cameras and/or radars sensing the vicinity of the vehicle 1. In some embodiments, the determination 12 of the vehicle exit score may be carried out using machine learning methods outputting the vehicle exit score, such as neural networks and/or supervised learning and/or unsupervised learning and/or reinforcement learning and/or decision trees.
In some embodiments, the vehicle assistance system, in response to the control signal to the vehicle assistance system, locks the door of the vehicle and/or outputs a warning signal.
The vehicle assistance system, in response to the control signal to the vehicle assistance system, may perform a vehicle safety function. The vehicle security function includes one or more actions which facilitate the safety of the vehicle and persons in the vehicle. For example, the vehicle security function includes locking a door of the vehicle. In some embodiments, the vehicle assistance system locks the door 2 which has been detected to be about to be opened by the person.
In some embodiments, the locking of the door can be manually overridden by a person in the vehicle 1. This enhances the safety for the persons inside the vehicle 1. In the case of a danger threatening the persons inside the vehicle 1, e. g. smoke or fire in the cabin, the persons are able to exit the vehicle 1 anyway, regardless whether the vehicle assistance system has previously locked the doors. Alternatively, the door locking may only be temporarily, e.g., 5 seconds, so that an emergency exit is possible after a short time period.
In addition or alternative to the door locking of
In some embodiments, the warning signal includes an audible warning and/or a visual warning and/or displaying a map of the vicinity of the vehicle and the object in the vicinity of the vehicle.
The outputting of the audible warning and/or a visual warning signal is shown in activity 17 of
In some embodiments, the warning signal includes displaying a warning information on a vehicle infotainment system. As an example,
In some embodiments the warning information 32 displayed on the vehicle infotainment system 30 include a symbol usually associated with imminent danger, such as a red flashlight or a corresponding traffic sign, such as an exclamation mark within a red triangle. The warning information 32 may also include written information e. g. the expression: “DANGER!” and/or “DO NOT OPEN DOOR!”. In some embodiments symbol based and written warning information 32 are combined. This provides an additional possibility to enhance the probability that the person intending to open the door of the vehicle realizes that an object is in the given vicinity of the vehicle and the vehicle door is not be opened, thereby strengthening the warning effectivity of the vehicle assistance system.
The warning information 32 may be outputted on the infotainment system in combination with an audible warning signal and/or other visible warning signals.
The outputting of the warning signal may cause displaying a map of the vicinity of the vehicle and the object in the vicinity of the vehicle.
In some further embodiments, the map is displayed on a vehicle head-up display.
In some embodiments, the determining of the door opening act and/or the object in the vicinity of the vehicle utilizes machine learning based methods. In some further embodiments, the machine learning based methods include neural networks and/or supervised learning and/or unsupervised learning and/or reinforcement learning and/or decision trees.
According an aspect, a system 100 for vehicle exit assistance is provided, including, as shown in
Furthermore, the data processing system 200 may also include a specified sensing interface 204 to communicate with vehicle sensing system 101 of the vehicle 1. Alternatively, the data processing system 200 may communicate with the vehicle sensing system 101 via the network interface 203. The vehicle sensing system 101 is used for generating interior cabin data to detect a position and/or movement of a person in a cabin of the vehicle 1 and exterior environment data to detect an object in a given vicinity of the vehicle 1. The data processing system 200 may also be connected to database systems (not shown) via the network interface, wherein the database systems store at least part of the images needed for providing the functionalities described herein.
The main memory 206 may be a random-access memory (RAM) and/or any further volatile memory. The main memory 206 may store program code for the vehicle sensing system control 208 and the determination of a door opening 209. The memory 206 may also store additional program data required for providing the functionalities described herein. Part of the program data 210, the determination of a door opening 209 and/or the vehicle sensing system control 208 may also be stored in a separate, e.g., cloud memory and executed at least in part remotely. In such an embodiment, the memory 206 may store the current occupancy and seat states according to the methods describes herein in a cache 211.
According to an aspect, as shown in
According to an aspect, a computer program including instructions is provided. These instructions, when the program is executed by a computer, cause the computer to carry out the methods described herein. The program code embodied in any of the systems described herein is capable of being individually or collectively distributed as a program product in a variety of different forms. In particular, the program code may be distributed using a computer readable storage medium having computer readable program instructions thereon for causing a processor to carry out aspects of the embodiments described herein.
Computer readable storage media, which are inherently non-transitory, may include volatile and non-volatile, and removable and non-removable tangible media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Computer readable storage media may further include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid state memory technology, portable compact disc read-only memory (CD-ROM), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and which can be read by a computer.
A computer readable storage medium should not be construed as transitory signals per se (e.g., radio waves or other propagating electromagnetic waves, electromagnetic waves propagating through a transmission media such as a waveguide, or electrical signals transmitted through a wire). Computer readable program instructions may be downloaded to a computer, another type of programmable data processing apparatus, or another device from a computer readable storage medium or to an external computer or external storage device via a network.
It should be appreciated that while particular embodiments and variations have been described herein, further modifications and alternatives will be apparent to persons skilled in the relevant arts. In particular, the examples are offered by way of illustrating the principles, and to provide a number of specific methods and arrangements for putting those principles into effect.
In certain embodiments, the functions and/or acts specified in the flowcharts, sequence diagrams, and/or block diagrams may be re-ordered, processed serially, and/or processed concurrently without departing from the scope of the invention. Moreover, any of the flowcharts, sequence diagrams, and/or block diagrams may include more or fewer blocks than those illustrated consistent with embodiments of the invention.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of the disclosure. It will be further understood that the terms “comprise” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. Furthermore, to the extent that the terms “include”, “having”, “has”, “with”, “comprised of”, or variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term “comprising”.
While a description of various embodiments has illustrated all of the inventions and while these embodiments have been described in considerable detail, it is not the intention of the applicants to restrict or in any way limit the scope of the appended claims to such detail. Additional advantages and modifications will readily appear to those skilled in the art. The invention in its broader aspects is therefore not limited to the specific details, representative apparatus and method, and illustrative examples shown and described. Accordingly, the described embodiments should be understood as being provided by way of example, for the purpose of teaching the general features and principles, but should not be understood as limiting the scope, which is as defined in the appended claims.
The term non-transitory computer-readable medium does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave). Non-limiting examples of a non-transitory computer-readable medium are nonvolatile memory circuits (such as a flash memory circuit, an erasable programmable read-only memory circuit, or a mask read-only memory circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).
The phrase “at least one of A, B, and C” should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.” The phrase “at least one of A, B, or C” should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR.
Number | Date | Country | Kind |
---|---|---|---|
22199861 | Oct 2022 | EP | regional |