Patent Grant
12014448
This application claims priority to German Patent Application No. DE 10 2019 206 490.6, filed on May 6, 2019 with the German Patent and Trademark Office. The contents of the aforesaid Patent Application are incorporated herein for all purposes.
The invention relates to a park assistance system for a motor vehicle having a distance sensor system to generate a sensor signal depending on a lateral distance between the motor vehicle and an object in an environment of the motor vehicle, and a computing unit to generate graphics data based on the sensor signal. The invention furthermore relates to a method for helping with parking a motor vehicle, a computer program, and a computer-readable storage medium.
This background section is provided for the purpose of generally describing the context of the disclosure. Work of the presently named inventor(s), to the extent the work is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.
When helping a motor vehicle with parking, ultrasonic sensors, optical sensors, or cameras are used. In known park assistance systems, the spacing information is displayed in the form of lateral spacing indicators on an infotainment system of the motor vehicle. The driver of the motor vehicle must therefore remove his gaze from the road and direct it to the display within the automobile. In addition, the driver must interpret the actual distance between the motor vehicle and a potential obstacle based on the displayed information, which may lead to misestimations. Both circumstances lead to an endangerment of the driver's safety.
A need exists to provide improved systems and methods for parking a motor vehicle by means of which safety for a driver of the motor vehicle is increased.
The need is addressed by a park assistance system, a method for helping with parking, a computer program, as well as a computer-readable storage medium according to the independent claims. Embodiments of the invention are described in the dependent claims, the following description, and the drawings.
The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description, drawings, and from the claims.
In the following description of embodiments of the invention, specific details are described in order to provide a thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the instant description.
The teachings herein are based on the idea of superimposing a graphic element by means of an augmented reality system onto a field of view of the driver so that an extension of the graphic element along a vertical axis depends on a value of a lateral distance between the motor vehicle and an object in the environment of the motor vehicle.
In a first exemplary aspect, a park assistance system is presented for a motor vehicle that has a distance sensor system and a computing unit. The distance sensor system is configured to generate a sensor signal depending on a lateral distance between the motor vehicle and an object in an environment of the motor vehicle. The computing unit is configured to generate graphics data based on the sensor signal. The park assistance system moreover contains an augmented reality system (AR system). The AR system is configured to superimpose a graphic element onto a field of view of a driver of the motor vehicle, in particular to project it onto a display element of the AR system or the motor vehicle, or to display it on the display element. An extension of the graphic element along a vertical axis depends on a value of the distance.
The lateral distance may in particular be a spacing or distance within a plane parallel to a plane of movement of the motor vehicle, wherein the plane of movement of the motor vehicle corresponds to a plane within which the motor vehicle may move, i.e., in particular parallel to a plane of a surface on which the motor vehicle is moving. The motor vehicle has in particular a longitudinal axis and a transverse axis. The plane of movement of the motor vehicle is then defined in that the longitudinal and transverse direction of the motor vehicle lie within the plane of movement.
The distance sensor system may for example contain a camera system, an active optical sensor system or an ultrasonic sensor system, in particular an active ultrasonic sensor system. The active optical sensor system is for example configured to emit light and to detect reflected components of the light, and to determine the lateral distance based on the reflected components. Analogously, the ultrasonic sensor system may for example be configured to send ultrasonic waves into the environment, to detect reflected components of the ultrasonic waves, and to determine the lateral distance based thereupon.
The graphics data include for example data or information present in a suitable form so that the computing unit, or respectively the AR system, may generate the graphic element based on the graphic data and superimpose it onto the field of view, in particular depict it on the display element. The graphics data may for example be in the form of a graphics file, or in another suitable form.
The field of view of the driver is in particular a spatial region that the driver may visually perceive. Depending on the embodiment of the AR system, the field of view may correspond to an actual field of view of the driver, i.e., a spatial region that the driver may actually perceive visually independent on the direction in which, for example, his head has rotated. Alternatively in other embodiments, the field of view may correspond to a nominal field of view of the driver, i.e., a field of view of the driver when he has oriented his head in a given, in particular nominal direction and directs his gaze in this direction. The nominal direction may for example correspond to a direction from a given position within the cockpit through the windshield of the motor vehicle toward the road, or respectively into the environment of the motor vehicle.
The AR system contains for example all of the components that are needed to generate the graphic element and to superimpose the graphic element onto the field of view of the driver based on the graphics data. In particular, the AR system may also contain the display element. For example, the computing unit may also be part of the AR system.
The display element may for example correspond to a windshield of the motor vehicle, or to a part of the windshield. Alternatively, the display element may also be designed as a separate display or separate display surface of the AR system.
For example, the AR system may be designed as a head-up display or as a head-mounted display.
An augmented reality system or AR system may be understood to be a system by means of which images of the real environment detected by the driver are supplemented by computer-generated additional information or virtual objects, in particular by the graphic element, by means of superimposing or overlaying.
The extension of the graphic element may in particular be a length of the graphic element along the vertical axis.
The vertical axis is in particular an axis that runs vertically from the perspective of the driver, i.e., from bottom to top or vice versa in the field of view of the driver, in particular when the driver has directed his gaze to the graphic element or the display. In particular, the vertical axis corresponds to an axis that stands upright on the longitudinal or transverse axis of the motor vehicle when the driver gazes in the driving direction, or in a direction parallel to the plane of movement of the motor vehicle.
In particular, the direction of the vertical axis deviates from a connecting axis between the distance sensor system and the object.
The graphic element may in particular be an abstract graphic element, i.e., in particular not a depiction of a real existing object.
With a park assistance system according to the teachings herein, the driver does not have to remove his gaze from the road in order to be able to detect the graphic element, or respectively its extension along the vertical axis, and accordingly a value of the distance between the motor vehicle and the object. This is possible by superimposing the graphic element onto the field of view of the driver. This improves the safety of the driver.
Since the graphic element is in particular an abstract graphic element, a meaning of the graphic element does not have to be explicitly understood. In particular in comparison to a camera image, a detailed interpretation of the graphic element is unnecessary since the extension of the graphic element as an indicator of the value of the distance may be intuitively detected by the driver, in particular after a certain familiarization phase. In contrast, an actual spacing from a camera image of the environment needs to be interpreted and may be difficult to estimate.
Since the extension of the graphic element along the vertical axis is used as an indicator of the lateral distance, the visual axis is effectively tilted out of the plane of movement of the motor vehicle. In particular, this allows a close range in an immediate environment of the distance sensor system to be completely depicted. This is in particular beneficial because AR systems such as head-up displays or head-mounted displays may have a minimum distance, wherein objects within the minimum distance cannot be depicted. The region of the minimum distance may for example end a few meters from the motor vehicle in head-up displays so that, in particular, a close range of the environment of the motor vehicle may also be intuitively and reliably depicted by means of the teachings herein.
In some embodiments of the park assistance system, the extension of the graphic element along the vertical axis is all the more smaller the smaller the value of the lateral distance is.
In some embodiments of the park assistance system, the AR system is configured to superimpose another graphic element onto the field of view of the driver, in particular to depict it on the display element or project it onto the display element. An extension of the additional graphic element along the vertical axis also depends on the value of the distance. A spacing between the graphic element and the additional graphic element depends on the value of the distance.
In particular, the spacing between the graphic element and the additional graphic element may be all the more smaller the smaller the value of the distance is.
In some embodiments, the graphic element and the additional graphic element may be polygons such as rectangles or trapezoids. The extension of the polygons along the vertical axis then corresponds to a particular side length or height of the polygon.
In particular, the extension of the graphic element along the vertical axis may be the same as the extension of the additional graphic element along the vertical axis.
By superimposing the additional graphic element and scaling the spacing between the graphic elements and the value of the distance, a more intuitive and therefore safer depiction may be enabled.
In some embodiments, the AR system is configured to superimpose the graphic element and, if applicable, the additional graphic element onto the field of view according to a given color scheme such that the color of the graphic element and, if applicable, the additional graphic element depends on a value of the distance.
By depicting the graphic elements according to the color scheme, an even more intuitive and therefore even safer representation of the lateral distance may be achieved.
In some embodiments, the AR system contains a head-up display system for superimposing the graphic element, and if applicable the additional element, onto the field of view.
In such embodiments, in particular the windshield of the motor vehicle may serve as a display element.
In particular when a head-up display system is used, the advantage of the teachings herein is particularly evident since, in a lateral depiction of the distance by means of a head-up display, a region of around 5 to 7 m directly in front of the motor vehicle cannot be imaged due to a corresponding blind spot of the head-up display. For head-up displays, use as a park assistance system is only meaningfully possible by tilting the visual axis out of the lateral plane into the vertical direction according to the present aspect.
In some embodiments, the AR system contains a visual output unit that may be worn on one's head for superimposing the graphic element, and in particular the additional graphic element, onto the field of view.
The visual output unit that may be worn on one's head (head-mounted display, HMD) is in particular a so-called AR glasses system, also termed data glasses or smart glasses.
In such embodiments, for example a display element that may be fastened to the head of the driver serves as a display element for superimposing the graphic elements.
According to another exemplary aspect, a method is presented for helping with parking a motor vehicle, wherein a sensor system is generated by means of a distance sensor system of the motor vehicle depending on a lateral distance between the motor vehicle and an object in an environment of the motor vehicle. By means of a computing unit, graphic data are generated based on the sensor system. A graphic element is superimposed by means of an augmented reality system of the motor vehicle onto a field of view of a driver of the motor vehicle, wherein the graphic element is superimposed onto the field of view such that its extension along a vertical axis depends on a value of the distance.
In some embodiments of the method for helping with parking, another graphic element is superimposed onto the field of view by means of the augmented reality system. An extension of the additional graphic element along the vertical axis depends on the value of the distance, and a spacing between the graphic element and the additional graphic element depends on the value of the distance.
Other embodiments of the method for helping with parking follow directly from the different embodiments of the park assistance system according to the teachings herein, and vice versa. In particular, a park assistance system according to the teachings herein may be configured or programmed to perform a method for helping with parking according to the teachings herein, or a park assistance system according to the teachings herein performs a method according to the teachings herein.
According to another exemplary aspect, a motor vehicle having a park assistance system according to the teachings herein is presented.
According to another exemplary aspect, a computer program with commands is presented, wherein when the computer program is run by a park assistance system according to the teachings herein, in particular by a computing unit of a park assist system, the commands of the computer program cause the park assistance system to perform a method according to the teachings herein.
According to another exemplary aspect, a computer-readable storage medium is presented on which a computer program according to the teachings herein is saved.
Also belonging to the invention are embodiments of the method that have features which have already been described in conjunction with the embodiments of the park assistance system. For this reason, the corresponding embodiments of the method will not again be described.
The invention also includes combinations of the features of the described embodiments.
In the following, further exemplary embodiments of the invention are described.
In the exemplary embodiments described herein, the described components of the embodiments each represent individual features that are to be considered independent of one another, in the combination as shown or described, and in combinations other than shown or described. In addition, the described embodiments may also be supplemented by features of the invention other than those described.
Specific references to components, process steps, and other elements are not intended to be limiting. Further, it is understood that like parts bear the same or similar reference numerals when referring to alternate FIGS. It is further noted that the FIGS. are schematic and provided for guidance to the skilled reader and are not necessarily drawn to scale. Rather, the various drawing scales, aspect ratios, and numbers of components shown in the FIGS. may be purposely distorted to make certain features or relationships easier to understand.
The lateral distance 4 is in particular a spacing between the motor vehicle 1 and object 7 within a plane that is parallel to a plane of movement of the motor vehicle, wherein the plane of movement lies between a longitudinal axis 16 of the motor vehicle 1 and a transverse axis 15 of the motor vehicle 1.
The park assistance system 2 also has a computing unit 9 that is connected or coupled to the distance sensor system 3 in order to receive the sensor signal, as well as an AR system 10 that for example is designed as a head-up display and is also coupled to the computing unit 9.
The AR system 10 may in particular have an imaging unit which may generate an image based on given image or graphics data by generating corresponding light signals. The AR system 10 may moreover include a projection surface 10′ that for example includes a part of the windshield of the motor vehicle 1. Moreover, the AR system 10 may have an optics unit which is configured to project the image generated by the imaging unit onto the projection surface 10′. The AR system 10 may therefore project an image or a graphic element onto the part of the windshield, and therefore into the field of view of the driver of the motor vehicle 1.
The computing unit 9 or the park assistance system 2 may optionally include a computer-readable storage medium 19 on which for example a computer program according to the teachings herein is saved that, while being run by the computing unit 9, causes the park assistance system 2 to execute a method for helping with parking according to the teachings herein.
While the park assistance system 2 is operating, it may actuate the distance sensor system 3 to execute a method for helping with parking according to the teachings herein in order to determine the lateral distance 4 between the motor vehicle 1 and the object 7, or respectively to generate a sensor signal based on the distance and transmit it to the computing unit 9. Based on the sensor signal, the computing unit 9 may generate graphics data that, for example, may be transmitted to the AR system 10 in the form of control commands or a graphics file.
The imaging unit of the AR system 10 may generate a corresponding image based on the graphics data and, by means of the optics unit, superimpose one or more graphic elements 11, 12, 13, 14 onto the projection surface 10′ onto the field of view of the driver. The graphic elements 11, 12, 13, 14 are in particular superimposed onto the field of view of the driver such that their extension along a vertical axis 17 in the field of view of the driver depends on the lateral distance 4.
In image a) in
In the image b) in
Optionally, the spacing 18 between the individual graphic elements 11, 12, 13, 14 in image b) may be less than in image a) and therefore reflect the lesser lateral distance 4.
In image c), the projection surface 10′ is shown in another situation in which for example 0.12 m was determined as the lateral distance 4. Correspondingly, the graphic elements 11, 12, 13, 14 are again depicted with a lesser extension along the vertical axis 17 in comparison to image b), and optionally again with a reduced respective spacing 18 in comparison to image b) in order to visually depict the smaller lateral distance 4.
In the image d) in
Optionally, the graphic elements 11, 12, 13, 14 in the different images a), b), c) and d) may be depicted in different colors or shades. In doing so, signal colors may be used to intuitively depict an impending collision as being even more immediate by very short distances.
For example in image a), the graphic elements 11, 12, 13, 14 may be depicted in a neutral color such as blue, gray, or green. In images b) and c), the graphic elements 11, 12, 13, 14 may for example be depicted in yellow or orange tones. In image d), the graphic elements 11, 12, 13, 14 may for example be depicted in red tones.
The teachings herein make it possible to safely help with parking a motor vehicle. On one hand, the safety of the driver may be improved by superimposing the corresponding information onto the field of view of the driver. To be able to completely depict the superimposing of the information by an AR system, the visual axis along which the spacing is measured is tilted out of a lateral plane in a vertical direction of the field of view of the driver so that one effectively switches from a perspective view to a birds-eye view. In particular, this also allows a close range of the motor vehicle to be completely or almost completely imaged using the AR system. By depicting the graphic element with an extension along the vertical axis that depends on the lateral distance, the driver does not have to divert his gaze from the road to detect the information, and a depicted situation does not have to be interpreted as with a camera image. The perception of the information is thereby more immediate and intuitive, and the driver may therefore process the information faster, which results in increased safety. Particularly in tight situations, the teachings herein may contribute to the driver being able to better estimate the distance between the motor vehicle and obstacle and thereby help optimally exploit the available parking space.
The invention has been described in the preceding using various exemplary embodiments. Other variations to the disclosed embodiments may be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor, module or other unit or device may fulfil the functions of several items recited in the claims.
The term “exemplary” used throughout the specification means “serving as an example, instance, or exemplification” and does not mean “preferred” or “having advantages” over other embodiments. The term “in particular” used throughout the specification means “serving as an example, instance, or exemplification”.
The mere fact that certain measures are recited in mutually different dependent claims or embodiments does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.
What is claimed is:
| Number | Date | Country | Kind |
|---|---|---|---|
| 10 2019 206 490.6 | May 2019 | DE | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2020/059315 | 4/1/2020 | WO |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2020/224877 | 11/12/2020 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 8965678 | Herrmann | Feb 2015 | B2 |
| 9792785 | Rittger | Oct 2017 | B2 |
| 20080100614 | Augst | May 2008 | A1 |
| 20100253918 | Seder | Oct 2010 | A1 |
| 20130158851 | Herrmann | Jun 2013 | A1 |
| 20160170487 | Saisho | Jun 2016 | A1 |
| 20160200249 | Boyd | Jul 2016 | A1 |
| 20160327402 | Funabiki | Nov 2016 | A1 |
| 20160327790 | Kasano | Nov 2016 | A1 |
| 20170192091 | Felix | Jul 2017 | A1 |
| 20170355307 | Ha | Dec 2017 | A1 |
| 20180029641 | Solar | Feb 2018 | A1 |
| 20180061134 | Ota | Mar 2018 | A1 |
| 20180105173 | Yamada | Apr 2018 | A1 |
| 20180157036 | Choi | Jun 2018 | A1 |
| 20190294895 | Kleen | Sep 2019 | A1 |
| 20190299855 | Ostapenko | Oct 2019 | A1 |
| 20200247412 | Wang | Aug 2020 | A1 |
| 20200307616 | Nithiyanantham | Oct 2020 | A1 |
| 20210197669 | Jeong | Jul 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| 102910130 | Feb 2013 | CN |
| 106945521 | Jul 2017 | CN |
| 102007042966 | Apr 2009 | DE |
| 102010025354 | Dec 2011 | DE |
| 102012017931 | Mar 2013 | DE |
| 102011121285 | Jun 2013 | DE |
| 102011121763 | Jun 2013 | DE |
| 102013206435 | Oct 2014 | DE |
| 102014118395 | Jun 2016 | DE |
| 102019206490 | Mar 2020 | DE |
| 3351417 | Jul 2018 | EP |
| 2020224877 | Nov 2020 | WO |
| Entry |
|---|
| German Office Action, Application No. 102019206490.6, 6 pages, Aug. 29, 2019. |
| International Search Report and Written Opinion, Application No. PCT/EP2020/059315, 7 pages, Jul. 23, 2020. |
| Chinese Office Action, Application No. 202080034076.X, 7 pages, Nov. 30, 2023. |
| Number | Date | Country | |
|---|---|---|---|
| 20220277496 A1 | Sep 2022 | US |
