Patent Application
20190096076
The present application claims the benefit under 35 U.S.C. § 119 of German Patent Application DE 102017217156.1 filed on Sep. 27, 2017, which is expressly incorporated herein by reference in its entirety.
The present invention is directed to a device and to a method for activating a driver assistance system. The present invention also relates to a computer program.
Stereo camera systems made up of two identical cameras oriented toward the same target objects are increasingly used for surroundings monitoring, in particular for driver assistance systems, since in this way the distance from objects may be ascertained via the perspective representation of the two camera images. Different methods may be used for determining the distance from two image pairs. The effect of the “periodic structures” in the image pairs, however, may in part cause problems in detecting a distance from objects which may result in an erroneous activation of a driver assistance system if the image pairs are used as a basis for the functions of the driver assistance system.
In accordance with the present invention, a method is provided for activating a driver assistance system using a stereo camera system including a first and a second camera, furthermore a device which uses this method, and finally a corresponding computer program.
Advantageous refinements of and improvements on the example device are described herein.
An example method in accordance with the present invention for activating a driver assistance system using a stereo camera system including a first and a second camera is provided, the method including the following steps:
reading in a first image from the first camera and a second image from the second camera;
forming a cost function, using the first and second images;
determining a periodicity parameter (for example of a cost function) representing a periodic structure of an object (for example from the stereo camera system), at least using a local minimum of the cost function;
using the periodicity parameter for activating the driver assistance system.
The driver assistance system may be an electronic additional unit in a vehicle for assisting the driver in certain driving situations. Interventions and/or hints with respect to vehicle safety, but also the enhancement of the driving comfort for the driver and further vehicle occupants may be carried out or output. The stereo camera system may be a camera system which includes at least two lenses provided next to one another and is thus able to record stereoscopic images. A cost function may be understood to mean a value calculated using a functional relationship, which represents disparities along the associated epipolar line and a depth distance. A cost function may thus represent a relationship between the calculated or virtual costs, for example of all disparities possible for a pixel of a base image along the associated epipolar line, and a depth distance of an object. A periodic structure in an image pair may have been caused, for example, by a lattice-shaped fence, guard rails or a corrugated sheet roof, which during the ascertainment of the cost function also results in a periodic structure in this cost function. A periodic structure within the cost function may be curves or sequences of function values which recur at regular intervals and have the same cost function values (within tolerance limits). The intervals between the occurrence of the same function values may be referred to as a period. A periodicity parameter may thus constitute or represent a characterizing variable of the periodic structure within a cost function or a measure of the periodicity of the cost function. A local minimum may represent a value of the cost function at a point in whose surroundings (on both sides) the cost function does not assume any smaller or lower values. The local minimum, however, does not necessarily have to be the global minimum of the entire cost function.
The approach in accordance with the present invention is based on the finding that, by evaluating the curve of the cost function, the periodicity parameter may be obtained in a technically simple and efficient manner as a parameter which provides an indication of periodic structures occurring in the image pair. In a simple manner, this allows a conclusion to be drawn of periodic structures occurring in the image pair which could result in a potentially erroneous function of the driver assistance system, for example carry out an erroneous distance warning or an erroneous emergency brake application. For identification, in particular the knowledge of the location of at least one local minimum may be used, since such a local minimum may be easily and reliably detected.
According to one specific embodiment, a sub-section of the first image may be compared to at least one further sub-section of the second image in the step of forming, in particular a row of the first image is compared to a row of the second image and/or a column of the first image being compared to a column of the second image. Advantageously, the processing of sub-sections may be carried out in rows and/or in columns, which may be implemented in a technically fast and simple manner.
According to one specific embodiment, in the step of determining, the cost function may be determined as a function of a disparity parameter representing the distance of the object from the stereo camera system. Furthermore, in the step of determining, the disparity parameter may also be used, which represents a reciprocal measure of the distance of the object from the stereo camera system. Such a specific embodiment of the approach described here offers the advantage of ascertaining a cost function which has a high degree of similarity of mapping parameters relevant for a driver assistance system, such as the distance of an object ahead of a vehicle, from vehicle surroundings. Since the calculation of a cost function based on a disparity value is computationally intensive, preferably simple and fast algorithms are used to prevent an unnecessary increase in the computing complexity by checking the cost function for a parameter representing a periodicity. The global minimum or a local minimum may generally be easily ascertained during the passage of the disparity curve or the cost function from left to right (i.e., during the ascertainment of the cost function values from small disparity values to large disparity values) without computationally intensive regressions. Checking the cost function for a parameter representing the periodicity may also be ascertained during the passage of the disparity curve or cost function from left to right (i.e., during the ascertainment of the cost function values from large disparity values to small disparity values) without regressions. A possible expansion of the method is to simultaneously ascertain a parameter representing the periodicity and the global minimum, and check a frontoparallelity of an object in the image pair.
According to one specific embodiment, in the step of determining furthermore at least one local maximum and a global minimum and a global maximum of the cost function may be determined, in particular the local maximum situated between the global and the local minimum. Such a specific embodiment of the approach described here offers the advantage of using suitable variables to obtain, through the periodicity parameter, reliable information which is technically simple to ascertain regarding the presence of a periodicity in the cost function or in corresponding sub-sections of an image pair.
According to one specific embodiment, in the step of determining, the periodicity parameter may be determined as a function of a difference of a value of the cost function at the local minimum and a value of the cost function at the local maximum. In this way, it is advantageously possible to identify how strongly the values of the cost function vary in the surroundings of the local minimum, to be able to draw a conclusion, for example, of the presence of interferences caused by image noise. According to this specific embodiment, a variable or the periodicity parameter is determined as a measure of the periodicity, which may be compared to a threshold value, for example, in a technically simple manner.
According to another specific embodiment, in the step of determining, the periodicity parameter may be determined as a function of a further difference of cost values of an adjoining further local maximum and a further local minimum. Such a specific embodiment also offers the advantage that a meaningful conclusion regarding the presence of a periodicity or a measure of the periodicity by using the further difference is able to be drawn.
According to one further specific embodiment, in the step of determining, the periodicity parameter may be determined as a function of a maximum of the difference and the further difference. As an alternative or in addition, in the step of determining, the periodicity parameter may be determined as a function of the further local maximum and the further local minimum, the global minimum being situated between the further local maximum and the further local minimum on the one hand, and the local maximum and the local minimum on the other hand. By using a dependence of the periodicity parameters on the difference and the further difference and/or location of the further minimum or the location of the further maximum, a higher precision or more precise information regarding the presence of a periodicity in the cost function or corresponding sub-sections in the image pair may be achieved.
According to one specific embodiment, in the step of determining, the periodicity parameter may be determined as a function of a value of the cost function at a local maximum and a value of the cost function at the global minimum. In particular, in the step of determining, the periodicity parameter may be determined as a function of a difference from a value of the cost function at the local maximum and the value of the cost function at the global minimum or a value of the cost function at the local maximum and a value of the cost function at the global minimum. This specific embodiment of the approach described here also offers a higher precision or more precise information regarding the presence of a periodicity in the cost function or corresponding sub-sections in the image pair.
According to one specific embodiment, in the step of determining, the periodicity parameter may be determined as a function of a ratio of a value of the cost function at the global minimum to a value of the cost function at the global maximum, in particular the ratio representing a measure of the periodicity of the cost function as a function of a threshold value. This specific embodiment of the approach described here also offers a higher precision or more precise information regarding the presence of a periodicity in the cost function or corresponding sub-sections in the image pair.
According to one specific embodiment, in the step of determining, the periodicity parameter may be formed as a bit value. It therefore requires only little memory space, may be transmitted quickly, and may also be evaluated well in other functions. In this way, furthermore a processing of this parameter, for example using a 2 bit shift operation, is implementable in a technically simple manner.
The method described here may be implemented, for example, in software or hardware or in a mixed form made up of software and hardware, for example in a control unit.
The approach described here also creates a device which is designed to carry out, activate or implement the steps of one variant of a method described here in corresponding units. The object underlying the present invention may also be achieved quickly and efficiently by this embodiment variant of the present invention in the form of a device.
For this purpose, the example device may include at least one processing unit for processing signals or data, at least one memory unit for storing signals or data, at least one interface to a sensor or an actuator for reading in sensor signals from the sensor or for outputting data signals or control signals to the actuator and/or at least one communication interface for reading in or outputting data which are embedded into a communication protocol. The processing unit may be a signal processor, a microcontroller or the like, for example, it being possible for the memory unit to be a Flash memory, an EEPROM or a magnetic memory unit. The communication interface may be designed to read in or output data wirelessly and/or in a wire-bound manner, a communication interface which is able to read in or output wire-bound data being able to read these data in, for example electrically or optically, from a corresponding data transmission line or output these into a corresponding data transmission line.
A device may presently be understood to mean an electrical device which processes sensor signals and outputs control and/or data signals as a function thereof. The device may include an interface which may be designed as hardware and/or software. In the case of a hardware design, the interfaces may, for example, be part of a so-called system ASIC which includes a wide variety of functions of the device. However, it is also possible for the interfaces to be separate integrated circuits, or to be at least partially made up of discrete elements. In the case of a software design, the interfaces may be software modules which are present on a microcontroller, for example, in addition to other software modules.
In addition, a computer program product or computer program is advantageous, having program code which may be stored on a machine-readable carrier or memory medium such as a semiconductor memory, a hard disk memory or an optical memory, and which is used to carry out, implement and/or activate the steps of the method according to one of the specific embodiments described above, in particular if the program product or program is executed on a computer or a device.
Exemplary embodiments of the present invention described here are shown in the figures and are described in greater detail below.
In the following description of favorable exemplary embodiments of the present invention, identical or similar reference numerals are used for similarly acting elements shown in the different figures, and a repeated description of these elements is dispensed with.
The drawing shows one example of a stereo camera system 100 made up of two identical cameras 102, 104, both cameras 102, 104 being oriented toward the same target object 106, a house 106 here, to which a path 108 leads. The two cameras 102, 104 record the same scene, i.e., house 106, from different spatial points of view. The distance of house 106 from stereo camera system 100 is to be ascertained via a perspective representation of the two camera images. The epipolar geometry describes the relationship between the two different camera images of the same target object 106. In this way, the dependence between the corresponding image points, i.e., the points which an individual object point generates in the two camera images, may be described. In addition to the distance of the object, such as house 106 here, the frontoparallelity of the object, of house 106 here, for example, may also be ascertained by the evaluation of the images of the two cameras 102 and 104. For example, in a portion of the images of the two cameras 102 and 104 which depict a section of house 106, it may be ascertained, for example, which orientation identified edges in the respective images have, so that an orientation of the object, such as house 106, with respect to cameras 102 and 104 of stereo camera system 100 may be ascertained therefrom. In contrast, for example, it may also be identified from a degree of inclination of the edge progressions of identified edges in the area of path 108 that this path 108 does not represent an object which is oriented in a frontoparallel manner with respect to the image recording plane of stereo camera system 100.
To be able to identify a periodic structure of an object, such as the rows of periodic windows in house 106, device 101 briefly mentioned above is used to activate a driver assistance system 110, using stereo camera system 100 including first 102 and second 104 camera. For this purpose, device 101 includes an interface 120 for reading in a first image from first camera 102, and a second image from second camera 104. Device 101 furthermore includes a unit 125 for forming a cost function using the first image and the second image, and a unit 130 for determining a periodicity parameter of a cost function representing a periodic structure of object 106, at least using a local minimum of the cost function. Finally, device 101 includes a unit 135 for using and/or outputting the periodicity parameter for activating driver assistance system 110.
The illustration includes a first left image 202 (for example, of first camera 102 from
In general, it may be noted that it is irrelevant in which direction the cost function is passed. What is important is that it is preferably passed only once for the calculation.
Different methods may be used for determining the distance from an image pair. Frequently, local methods are used which, in principle, search for a small sub-section 206 of first image 202 in a second rectified image 204 along epipolar line 208. The similarity of first sub-section 206 with further sub-section 222 of second image 204 along epipolar line 208 is represented as cost function 210. An extreme value, global minimum 224 here, of cost function 210 represents the disparity, i.e., for example, the offset of the identical image content, between first image 202 and second image 204. This disparity is a reciprocal measure of the distance of object 106 from camera. The shape of cost function 210, for example the value of global minimum 224, may be used for evaluating the quality of the disparity.
When the disparities of all sub-sections of left image 202 from the content of the right image 204 are determined, a disparity map, and thus a depth map, for the entire image may be created. Further methods may be applied to this depth map, for example to detect the surface or the location of objects, such as road surfaces, pedestrians or vehicles. In driver assistance systems, such as driver assistance system 110, the erroneous detection of objects could result in incorrect and dangerous driving maneuvers, for example an emergency brake application or an evasive maneuver.
Ideally, cost function 210 has a clearly detectable global minimum 224 having a steep rise. Further possible local minima, in terms of their costs, are far above the costs of global minimum 224. The position of global minimum 224 is sought-after disparity value 216. This ideal case, however, only occurs with ideal ambient conditions.
Periodic structures in cost function 210 are problematic in the evaluation of cost function 210 since these could result in an identification of global minimum 224 which is no longer unambiguous, specifically when the differences between the cost function values of the local minima are so low that these minima could also result from image interferences or other errors, for example. For this reason, the approach described here is to show a way as to how it may be identified that a periodic structure occurs in the cost function which, for example, is due to a periodic structure or a periodic pattern in the sub-sections of the image pairs to be evaluated. This may, for example, then result in a corresponding periodicity parameter, which represents an occurrence of such a periodic structure in the cost function, being ascertained and used to activate driver assistance system 110. For example, a threshold value for a steering intervention or an emergency braking intervention may be changed in driver assistance system 110 when a certain periodicity parameter provides an indication of the presence of a periodic structure in cost function 210 or in sub-sections of image pairs.
The image shows a vehicle 302 driving on a road, which approaches target object 106, a crosswalk here. Furthermore, a function 210 and an epipolar line 208, which is situated on top of the illustration of target object 106, are shown. When the image shown in
In real surroundings as are shown in
The diagram illustration includes a coordinate system 212, which indicates a cost function 210 having a periodic structure, which hereafter is also referred to as a periodic cost function 210. Here, x axis 214 of coordinate system 212 represents an increasing disparity value 216. In contrast, y axis 220 represents a cost function value. It is to be noted that, with an ideal periodic function, for example a sine curve, the cost function values of all maxima 402, 404 and all minima 304, 306 are identical, and thus the difference between two arbitrary minima 304, 306 and maxima 402, 404 is always the same.
In the approach of an evaluation of periodic cost function 210 described here, additionally a quality criterion in the form of the periodicity parameter is to be calculated or determined, which shows the existence of periodic structures in the disparity curve. With the aid of this quality criterion, it is possible to determine the plausibility of the global minimum, i.e., of the disparity. One aspect of the approach described here for the identification of periodic structures is based on the evaluation of cost function 210. This takes advantage of the fact that periodic structures in the image pairs also form periodic structures in cost function 210.
The diagram illustration again includes a coordinate system 212, which indicates a cost function 210 having a periodic structure, which hereafter is also referred to as a periodic (cost) function 210. An (increasing) disparity value 216 is plotted on x axis 214 of coordinate system 212. In contrast, y axis 220 indicates a cost function value. Periodic (cost) function 210 has a local minimum 502, a local maximum 504, a global minimum 224 and a global maximum 506.
In a real periodic function 210, the difference between global minimum 224 and global maximum 506 is only slightly larger than a pair having a large distance between a local minimum 502 and a local maximum 504, local maximum 504 being situated between global minimum 224 and local minimum 502. The ratio of the difference from global minimum 224 and global maximum 506 to the difference from local minimum 502 and local maximum 504 represents a measure of the periodicity of function 210.
The diagram illustration shows a coordinate system 212, which indicates a cost function 210 having a periodic structure, which hereafter is also referred to as a periodic function 210. An (increasing) disparity value 216 is plotted on x axis 214 of coordinate system 212. In contrast, y axis 220 indicates a cost function value. Periodic function 210 has local minimum 502 and a further adjoining local minimum 602, local maximum 504 and a further adjoining local maximum 604, global minimum 224 and global maximum 506.
In a real (cost) function 210, minima 502, 602 and maxima 504, 604 occur in arbitrary positions. Since in a certain or sought-after pair, for example local minimum 502, 602 and local maximum 504, 604, local maximum 504, 604 is to be situated between local minimum 502, 602 and global minimum 224, it may occur on x axis 214, 216 to the left or the right of global minimum 224.
Since the order between local minimum 502, 602 and local maximum 504, 604 is reversed for the left and right pairs, this affects the algorithmic search. The realization of the implementation for the pair search may take place separately from one another. In this regard, the following procedure according to one favorable exemplary embodiment has proven suitable:
Initially, global minimum 224 and global maximum 506 are sought, and their cost difference is calculated. Subsequently, local minimum 502 and local maximum 504 having the maximum cost difference are sought, pair 502, 504 being situated before global minimum 224 on x axis 214, 216, and the local maximum being situated between local and global minima. Finally, local minimum 602 and local maximum 604 having the maximum cost difference are sought, the pair being situated after global minimum 224 on x axis 214, 216, and the local maximum again being situated between global and local maxima. These steps may advantageously be algorithmically combined with one another, so that all sought-after variables may be determined by a one-time passage of cost function 210 from left to right (i.e., from small to large disparity parameters). However, it is primarily of interest here that a one-time passage may take place during the determination of the cost function or the respective extreme values. Whether this passage takes place from left to right (small to large) or right to left (large to small) is essentially a matter of view. A periodic structure is present when the largest cost difference of local minima 502, 602 and local maxima 504, 604 is greater than the cost difference of global minimum 224 and of global maximum 506, multiplied by a constant factor, the constant factor in this example corresponding to a value of ¼. This value may be easily implemented by a rapid 2 bit shift operation.
As an alternative, the periodicity parameter may be determined as a function of a value of cost function 210 at a local maximum 504, 604 and a value of cost function 210 at global minimum 224, in particular as a function of a difference from a value of cost function 210 at local maximum 504 and the value of cost function 210 at global minimum 224, a value of cost function 210 at local maximum 504, 604 and a value of cost function 210 at global minimum 224.
Additionally, it should be noted that the approach described here introduces a method for evaluating the cost function in which, in addition to the extreme values, a quality criterion in the form of the periodicity parameter is calculated, which shows or maps the existence of periodic structures in the disparity curve or the cost function. With the aid of this quality criterion, it is possible to determine the plausibility of the global minimum, i.e., of the disparity.
One aspect of the approach described here for the ascertainment of periodic structures is thus based on the evaluation of the cost function. This takes advantage of the fact that periodic structures form a periodic function in the cost function. As a basis in this regard, it may be noted that, with an ideal periodic function (e.g., a sine curve), the values of all maxima and all minima are identical, and thus the difference between two arbitrary minima and maxima is always the same, as was already shown and described in
In a “beautiful” real periodic function, the difference between the global minimum and the global maximum is only slightly larger than a pair having a large distance between a local minimum and maximum, the local maximum being situated between the global and local minima, as was shown and described with respect to
An actual realization of the above-described approach in real surroundings may be implemented as follows: In a real function, the minima and maxima occur in arbitrary positions. Since, in the sought-after pair (i.e., local minimum and local maximum), the local maximum is to be situated between the local minimum and the global minimum, this may occur to the left or right of the global minimum. In
Since the order between the local minimum and the local maximum is reversed for the left and right pairs, this affects the algorithmic search. The realization of the implementation for the pair search may take place separately from one another.
In this regard, the following procedure has proven suitable:
These steps may advantageously be algorithmically combined with one another, so that all sought-after variables may be determined during a one-time passage of the cost function from left to right.
A check for the presence of a periodicity may then be calculated as follows:
Alternatively, it is also possible to use the following ratio of formula (3) for checking the presence of a periodicity:
the designation Cost ( . . . ) being understood to mean the cost function value at the site provided as an argument here.
For example, the following aspects may be mentioned as advantages of the approach described here:
In a step 701, a first image from the first camera and a second image from the second camera are read in. In a step 703, a cost function is formed, using the first and the second image. In a step 705, a periodic structure of an object to the periodicity parameter representing the stereo camera system is determined, using at least one local minimum of the cost function. Finally, in a step 707, the periodicity parameter is used to activate the driver assistance system.
If one exemplary embodiment includes an “and/or” linkage between a first feature and a second feature, this is to be read in such a way that the exemplary embodiment according to one specific embodiment includes both the first feature and the second feature, and according to an additional specific embodiment includes either only the first feature or only the second feature.
| Number | Date | Country | Kind |
|---|---|---|---|
| 102017217156.1 | Sep 2017 | DE | national |
