The invention relates to a method for traffic sign recognition by analyzing and classifying image data.
Modern driver assistance systems are being increasingly equipped with an electronic traffic sign recognition system in order to, e.g., warn the driver in the event of speeding. For this purpose, a camera records images of the region in front of the vehicle and delivers corresponding image data to an onboard computer that analyzes and classifies the image data by means of an algorithm in order to identify a traffic sign therefrom.
Such a method is known from, e.g., DE 198 52 631 A1.
The aim of such methods for traffic sign recognition consists in minimizing the rejection rate, i.e., the share of signs that are not recognized or recognized wrongly, wherein it would be advantageous if all traffic signs were standardized with respect to their design, whereby the great variety of different traffic signs, particularly in road traffic in foreign countries, would be reduced. Therefore, several European countries partially agreed on a standardization of traffic signs (Vienna Convention on Road Signs and Signals), e.g., on a characteristic design of speed limit signs (circular sign having a red outer ring and a number indicating the speed limit).
In a detection phase of such methods for traffic sign recognition, image regions that may contain potential traffic signs are identified in the camera image. After that, in a second procedure step, these sign hypotheses are submitted to a classificator that decides whether a traffic sign is present in the image region and which traffic sign it is.
The schematic block diagram in
The classificator or classification unit 5 may operate in a learning-based manner (known from, e.g., DE 10 2005 062 154 A1), i.e., it is appropriately trained in advance using a set of learning examples whose specific designs depend on the selected detection method. For example, a known method for speed limit recognition consists in searching for circles in the camera image by means of image processing phases during the detection phase and submitting the surrounding rectangle as an image detail to the classificator, wherein this feature “circles” defines a particular class of traffic signs.
A method for the recognition of circular objects in image data of an image sensor is known from, e.g., DE 10 2005 017 541 A1.
As explained above, most countries use speed limit signs that are standardized according to the Vienna Convention and are characterized in that only a centered numerical block indicating the speed limit is embedded in the traffic sign (see
In addition to these standardized speed limit signs, however, there are signs, e.g., in Austria or Belgium, whose designs differ from the standardized one. In these different signs, a smaller text (e.g., “km”) is supplemented to the relevant numerical block (see
Concerning a classificator that operates in a learning-based manner, these deviations from the usual design of a speed limit sign (centered numerical block on the traffic sign) will result in highly increased variability since learning or training examples must be generated and provided also for these variations. Furthermore, more sign hypotheses will pass through such a classificator on account of said increased variability so that there will be the risk of an increased false alarm rate (false positives).
It is therefore an object of one or more embodiments of the invention to provide a method for traffic sign recognition of the type mentioned above in which the number of necessary variations of learning and training examples for the classificator is limited without decreasing the recognition rate or endangering the real-time requirements.
The above object can be achieved according to the invention in an embodiment of a method in which the image data of a sensor are analyzed and classified in an information processing unit, wherein
If a class-specific feature of a particular class of traffic signs, e.g., of the class of circular speed limit signs, deviates from the unified standard, i.e., if, e.g., the numerical block is not centered on such speed limit signs, the image detail is modified in such a manner that the class-specific feature, i.e., this numerical block, is shifted to the image center of the image detail, said modification being performed prior to submitting the image detail to the classificator. The classificator performs classification on the basis of this modified image detail. Thus, the training of the classificator can be reduced to traffic signs that have the class-specific feature in the center of the submitted image detail, whereby the variation of the learning examples that are necessary for the training of the classificator is reduced so that the number of training examples to be provided can be reduced, too.
Because of this limitation, i.e., since the classificator must only classify class-specific features that are positioned in the center of the image detail to be processed so that the number of training examples is also reduced, it is possible to develop a robust classification method.
In a particularly advantageous further development of the invention, the modified image detail is generated by inserting the class-specific feature into the representation of a traffic sign of the particular class of traffic signs, said traffic sign being stored in a database. This means that instead of the real image detail, an artificial traffic sign of the particular class that does not have the class-specific feature and is stored in a database is used and inserted into the class-specific feature to be classified. After that, the modified image detail generated in this manner is submitted to the classificator for classification. This is another possibility of advantageously reducing the variability of the training examples, said reduction resulting in an increased recognition rate.
According to an advantageous further development of the invention it is not only possible to normalize the position of the class-specific feature. It is also possible to normalize the size by setting the class-specific feature to a predetermined image size in the modified image detail, thereby making a further reduction of the number of classificator training examples to be provided possible, whereby the robustness of the classification method can be additionally improved.
In a preferred further development of the invention, the position and/or the size are/is only normalized if the image detail to be processed has a particular size, thereby making it possible to prevent objects having a structure similar to the class-specific feature from being identified as potential traffic signs or to sort objects out that are too far from the vehicle or are assigned to an adjacent traffic lane.
Advantageously, in another further development of the inventive method, a modified image detail is only generated if at least one classification of the image data, preferably a predetermined number of image cycles, is unsuccessful. This results in an advantageous behavior of the classification method with respect to the real-time requirements on the recognition method.
Padding the image regions created by shifting the class-specific feature to the image center with pixels corresponding to the pixels of the surroundings of the class-specific feature is particularly advantageous with respect to the classification method. This results in an increased recognition rate. Preferably, these image regions may also be padded with pixels of a particular color, with pixels calculated from the pixels of the surroundings of the modified image detail, e.g., as a mean value with respect to a mean brightness or as a constant color, e.g., the background color of the traffic sign, the aim thereof being to achieve a representation for the classificator that corresponds to the training examples.
Particularly advantageously, the inventive method can be applied to a class of circular traffic signs and within this class of traffic signs to traffic signs with a numerical block, e.g., speed limit signs.
In the following, the invention will be explained in greater detail with reference to the drawings in which
The structure of the block diagram of a circuit arrangement according to
The image data of image cycles recorded by the camera 1 are stored in the storage unit 3 and submitted to the detection unit 4 for the detection of relevant image details. Relevant image details are image data that sufficiently probably contain a traffic sign that belongs to a particular class, e.g., a circular traffic sign. Furthermore, a class-specific feature is detected for this particular class of traffic signs, said feature being, e.g., a numerical block of a speed limit sign.
In the next procedure step, this image detail 10 according to
Thus, the image generated by means of the modified image detail 12 corresponds to the standardized traffic sign according to
The modified image detail 12 is now submitted to the classification unit 5 that identifies the traffic sign according to
A further example for the recognition of a traffic sign by means of the inventive method will be explained on the basis of
Prior to classifying this image detail 10, this numerical block 11 is centered in the image detail 10 by means of the centering unit 8 (as explained above in connection with the description of
Furthermore, when generating this modified image detail 12 with the centered numerical block 11, image regions 14 and 15 of the original position of the numerical block 11 in the image detail 10 are created that show a part of the structure of the digit “6” of the numerical block 11 or a part of the structure of the supplement “km”. As described in the previous example in connection with
The modified image detail 12 may also be generated by using artificial traffic signs of the particular class that are stored in a memory of the detection unit 4, said artificial traffic signs not having the class-specific feature. A representation of such a traffic sign corresponds to a circular traffic sign with a red edge (see
In addition to the normalization of the position of a numerical block 11 in an image detail 10, the size of the recognized numerical block 11 may be normalized. The size is normalized by means of a size normalization unit 9 that is shown as an additional component in the circuit arrangement according to
The inventive traffic sign recognition method may also be modified in such a manner that particular conditions concerning the image detail 10 must be met prior to centering and/or size normalization.
For example, the classification unit 5 may classify the image detail 10 at first, i.e., the image detail 10 generated in the detection unit 4 may be submitted to the classification unit 5 at first, and only if a successful classification cannot be performed after a particular number of image cycles, this image detail 10 is returned to the detection unit 4 and then supplied to the centering unit 8 or to the size normalization unit 9.
Furthermore, such a condition may be that image details 10 must have a particular size so that objects that are too far from the vehicle or do not represent a traffic sign or are traffic signs of adjacent traffic lanes are sorted out.
Number | Date | Country | Kind |
---|---|---|---|
10 2009 048 066 | Oct 2009 | DE | national |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/DE2010/001123 | 9/24/2010 | WO | 00 | 1/31/2012 |
Publishing Document | Publishing Date | Country | Kind |
---|---|---|---|
WO2011/038714 | 4/7/2011 | WO | A |
Number | Name | Date | Kind |
---|---|---|---|
4566124 | Yamamoto et al. | Jan 1986 | A |
4901362 | Terzian | Feb 1990 | A |
5594806 | Colbert | Jan 1997 | A |
6208758 | Ono | Mar 2001 | B1 |
6690842 | Silver et al. | Feb 2004 | B1 |
6801638 | Janssen et al. | Oct 2004 | B1 |
6842177 | Garg et al. | Jan 2005 | B2 |
6888892 | Chung et al. | May 2005 | B2 |
7034848 | Sobol | Apr 2006 | B2 |
7058206 | Janssen et al. | Jun 2006 | B1 |
7209141 | Garg et al. | Apr 2007 | B2 |
7466841 | Bahlmann et al. | Dec 2008 | B2 |
8064643 | Stein et al. | Nov 2011 | B2 |
8170340 | Klefenz | May 2012 | B2 |
8233670 | Moed | Jul 2012 | B2 |
8340420 | Smith et al. | Dec 2012 | B2 |
8346706 | Groitzsch et al. | Jan 2013 | B2 |
8370755 | Buecker et al. | Feb 2013 | B2 |
8379014 | Wiedemann | Feb 2013 | B2 |
8391612 | Natroshvili | Mar 2013 | B2 |
8396295 | Gao | Mar 2013 | B2 |
8452524 | Groitzsch et al. | May 2013 | B2 |
8953842 | Zobel | Feb 2015 | B2 |
9160993 | Lish | Oct 2015 | B1 |
20030059088 | Culp et al. | Mar 2003 | A1 |
20030202683 | Ma et al. | Oct 2003 | A1 |
20050086051 | Brulle-Drews | Apr 2005 | A1 |
20050111698 | Kawai | May 2005 | A1 |
20050232469 | Schofield et al. | Oct 2005 | A1 |
20060098877 | Barnes et al. | May 2006 | A1 |
20070081739 | Wilbrink | Apr 2007 | A1 |
20080107345 | Melikian | May 2008 | A1 |
20080137908 | Stein et al. | Jun 2008 | A1 |
20080199050 | Koitabashi | Aug 2008 | A1 |
20090074249 | Moed et al. | Mar 2009 | A1 |
20100198488 | Groitzsch et al. | Aug 2010 | A1 |
20100328316 | Stroila et al. | Dec 2010 | A1 |
20130011016 | Haas et al. | Jan 2013 | A1 |
20130058534 | Zobel | Mar 2013 | A1 |
20140119605 | Zobel | May 2014 | A1 |
Number | Date | Country |
---|---|---|
199 38 256 | Feb 2001 | DE |
198 52 631 | May 2005 | DE |
102005017541 | Oct 2006 | DE |
102005062154 | Jul 2007 | DE |
102006053289 | May 2008 | DE |
102006059663 | Jun 2008 | DE |
102008057675 | Jul 2009 | DE |
0 149 457 | Jul 1985 | EP |
1 508 889 | Feb 2005 | EP |
2 026 313 | Feb 2009 | EP |
2 048 597 | Apr 2009 | EP |
WO 9117518 | Nov 1991 | WO |
Entry |
---|
Ishida H., et al., “Identification of degraded traffic sign symbols by a generative learning method,” Proceedings of the 18th International Conference on Pattern Recognition (ICPR'06), 2006. |
Wang, Yongping et al., National University of Defense Technology, ChangSha, China, “A Method of Fast and Robust for Traffic Sign Recognition”, Fifth International Conference on Image and Graphics, Sep. 20, 2009, IEEE, NJ, USA, XP031652742, pp. 891 to 895. |
Priese, Lutz et al., University of Koblenz-Landau, Koblenz, Germany, “Ideogram Identification in a Realtime Traffic Sign Recognition System”, Intelligent Vehicles '95 Symposium, Sep. 25, 1995, Michigan, USA; IEEE, NY, USA, XP010194135, pp. 310 to 314. |
Hoessler, Helene et al., “Classifier Training Based on Synthetically Generated Samples”, 5th International Conference on Computer Vision Systems, Mar. 21, 2007, Applied Computer Science Group, Bielefeld University, Germany, XP002510914, pp. 1 to 10. |
International Search Report of the International Searching Authority for International Application PCT/DE2010/001123, mailed Jan. 13, 2011, 3 pages, European Patent Office, HV Rijswijk, Netherlands. |
PCT International Preliminary Report on Patentability, including English Translation of PCT Written Opinion of the International Searching Authority for International Application PCT/DE2010/001123, issued Apr. 11, 2012, 9 pages, International Bureau of WIPO, Geneva, Switzerland. |
German Search Report for German Application No. 10 2009 048 066.8, dated Oct. 12, 2010, 3 pages, Muenchen, Germany, with English translation, 3 pages. |
X. W. Gao et al., “Recognition of traffic signs based on their colour and shape features extracted using human vision models”, Journal of Visual Communication and Image Representation, vol. 17, Issue 4, Aug. 2006, pp. 675 to 685. |
English translation of DE 10 2008 057 675 A1. |
Number | Date | Country | |
---|---|---|---|
20120128210 A1 | May 2012 | US |