Patent Grant
8891879
1. Field of the Invention
The present invention relates to an image feature extraction method, and image processing using the image feature extraction method.
2. Description of the Related Art
A technique of extracting the contour of an object in an image by using image features near pixels as contour candidates is known. Japanese Patent Laid-Open No. 8-16793 discloses a technique in which when extracting an edge area, a threshold for identifying the edge area or other areas is changed based on the statistical feature amount of pixels within a predetermined mask area around a pixel of interest and the edge intensity of the pixel of interest. This allows an edge area to be reliably and efficiently identified as compared with a method to identify an edge area using a fixed threshold.
Japanese Patent Laid-Open No. 5-324830 discloses a contour tracking method. In this method, one pixel on the contour of an object in an image is considered as an initial point, and pixels within a predetermined range from that pixel are determined as contour candidate points. Based on image data indicating the colors of each contour candidate point and pixels around it, it is determined whether a corresponding contour candidate point serves as a point on the contour to track the contour.
In the above techniques, however, it is impossible to automatically determine an appropriate range or the area of a shape when setting a neighboring area of the pixel of interest.
In the technique disclosed in Japanese Patent Laid-Open No. 8-16793, a mask area is set for a specified point in an image. There is, however, left unsolved a problem of how to determine the shape and range of the mask area. If an image in an environment where the background is complex is to be processed, there is not only a boundary which divides an area into two parts but also a junction such as a T-junction. In this case, it is impossible to simply set an inner or outer area as a mask area, and it is necessary to set a mask area in consideration of the shape of the contour and a surrounding luminance gradient.
In the technique disclosed in Japanese Patent Laid-Open No. 5-324830, based on image data indicating the colors of each contour candidate point and pixels around it, whether the corresponding contour candidate point serves as a point on the contour is determined. It is, however, impossible to determine a range of pixels around each candidate point to be considered. If, for example, only neighboring pixels are considered to fall within the range, the influence of image noise is great. It is, therefore, necessary to set a neighboring area having an appropriate range, and to attempt to reduce the influence of noise by using a statistical amount such as the average value or histogram of pixel values in the area. At this time, if a large area relative to a degree of unevenness of the contour or an area having an inappropriate shape for the shape of the contour is set especially in the vicinity of a target contour, a pixel group belonging to the background may be included in an area which should essentially include only a pixel group belonging to the target contour. In this case, as described above, it is necessary to set a partial area in consideration of the shape of the contour and a surrounding luminance gradient.
As a method of setting a partial area in accordance with a surrounding luminance gradient, Harris-Affine Region Detector is well known. Harris-Affine Region Detector is described in Krystian Mikolajczyk and Cordelia Schmid, “Scale & Affine Invariant Interest Point Detectors, “International Journal of Computer Vision, 60(1), pp. 63-86, 2004.
This technique is used to associate, if there are two images which are obtained when the same scene or object is viewed from different angles, partial images with each other. When applying this technique to a method of extracting a contour by sequentially connecting the contours of objects, such as the contour tracking method, it is necessary to associate parts belonging to a target object in a partial image because of a nonuniform background in an actual image.
In the present invention, when extracting a contour, a reference area including small areas is set in accordance with the global shape of the contour within an appropriate range of a target object. This suitably sets image features which are used for determining a search range of contour candidate points and whether each contour candidate point is on the contour, and then stably extracts pixels around the contour without depending on the shape of the contour. That is, even if the contour includes a smooth curve and a curve with fine unevenness, the contour is correctly extracted.
One aspect of embodiments of the present invention relates to an image processing apparatus for extracting a contour of an object in an image, comprising an area setting unit configured to extract a plurality of boundary candidate points from the image, and setting reference areas each of which has one of the plurality of boundary candidate points as a center and includes a plurality of small areas, a first reference area setting unit configured to set a first reference area from the plurality of reference areas; a first small area setting unit configured to set a first small area from a plurality of small areas included in the first reference area, a second reference area setting unit configured to set, from the plurality of reference areas, a second reference area whose feature associated with the shape is similar to that of the first reference area, a second small area setting unit configured to set, as a second small area, a small area which has a feature amount similar to that of the first small area among a plurality of small areas included in the second reference area, a storage unit configured to store, in association with each other, the first reference area, the second reference area, the first small area, and the second small area, and a contour extraction unit configured to track the boundary candidate points to extract the contour of the object based on the information stored in the storage unit.
Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings).
An embodiment of the present invention will be described in detail below with reference to the accompanying drawings. An image processing apparatus according to the embodiment uses an image feature amount within a reference area including a plurality of small areas, which is set based on the global shape of a contour in an image, and groups pixels near the contour.
An example will be explained below using the accompanying drawings. In this embodiment, a case will be explained in which pixels near a boundary are evaluated based on image features using color information in a still image including a plurality of objects and the background. Although the image features according to the embodiment use color information, the present invention is not limited to this. It is also possible to use another image information including image information which is obtained from a still image and moving image, such as luminance values and motion vectors.
Each reference area is divided into small areas in accordance with surrounding color information. Among smaller areas included in the reference area 210 and those included in the reference area 220, small areas located on the same side are compared. If, as a result of comparison, their image feature amounts are similar to each other, the reference areas 210 and 220 are determined to belong to the same group based on the image features of the small areas used for comparison. By repeating the determination processing, the contours are grouped using color information and contour shape information held by each figure as shown in
In this embodiment, if the shape of the boundary is complex as shown in
An image input unit 501 may serve as an image capturing apparatus for capturing an object in real time to generate image data, or a storage for storing image data. An image input from the image input unit 501 is output to a reference area setting unit 502. The reference area setting unit 502 obtains a scale so that an image feature amount at a given point in the image acquired from the image input unit 501 assumes an extreme value, and sets an area having a range based on the obtained scale.
A method for that purpose will be explained in detail below. A scale is obtained first so that a feature amount which is calculated using pixels of an area with a given point as the center, and given as a scalar amount assumes an extreme value in a Gaussian scale space. The feature amount includes, for example, a luminance gradient intensity, the Laplacian output, and various outputs of wavelet filters. The Gaussian scale space is described in Tony Lindeberg, “Edge Detection and Ridge Detection with Automatic Scale Selection, International Journal of Computer Vision, vol. 30, number 2, pp 117-154, 1998. If the extreme value obtained here is larger than a predetermined threshold, the point is determined as a boundary candidate point, and a circular or rectangular area having a size proportional to the scale at the point is determined as a temporary area.
If the extreme value is not larger than the predetermined threshold, and then the point is not determined as a boundary candidate point, that point is not used in subsequent processing. An eigenvalue and eigenvector of a matrix indicating a luminance value and the spread of a gradient within the temporary area are obtained. The matrix includes a covariance matrix of a luminance value and luminance gradient, and a second moment matrix. An elliptic area in which the direction of the eigenvector is determined as an axial direction and a set size proportional to the corresponding eigenvalue is determined as the axial length is set as a reference area. Although the shape of a reference area is an ellipse in this embodiment, an arbitrary shape such as a rectangle or a shape surrounded by a curve represented by parameters may be used.
A reference area dividing unit 503 divides the reference area set by the reference area setting unit 502 into a plurality of small areas. A practical example of the dividing processing will be described with reference to
{(L1−L2)2+(a1−a2)2+(b1−b2)2}1/2
The reference area dividing unit 503 integrates similar neighboring areas, among the small areas obtained by dividing the reference area, into one small area, and eventually outputs m (m≦M) small areas.
A reference area 610 shown in
A small area feature extraction unit 504 calculates an image feature amount within each small area output from the reference area dividing unit 503. The image feature amount here can be determined as the same one calculated by the reference area dividing unit 503. Note that the image feature amount is not limited to color information, but may be calculated by using another image information and then used. Next, a reference area information storage unit 505 stores, in association with each other, information on the position, boundary and scale of the reference area and that on the position and boundary of the ith small area output from the reference area dividing unit 503, and the image features of the small areas output from the small area feature extraction unit 504. Note that i assumes an integer of 1 to m, and m represents the number of small areas constituting a target reference area.
This will be described in detail with reference to
The information on the position and boundary of the ith small area is defined as follows. The information on the boundary of the small area will be explained first. Sectors which are obtained by equally dividing, into m, the central angle generated during the processing in the reference area dividing unit 503 are numbered counterclockwise from the positive direction of the major axis of the ellipse within the reference area 700. In
As can be seen in the reference area 700 of
A first reference area setting unit 506 selects the first reference area to be used for comparison among reference areas stored in the reference area information storage unit 505. A first small area setting unit 507 defines the first small area to be used for comparison among small areas belonging to the reference area output from the first reference area setting unit 506. A second reference area setting unit 508 defines the second reference area to be compared with the first reference area. The second reference area is selected based on information on the scale and shape of the first reference area, and that on the position and boundary of the first small area. Among reference areas at boundary candidate points within an area obtained by reducing or enlarging an ellipse of the first reference area at a predetermined ratio, a reference area whose features (the shape and direction) associated with the shape are most similar to those of the first reference area and whose center is closest to the that of the first reference area is selected as the second reference area. Note that it is possible to represent the shape at a ratio of the major axis to the minor axis of a reference area, and to represent the direction as a slope with respect to a predetermined direction of the major or minor axis.
A method of selecting the second reference area will be described in detail with reference to
The similarity of the shapes and directions of the reference areas is determined as follows. As for a given reference area Ri, among pieces of reference area information in the reference area information storage unit 505, a ratio of the length la of the major axis of an ellipse forming the boundary of the reference area to the length lb of the minor axis of the ellipse is defined by Si. Let θi be an angle, and σi be a scale. In this case, the similarity of a first reference area Rf and the reference area Ri is evaluated by the following three evaluation inequalities with respect to a ratio Sf of the major axis to the minor axis of an ellipse, an angle θf, and a scale σf.
|Si−Sf|<Sth, |θi−θf|<θth, |σi−σf|<σth
In this case, Sth, θth, and σth represent predetermined thresholds. θi and σi indicate an angle of the major axis and a scale parameter in the corresponding reference area Ri, respectively, and stored in the reference area information storage unit 505. Among a plurality of reference areas which satisfy all the three inequalities, a reference area which has a shortest distance between its center and that of the first reference area is selected as the second reference area.
This makes it possible to perform selection in consideration of surrounding image information such as a global contour shape in selecting the second reference area. This solves the problems of discontinuity of edge points and failure of grouping due to noise, which are associated with the contour tracking method of simply searching for neighboring edge points. In contrast to an existing area-growing method only referring to neighboring pixels, in the present invention, it is possible to reduce the influence of local noise by referring to an image feature amount in a range defined by a scale which optimally characterizes a boundary area.
Based on the information on the scale and shape of the first reference area and that on the position and boundary of the first small area, a second small area setting unit 509 defines the second small area to be compared with the first small area. A small area whose determination result exists in a determination result storage unit 511 is excluded from selection targets.
ν0·νi
In this case, ν0 represents a vector whose start point is the center of the reference area 210 and end point is the geometric centroid of the small area 901, and νi indicates a vector whose start point is the center of the reference area 220 and end point is the geometric centroid of any of the small areas 903 to 905.
A feature amount comparison unit 510 calculates a similarity using an image feature amount within the small area set by the first small area setting unit 507 and that within the small area set by the second small area setting unit 509. If the calculated similarity is equal to or larger than a predetermined threshold, the small areas compared to each other are determined to belong to the same group. The similarity calculation processing uses the same calculation unit as that utilized in the reference area dividing unit 503. If the feature amount comparison unit 510 determines that the small areas are similar to each other, the determination result storage unit 511 stores, in association with each other, the coordinates of the second reference area and the number of the second small area, and group numbers to which these areas are determined to belong.
According to the information stored in the determination result storage unit 511, a contour extraction unit 512 tracks boundary candidate points in the reference area grouped as a structure to extract the contour of an object in an image.
Constituent parts associated with the image processing apparatus according to the embodiment have been described. Processing executed by the image processing apparatus 100 of the embodiment will be explained next with reference to a flowchart in
In step S1, the reference area setting unit 502 sets an elliptic reference area with respect to a pixel P in an image input from the image input unit 501. In step S2, the reference area dividing unit 503 divides the set reference area into m small areas. In step S3, the feature amount of each of the small areas divided in step S2 is extracted. The feature amount here is the average value of pixel colors within the area, which is represented by the L*a*b* color system. The reference area dividing unit 503 integrates neighboring small areas whose similarity of the feature amounts is high.
In step S4, the reference area information storage unit 505 saves reference area information on the reference area set in step S1. The reference area information contains the position and scale of the reference area, the shape and size of the boundary of the reference area, the position and boundary of an integrated small area, and a feature amount within the small area. If, in step S5, there exists any pixel within a frame, which has not undergone the processing in steps S1 to S4, the processing in steps S1 to S4 is executed on the pixel. The processing in steps S1 to S5 completes the feature amount extraction in the image.
In steps S6 to S14, boundary candidate points are grouped based on the similarity of the extracted feature amounts. In step S6, a given reference area A in an image is set as the first reference area. It is possible to select, for the reference area A, a boundary candidate point whose coordinates are smallest (the upper left point of the image) among the boundary candidate points extracted from the image. A method of selecting the first reference area is not limited to this, and it is possible to select a reference area whose area is largest or a boundary candidate point whose coordinates are largest. In step S7, a small area a0 within the first reference area is set as the first small area. For example, it is possible to select, as the small area a0, a small area including an area which has been given the smallest or largest number among small areas which are not given a code indicating a checked state in step S14.
In step S8, based on the information on the shape and direction of the first reference area, a reference area B is set as the second reference area for comparing with the first reference area for their similarity. A method of selecting the second reference area is as described above. In step S9, a small area bc within the reference area B is set as the second small area. A small area whose inner product of a vector pointing from the center of the first reference area to that of the first small area and a vector pointing from the center of the second reference area to that of a small area bi within the second reference area is maximum is selected as the second small area bc. A small area which is not given a code indicating a checked state in step S14 is selected as the small area bc. This causes an edge to branch into a plurality of edges like a T-junction. Even if an edge point runs across a plurality of areas, it is possible to correctly select a small area to be compared.
If it is determined in step S10 that it was possible to select a small area satisfying the condition in step S9, the process advances to step S11; otherwise, the process ends. In step S11, the feature amounts of the first and second small areas are compared. If the feature amounts are similar to each other, the process advances to step S12; otherwise, the process advances to step S14. In step S12, pieces of reference area information of the first reference area, first small area, second reference area, and second small area are stored in a storage medium as one structure. That is, information of x, y, la, lb, θ, and σ pertaining to the first and second reference areas, and that of vi pertaining to the first and second small areas are combined into one structure. As long as it is recognized that the first and second small areas belong to the same group, a method of combining the information into a structure is not specifically limited, and for example, a method of assigning the number of group may be used.
In step S13, the reference area B is newly set as the first reference area, and the small area bc is newly set as the first small area. In step S14, the small area bc of the reference area B is given a code indicating a checked state. It is possible to implement this by giving the information vi of each small area a flag area indicating whether a check is complete or not when the reference area information storage unit 505 manages reference area information. The processing from step S5 is then repeated using the first reference area and reference small area newly set in step S13.
The above processing enables to group small areas with respect to a boundary candidate point, and to extract the contour in
Aspects of the present invention can also be realized by a computer of a system or apparatus (or devices such as a CPU or MPU) that reads out and executes a program recorded on a memory device to perform the functions of the above-described embodiment(s), and by a method, the steps of which are performed by a computer of a system or apparatus by, for example, reading out and executing a program recorded on a memory device to perform the functions of the above-described embodiment(s). For this purpose, the program is provided to the computer for example via a network or from a recording medium of various types serving as the memory device (for example, computer-readable medium).
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
This application claims the benefit of Japanese Patent Application No. 2009-204137, filed Sep. 3, 2009, which is hereby incorporated by reference herein in its entirety.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2009-204137 | Sep 2009 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5119439 | Osawa et al. | Jun 1992 | A |
| 5396584 | Lee et al. | Mar 1995 | A |
| 5586199 | Kanda et al. | Dec 1996 | A |
| 5668893 | Kanda et al. | Sep 1997 | A |
| 5680471 | Kanebako et al. | Oct 1997 | A |
| 5732149 | Kido et al. | Mar 1998 | A |
| 5751831 | Ono | May 1998 | A |
| 5754180 | Kivolowitz et al. | May 1998 | A |
| 5870494 | Kanda et al. | Feb 1999 | A |
| 5881170 | Araki et al. | Mar 1999 | A |
| 5887082 | Mitsunaga et al. | Mar 1999 | A |
| 5896466 | Kim | Apr 1999 | A |
| 5920356 | Gupta et al. | Jul 1999 | A |
| 5982386 | Fujimoto | Nov 1999 | A |
| 6031935 | Kimmel | Feb 2000 | A |
| 6178264 | Kamatani | Jan 2001 | B1 |
| 6252985 | Mitsunaga et al. | Jun 2001 | B1 |
| 6493465 | Mori et al. | Dec 2002 | B2 |
| 6674902 | Kondo et al. | Jan 2004 | B1 |
| 6687392 | Touzawa et al. | Feb 2004 | B1 |
| 6728404 | Ono et al. | Apr 2004 | B1 |
| 6757444 | Matsugu et al. | Jun 2004 | B2 |
| 6924836 | Ohki | Aug 2005 | B2 |
| 6980237 | Ohki | Dec 2005 | B2 |
| 7076099 | Kondo et al. | Jul 2006 | B2 |
| 7590262 | Fujimura et al. | Sep 2009 | B2 |
| 7627175 | Le Clerc et al. | Dec 2009 | B2 |
| 7636455 | Keaton et al. | Dec 2009 | B2 |
| 7729536 | Eguchi et al. | Jun 2010 | B2 |
| 7903861 | Luo et al. | Mar 2011 | B2 |
| 8144961 | Lee et al. | Mar 2012 | B2 |
| 8144993 | Nishimura et al. | Mar 2012 | B2 |
| 8199981 | Koptenko et al. | Jun 2012 | B2 |
| 8204287 | Inoue et al. | Jun 2012 | B2 |
| 8223402 | Yao | Jul 2012 | B2 |
| 8249345 | Wu et al. | Aug 2012 | B2 |
| 8330997 | Quan | Dec 2012 | B2 |
| 8422788 | Demandolx et al. | Apr 2013 | B2 |
| 20020172286 | Ohki | Nov 2002 | A1 |
| 20040086184 | Kondo et al. | May 2004 | A1 |
| 20050157941 | Lim | Jul 2005 | A1 |
| 20050249430 | Lim | Nov 2005 | A1 |
| 20070076965 | Shimada | Apr 2007 | A1 |
| 20080008369 | Koptenko et al. | Jan 2008 | A1 |
| 20080097209 | Lee et al. | Apr 2008 | A1 |
| 20080292154 | Nishimura et al. | Nov 2008 | A1 |
| 20090087024 | Eaton et al. | Apr 2009 | A1 |
| 20090175537 | Tribelhorn et al. | Jul 2009 | A1 |
| 20090208114 | Tanaka et al. | Aug 2009 | A1 |
| 20090237406 | Chen et al. | Sep 2009 | A1 |
| 20090316990 | Nakamura et al. | Dec 2009 | A1 |
| 20090324078 | Wu et al. | Dec 2009 | A1 |
| 20100046830 | Wang et al. | Feb 2010 | A1 |
| 20100054595 | Demandolx et al. | Mar 2010 | A1 |
| 20100054622 | Adams | Mar 2010 | A1 |
| 20100080487 | Yitzhaky et al. | Apr 2010 | A1 |
| 20100085611 | Quan | Apr 2010 | A1 |
| 20100215271 | Dariush et al. | Aug 2010 | A1 |
| 20100277770 | Yao | Nov 2010 | A1 |
| 20130121606 | Chien et al. | May 2013 | A1 |
| Number | Date | Country |
|---|---|---|
| H05-324830 | Dec 1993 | JP |
| H08-016793 | Jan 1996 | JP |
| Entry |
|---|
| K. Mikolajczyk et al., “Scale and Affine Invariant Interest Point Detectors”, Intl. Jour. of Computer Vision, vol. 60, No. 1, pp. 63-86 (2004). |
| T. Lindeberg, “Edge Detection and Ridge Detection with Automatic Scale Selection”, Intl. Jour. of Computer Vision, vol. 30, No. 2, pp. 117-154 (1998). |
| Number | Date | Country | |
|---|---|---|---|
| 20110052071 A1 | Mar 2011 | US |
