The above and/or other aspects and advantages of the present invention will become apparent and more readily appreciated from the following detailed description, taken in conjunction with the accompanying drawings of which:
Reference will now be made in detail to the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The embodiments are described below to explain the present invention by referring to the figures.
A conventional min-cut method uses only shape information of an input image as described referring to
Hereinafter, a method of generating the prior shape information 302 and combining the shape information as shown in the shape information combination 304 will be described by referring to
In operation S410, a certain area extraction system combines image information with shape information, based on an input image and prior shape information. In this case, the prior shape information may include a shape model and a weight model. Also, as shown in
In sub-operation S411, the certain area extraction system generates a shape constraint based on the input image and the shape model. The shape constraint is made by establishing a connection to resist a cut between pixels separated by a certain distance, and a method of generating the shape constraint will be described in detail referring to
In sub-operation S412, the certain area extraction system generates a shape specified gradient image based on an approximate shape and a gradient image. The shape specified gradient image indicates that a gradient is projected to a vector in the norm direction of shape information to acquire a gradient image considering the shape information. A method of generating the shape specified gradient image will be described in detail referring to
In sub-operation S413, the certain area extraction system generates a shape specified weight image based on the input image, a tri-map of the input image, and the weight model. The shape specified weight image is to smooth the certain area by using the weight model introduced to smooth a weight map. A method of generating the shape specified weight image will be described in detail referring to
In operation S420, the certain area extraction system extracts the certain area from the input image by using the image information. In this case, as shown in
In sub-operation S421, the certain area extraction system generates a connection to the uncertain pixel by using the shape constraint, the shape specified gradient image, and the shape specified weight image. A method of generating the connection to the uncertain pixel will be described in detail referring to
In sub-operation S422, the certain area extraction system determines the uncertain pixel to be the foreground pixel or the background pixel by removing a connection having weak intensity from a plurality of connections to the uncertain pixel.
In sub-operation S423, the certain area extraction system extracts the certain area by extracting only pixels determined to be the foreground pixel, from the input image.
The method of extracting the certain area, described referring to sub-operations S421 through S423 will be described in detail referring to
The shape model 501 expresses a contour of an object and is formed of a line connecting a K number of control points. When the certain area is a figure, samples formed as described above may be arranged by using a position of eyes of the figure. Also, the shape model 501 may be used as a principal component analysis (PCA) model.
PCA is a method of contracting multidimensional data desired to be analyzed into two-dimensional or three-dimensional data by reducing loss of information included in the data. Applying the PCA, it can be visually recognized where an object of observation is located.
The weight model 502 expresses a weight map and indicates a probability that each pixel expressing the object corresponds to a foreground pixel or a background pixel. In this case, a weight exists in an N×M area, and an input dimension may be N×M and an output dimension may be L (L<<N×M). In this case, the weight model 502 may be also used for the PCA model.
Also, the uncertain pixel 603 may indicate a pixel that is not definitely determined to be the foreground pixel 601 or the background pixel 602. When definitely determining the uncertain pixel 603 to be the foreground pixel 601 or the background pixel 602, an edge of the certain area desired to be extracted may become smoother.
In operation S710, the certain area extraction system generates an approximate shape by using an input image and a shape model of prior shape information. In this case, in operation S710, the approximate shape may be generated by an approximate shape generation module by using the input image and the shape model as an input. The approximate shape generation module may include an active shape model (ASM) method.
In operation S720, the certain area extraction system generates the shape constraint based on the approximate shape. In this case, operation S720 may include sub-operations S721 through S724.
In sub-operation S721, the certain area extraction system checks a pixel existing at a predetermined distance from the uncertain pixel of the tri-map. As a preparatory operation to compare a virtual line connecting the uncertain pixel and the pixel within the approximate shape, a connection in which a weight is given according to a degree of being parallel to the virtual line and being parallel to the approximate shape may be established via sub-operations S722 and S723.
In sub-operation S722, the certain area extraction system calculates a difference between a distance between the uncertain pixel and the approximate shape and a distance between the pixel and the approximate shape. The smaller the difference, the more parallel the virtual line and the approximate shape.
In sub-operation S723, the certain area extraction system establishes a connection to resist a cut between the uncertain pixel and the pixel, in which the difference is less than a predetermined value. Namely, the connection having a higher weight is established between two pixels forming the virtual line more similar to the approximate shape, generating the shape constraint, as shown in sub-operation S724.
In sub-operation S724, the certain area extraction system generates the shape constraint via the connection. In this case, the shape constraint may form a distance map to process the connection at high speed. The method of generating the shape constraint, described referring to sub-operations S721 through S724, will be described in detail referring to
As described above, a connection to resist a cut between pixels in a direction similar to the approximate shape may be established. However, since a line connecting the pixel 801 and the pixel 803 is not in the direction similar to the part of the approximate shape 804, a connection is not established.
To recognize pixels in a similar direction and to more quickly calculate the connection, a distance map IDist is utilized. To give a greater weight to the connection between pixels having a direction similar to the part of the approximate shape 804, Equation 1 may be introduced.
N
shape(p,q)=λ1exp(−α1·|IDist(p)−IDist(q)|) [Equation 1]
In a pixel p, IDist(P) indicates a distance from the pixel p to the part of the approximate shape 804. In this case, based on the pixel 801, a pixel to which a highest weight is given exists in a direction 805.
In operation S901, the certain area extraction system calculates a vector in the norm direction in each local shape with respect to the approximate shape. The vector in the norm direction may be used to generate the shape specified gradient image by combining a gradient image generated by convoluting a sobel filter in the directions of x coordinates and y coordinates with respect to the input image with prior shape information, in operation S902 and S903.
In operation S902, the certain area extraction system calculates a gradient with respect to each edge of the gradient image.
In operation S903, the certain area extraction system calculates a final gradient by using an inner product of the gradient and the vector in the norm direction. The inner product indicates projecting the gradient to the vector in the norm direction.
In operation S904, the certain area extraction system generates the shape specified gradient image by using the final gradient.
The method of generating the shape constraint described referring to operations S901 through S904 will be described in detail referring to
To generate the shape specified gradient image, a vector in the norm direction {right arrow over (N)} 1002 with respect to a part of an approximate shape 1001 is calculated and a gradient ∇I with respect to each edge 1003 is calculated.
A final gradient G may be calculated by projecting the gradient ∇I with respect to each edge 1003 to the vector in the norm direction 1002, namely, by using an inner product of the vector in the direction of the norm 1002 and the gradient ∇I, as shown in Equation 2.
G=∇I·{right arrow over (N)} [Equation 2]
With respect to an image having a C channel, an n-link of neighboring pixels p and q may be shown as Equation 3.
In operation S1101, the certain area extraction system generates a weight image based on the input image and the tri-map. In this case, the weight image may include an image to which a probability of an uncertain pixel of the tri-map to a foreground pixel and a background pixel is given as a weight.
When histograms of the foreground pixel and the background pixel are HFore and HBack, respectively, a weight in the uncertain pixel p=(x, y) may be defined as shown in Equation 4.
The weight indicates a t-link with respect to the foreground pixel and the background pixel, and F and B indicate a foreground and a background, respectively.
In operation S1102, the certain area extraction system generates the shape specified weight image based on the weight image and the weight model 502. In this case, the shape specified weight image may include an image made by transforming the weight image to be more consistent with the weight model. In this case, as an example of transformation of the image, PCA may be used.
In operation S1201, the certain area extraction system generates a first connection between a predetermined semantic node and a pixel by using the shape specified weight image. In this case, the semantic node may include a semantic background and a semantic foreground. In addition, the semantic background may determine a connection with respect to a background weight of the pixel, and the semantic foreground may determine a connection with respect to a foreground weight of the pixel.
In operation S1202, the certain area extraction system generates a second connection between neighboring pixels by using the shape specified gradient image. The second connection has been described in detail referring to
In operation S1203, the certain area extraction system generates a third connection between pixels excluding neighboring pixels by using the shape constraint. In this case, the third connection may be generated with respect to the pixels existing at a certain distance from each other, as described referring to
As shown in
The connection by the shape specified gradient image may be determined by using a connection 1307 between two neighboring pixels. In this case, connection strength of the connection 1307 may be determined by a final gradient, as described referring to
Finally, the connection by the shape constraint may be determined by a connection 1308 between pixels excluding neighboring pixels. Connection strength of the connection 1308 may be determined by a weight calculated via Equation 1 as described referring to
As described above, when excluding the connection whose connection strength is weak from the connections 1306, 1307, and 1308, the uncertain pixel 1303 may be determined to be the foreground pixel 1301 or the background pixel 1302.
As described above, in the method of extracting a certain area, which is described referring to
As described above, a certain area such as an object area may be more smoothly extracted from an input image by using the method of extracting the certain area from the input image by using the prior shape information, namely, the method of considering both image information and the shape information. The certain area may be extracted as a smoother and more ideal shape by using shape information as well as image information by including a distance map in the min-cut segmentation method and projecting a gradient to a norm vector of the shape information to acquire compatible edges from the shape information. In addition, the certain area may be more smoothly extracted by introducing a weight model expressing a weight map as the prior shape information.
In pre-processed images 1401, an input image in which a tri-map and an approximate image are displayed is shown. In images without shape information 1402, a result of extracting a certain area without using prior shape information is shown. As shown in
However, it may be seen that a result of extracting the certain area by using the prior shape information as shown in images with shape information 1403 is smoother and clearer than the result of 1402.
Similarly to
In this case, it may be seen that the result of extracting show in images with shape information 1503 is smoother and clearer than the result of extracting shown in images without shape information 1502.
The shape information combiner 1610 combines image information with shape information based on an input image and prior shape information. In this case, the shape information combiner 1610 may include a shape constraint generation unit 1611 to generate a shape constraint based on the input image and a shape model, a shape specified gradient image generation unit 1612 to generate a shape specified gradient image based on an approximate shape and a gradient image, and a shape specified weight image generation unit 1613 to generate a shape specified weight image based on the input image, a tri-map of the input image, and a weight model.
As described above, the shape information combiner 1610 generates the shape constraint, the shape specified gradient image, and the shape specified weight image by combining the prior shape information including the shape model expressing a contour of an object and formed of a line connecting a K number of control points and the weight model expressing a weight map indicating a probability that each pixel expressing the object corresponds to a foreground pixel or a background pixel, with the input image together with the tri-map, thereby performing a preparatory process to extract the certain area from the input image.
The certain area extractor 1620 extracts the certain area from the input image by using the image information. In this case, the certain area extractor 1620 may extract the certain area from the input image, based on the shape constraint, the shape specified gradient image, and the shape specified weight image.
In this case, the certain area extractor 1620 may include a connection generation unit 1621 to generate a connection to an uncertain pixel by using the shape constraint, the shape specified gradient image, and the shape specified weight image, a pixel determination unit 1622 to determine the uncertain pixel to be a foreground pixel or a background pixel by removing a connection whose intensity is weak, from a plurality of connections to the uncertain pixel, and an extraction unit 1623 to extract the certain area by extracting only pixels determined to be the foreground pixel from the input image.
Also, the connection generation unit 1621 may include a first connection acquirer to generate a first connection between a predetermined semantic node and a pixel by using the shape specified weight image, a second connection acquirer to generate a second connection between neighboring pixels by using the shape specified gradient image, and a third connection acquirer to generate a third connection between pixels excluding neighboring pixels by using the shape constraint.
As described above, the certain area extractor 1620 may extract the certain area from the input image by using the prior shape information via the method in which a connection of the uncertain pixel is acquired by using the shape constraint, the shape specified gradient image, and the shape specified weight image, generated by the shape information combiner 1610, a connection whose intensity is weak is excluded, the uncertain pixel is definitely determined to be the foreground pixel or the background pixel, and only the foreground pixel is extracted.
The embodiments according to the present invention may be embodied as a program instruction capable of being executed via various computer units and may be recorded in a computer-readable recording medium. The computer-readable medium may include a program instruction, a data file, and a data structure, separately or cooperatively. The program instructions and the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those skilled in the art of computer software arts. Examples of the computer readable media include magnetic media (e.g., hard disks, floppy disks, and magnetic tapes), optical media (e.g., CD-ROMs or DVD), magneto-optical media (e.g., optical disks), and hardware devices (e.g., ROMs, RAMs, or flash memories, etc.) that are specially configured to store and perform program instructions. The media may also be transmission media such as optical or metallic lines, wave guides, etc. including a carrier wave transmitting signals specifying the program instructions, data structures, etc. Examples of the program instructions include both machine code, such as produced by a compiler, and files containing high-level language codes that may be executed by the computer using an interpreter. The hardware elements above may be configured to act as one or more software modules to implement the operations of this invention.
An aspect of the present invention also provides a method and system for more smoothly extracting a certain area such as an object area from an input image by using a method of considering both of image information and shape information.
An aspect of the present invention also provides a method and system to extract a certain area, in which shape information is used as well as image information by including a distance map in a min-cut segmentation method and projecting a gradient to a norm vector of the shape information to acquire compatible edges from the shape information, thereby extracting the certain area as a smooth and ideal shape.
An aspect of the present invention also provides a method and system for more smoothly extracting a certain area by introducing a weight model expressing a weight map as prior shape information.
Although a few embodiments of the present invention have been shown and described, the present invention is not limited to the described embodiments. Instead, it would be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
Although a few embodiments of the present invention have been shown and described, it would be appreciated by those skilled in the art that changes may be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the claims and their equivalents.
Number | Date | Country | Kind |
---|---|---|---|
10-2006-0051611 | Jun 2006 | KR | national |