The present principles relate to a method and an apparatus for image colorization, and more specifically to a method and an apparatus for image colorization using an auxiliary image.
In the field of image colorization two different types of color correction are discerned. In contrast to primary color correction, which generally affects the whole image, secondary color correction aims to correct the colors in only a sub-region of the image. This process is widely deployed in the movie industry.
In [1] a method for (re-) colorization of images is described. The method is based on the assumption that two neighboring pixels should receive similar colors when their intensities/original colors are similar. Although this approach yields results of high quality, the employed global optimization method yields a large, sparse system of linear equations that is computation intensive to solve.
A method for (re-) colorization of images based on a fast edge-preserving smoothing filtering approach is described in [2]. The filtering method basically only works for one-dimensional signals. Two-dimensional images need to be processed iteratively by alternating between horizontal and vertical filtering steps. Such an iterative approach does not work satisfactory if there are thin structures present in the image, like, for example, hair or fur. Furthermore, strong edges, no matter how thin they are, or even just outliers or noise, block the filtering completely. For some applications, this behavior is desirable, but not for others.
The present principles relate to a solution for image colorization using an auxiliary image.
According to one aspect of the present principles, a method for image colorization comprises:
Accordingly, a computer readable storage medium has stored therein instructions enabling image colorization, which when executed by a computer, cause the computer to:
The computer readable storage medium is a non-transitory volatile or non-volatile storage medium, such as, for example, but not limited to, a hard disk, an optical or magnetic disk or tape, a solid state memory device, etc. The storage medium thus tangibly embodies a program of instructions executable by a computer or a processing device to perform program steps as described herein.
Also, in one embodiment, an apparatus for image colorization comprises:
In another embodiment, an apparatus for image colorization comprises a processing device and a memory device having stored therein instructions, which, when executed by the processing device, cause the apparatus to:
One goal of the present solution is to colorize or re-colorize the object of interest under preservation of its (potentially fine) structures. A unique feature of the Guided Image Filter [3] is its “structure transfer” capability; i.e. the ability to transfer structures (color gradients) from the Guiding Image to the filter output. Therefore, a Guided Image Filter, preferably a Confidence-Aware Guided Image Filter [4], is used for (re-)colorization of images. In the process according to the present principles, the original image serves as the Guiding Image, while an additional auxiliary image is filtered, which contains information about the desired new color(s). In particular, the auxiliary image comprises information about a first type of pixels, for which a target color is specified, and a second type of pixels, for which no target color is specified. The auxiliary image may be a user-modified version of the original image.
The present solution is particularly suited for cases where the color of the object of interest, e.g. a foreground object, mixes with its surrounding, e.g. the background. This is typically the case for objects with fine structures, like hair or fur. For such cases the present solution outperforms the state of the art approaches mentioned above.
In one embodiment, for the first type of pixels, the target color of a pixel is one of the color of the pixel in the original image and a new color. There is no constraint on the target color, i.e. it can be different from or identical to the original color.
In case of a Confidence-Aware Guided Image Filter, a confidence or weight of ‘0’ is assigned to a pixel without a specified target color and a confidence or weight equal to or larger than ‘0’ is assigned to a pixel with a specified target color. In this way the per-pixel weights or confidences are set such that local linear models of the filter are learnt from the desired colors.
In one embodiment, a confidence larger than ‘0’ is assigned to a pixel having the color of the pixel in the original image as a specified target color if a distance of the pixel to another pixel without a specified target color is smaller than a distance threshold. For the set of specified pixels that carry their original color, only pixels at a distance to an unspecified pixel smaller than some distance threshold obtain an additional weight greater than ‘0’, while all other weights are set to ‘0’. This excludes pixels further away from the unspecified region from the learning process in the filter, which may otherwise mislead the local linear models.
In one embodiment, a refined auxiliary image is generated by:
Once the alpha matte has been obtained, a number of unspecified pixels may be converted into specified ones. Use of the refined auxiliary image allows further improving the outcome of the colorization process.
In one embodiment, assigning target colors to pixels of the auxiliary image without a specified target color based on the alpha matte comprises:
The lower alpha threshold may be in the vicinity of ‘0’ and the upper alpha threshold may be in the vicinity of ‘1’. This process reduces the size of the unspecified region by pre-classifying pixels according to their estimated opacity.
The present principles further relate to a solution for generating an auxiliary image for an image colorization process.
According to a further aspect of the present principles, a method for generating an auxiliary image for an image colorization process comprises:
Accordingly, a computer readable storage medium has stored therein instructions enabling generating an auxiliary image for an image colorization process, which when executed by a computer, cause the computer to:
Also, in one embodiment, an apparatus for generating an auxiliary image for an image colorization process comprises:
In another embodiment, an apparatus for generating an auxiliary image for an image colorization process comprises a processing device and a memory device having stored therein instructions, which, when executed by the processing device, cause the apparatus to:
According to this further aspect of the present principles a special type of auxiliary image is generated, which is particularly suited for secondary color correction, i.e. (re-)colorization of individual objects. The auxiliary image distinguishes between different types of image regions. For pixels in one image region the target color is unspecified and to be computed. For pixels in another image region the target color is specified.
for image colorization according to the present principles;
For a better understanding of the principles, example embodiments are explained in more detail in the following description with reference to the figures. It is understood that the present solution is not limited to these exemplary embodiments and that specified features can also expediently be combined and/or modified without departing from the scope of the present principles as defined in the appended claims.
One embodiment of an apparatus 20 for image colorization according to the present principles is schematically depicted in
Another embodiment of an apparatus 30 for image colorization according to the present principles is schematically illustrated in
For example, the processing device 31 can be a processor adapted to perform the steps according to one of the described methods. In an embodiment according to the present principles, said adaptation comprises that the processor is configured, e.g. programmed, to perform steps according to one of the described methods.
A processor as used herein may include one or more processing units, such as microprocessors, digital signal processors, or a combination thereof.
The local storage unit 22 and the memory device 32 may include volatile and/or non-volatile memory regions and storage devices such as hard disk drives and DVD drives. A part of the memory is a non-transitory program storage device readable by the processing device 31, tangibly embodying a program of instructions executable by the processing device 31 to perform program steps as described herein according to the present principles.
According to the present principles, an auxiliary image is used for an image colorization process applied to an original image. One exemplary original image is depicted in
One example of an auxiliary image, which is commonly used also in other application scenarios, is the original image overlaid with scribbles provided by the user. In this case, the scribbled pixels carry the specified color, while all remaining pixels are unspecified.
A different, rather uncommon example of an auxiliary image that is particularly useful for the present purpose is shown in
For the filtering method itself, it is not relevant what type of auxiliary image is actually being used. It is sufficient if the auxiliary image distinguishes between the two types of pixels.
The present principles do not aim to alter the luminance of the original image. Therefore, a color model that represents each color by a luminance value together with two color values is used. Two such examples are the YCbCr and the CIE Lab color models. The luminance channel of the original image may be adopted for the output image.
Each color channel of the auxiliary image is filtered by a Guided Image Filter or a Confidence-Aware Guided Image Filter. In this process, the original image serves as the guiding image. In contrast to most other image processing filters, the output of the Guided Image Filter [3, 4] is not a direct function of the filter input, but of a third image, the guiding image. Note the distinction between the “filter input”, which is the image containing the user input, and the “original image”, which will serve as the guiding image. For each input pixel to be filtered, an individual linear transformation function is computed. The function parameters are learnt by minimizing the squared error between the filter input and the filter output using linear regression. The transformation is finally used to turn a pixel's color in the guiding image into the filter output. The goal of the present solution is to re-colorize the object of interest under preservation of its (potentially fine) structures. Therefore, the Guided Image Filter is ideally suited for the desired color transformation. The local linear models may be learned by considering the colors of pixels for which the target color is specified, e.g. by user scribbles. The models are then applied for assigning new colors to the set of unspecified pixels.
A remaining question is how to set the additional per-pixel weights supported by the filter, i.e. the “confidence”. In one embodiment, these weights are set to ‘0’ for pixels with an unspecified target color. Pixels with a specified target color may generally obtain a larger weight, e.g. a weight of ‘1’. In one embodiment, for the set of specified pixels that carry their original color, only pixels at a distance to an unspecified pixel smaller than some distance threshold obtain an additional weight greater than ‘0’, while all other weights are set to ‘0’. This excludes pixels further away from the unspecified region from the learning process in the filter, which may otherwise mislead the local linear models. During the filtering, a local window should not cover only pixels with an additional per-pixel weight of 0. This can be avoided by choosing a sufficiently large filter kernel. Additionally, a small value of ε instead of 0 can be used for the weights.
In one embodiment, the user scribbles are extended into the unspecified region in a pre-processing step as shown in
Returning to
The present solution can also be applied for colorization of grayscale images.
One embodiment of an apparatus 40 for generating an auxiliary image for an image colorization process according to the present principles is schematically depicted in
Another embodiment of an apparatus 50 for image colorization according to the present principles is schematically illustrated in
For example, the processing device 51 can be a processor adapted to perform the steps according to one of the described methods. In an embodiment said adaptation comprises that the processor is configured, e.g. programmed, to perform steps according to one of the described methods.
A processor as used herein may include one or more processing units, such as microprocessors, digital signal processors, or a combination thereof.
The memory device 52 may include volatile and/or non-volatile memory regions and storage devices such as hard disk drives and DVD drives. A part of the memory is a non-transitory program storage device readable by the processing device 51, tangibly embodying a program of instructions executable by the processing device 51 to perform program steps as described herein according to the present principles.
Number | Date | Country | Kind |
---|---|---|---|
15305589.2 | Apr 2015 | EP | regional |