Patent Grant
7158178
This disclosure relates to color conversion and, more particularly, to converting from a sub-sampled color image in a first color space format to a full color image in a second color space format.
As is well-known, in a variety of circumstances, it is a desirable to apply a technique to convert from a sub-sampled color image to a full color image. For example, for a camera or other imager that has the capability of creating a color image, typically it is too expensive to have the capability to capture three independent color pixel signal values for each pixel location of an image of a particular size or dimension. Therefore, more typically, a sub-sampled color image is captured and the missing color pixel signal values at each pixel location are computed using a variety of techniques. In other words, each pixel location may have a single color signal value rather than three independent color signal values, for example. A need continues to exist for techniques for creating a full color image of good quality from a sub-sampled color image.
The subject matter regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification. The invention, however, both as to organization and method of operation, together with objects, features, and advantages thereof, may best be understood by reference to the following detailed description when read with the accompanying drawings in which.
In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present invention.
As previously indicated, for a color image, typically each pixel signal value is represented by three independent color components, such as a red component, a green component, and a blue component, for example. However, the invention is not limited in scope to these color components or to this particular color space format. Ideally, three image sensors would be used to capture the three color components. However, the use of three sensors in a single image capture device is typically expensive and also may not be suitable for use in low power handheld devices, such as a mobile handset designed to display such color images, for example.
In order to reduce cost, typically the color image is captured in a sub-sampled form using a single sensor in each pixel location with a suitable color filter array (CFA) “on top” of the sensor. Of course, the invention is not limited in scope to use with such an image capture device, however. Nonetheless, in such an approach, each pixel location comprises only one color component signal value, thereby forming a type of sub-sampled color image. For example, in one approach, each pixel signal value may comprise either a red color signal value, a green color signal value, or a blue color signal value represented by eight binary digital signals or bits. Although the invention is not limited in scope in this respect, one such pattern is a Bayer pattern, as illustrated schematically in FIG. 1.
Of course, for a color image formed using three color sensors in each pixel location, each location would include signal information for all three independent color components. Therefore, typically, for a color image captured by the latter device, each pixel is represented by a 24 bit color signal, 8 bits for each of the red, green, and blue color components, for example. Again, the invention is not restricted in scope by this example; however, in order to produce such an image from a device employing one color sensor in each pixel location, for example, signal interpolation would typically be employed. In one example of signal interpolation, the two missing color components for each pixel location are obtained by using color signal information provided by neighboring pixel signal values.
If a full color image is obtained in this manner, a transformation into a different color space format may then be employed. For example, it may be desirable to comply with certain image processing specifications and/or standards, such as those provided by the International Telecommunications Union (ITU) or the International Standards Organization (ISO), although the invention is not limited in scope in this respect. Likewise, it is also typical to represent color images in a luminance-chrominance (L-C) form in order to reduce the storage requirements by sub-sampling redundant color signal information that may be present in the image, typically in the chrominance color components. One widely used L-C color space format is the Y-Cr-Cb 4:2:0 representation, although, again, this is just an example.
Disadvantages of current techniques to produce a full color image from a sub-sampled color image include (1) the loss of signal information as a result of applying the particular technique and/or (2) high computational complexity, which also may typically consume excessive power when employed or performed by a computing engine, and/or (3) internal buffering large enough to store the signal information during and after processing. These disadvantages will be explained in more detail hereinafter. An embodiment of converting from a sub-sampled color image to a full color image in accordance with the present invention may avoid these disadvantages.
As is well-known, a color interpolation method or process, such as described in “Method of Interpolating Color Pixel Signals from a Subsampled Color Image,” by Acharya et al., filed on Oct. 1, 1999, U.S. patent application Ser. No. 09/410,800, assigned to the assignee of the present invention, for example, generates a full color digital image comprising three color planes, red, green, and blue, each of size M×N from a Bayer pattern sub-sampled image by generating the missing two color components for each pixel location. Of course, this is just one example of a color interpolation process, and the invention is not limited in scope to this or any other particular approach. There are a wide number of color interpolation techniques that may be employed. Typically, the color signal information of the neighboring pixels are employed to generate the missing color signal information for each pixel. A simple approach to implement would be to employ the median of the neighboring pixels. Of course, more complex approaches may also be employed.
Subsequently, for the approach illustrated in
Y(I,J)=0.299R(I,J)+0.587G(I,J)+0.114B(I,J)
Cr(I,J)=−0.169R(I,J)−0.331G(I,J)+0.500B(I,J)
Cb(I,J)=0.500R(I,J)−0.419G(I,J)−0.081B(I,J) [1]
where R,G, and B denote different color planes in the RGB color space format, Y, Cr, and Cb denote different color planes in the Y-Cr-Cb 4:2:0 color space format, and I,J denote pixel locations.
The Cr and Cb image planes are then, in this particular approach, decimated by dropping alternate rows and columns from each chrominance plane (Cb and Cr), as illustrated by Cr′ or Cb′ in FIG. 2. This generates the Y-Cr-Cb image in 4:2:0 color space (12 bit color) format. The reason for 12 bits for each pixel in Y-Cr-Cb 4:2:0 color space is that for every 4 luminance (Y) pixels (1 byte each for each luminance value) in this format there is 1 value (1 byte each) for each corresponding chrominance channel (Cb and Cr in this example). Hence, 4 full color pixels in this format are represented by 6 bytes or 48 bits. As a result, each pixel in Y-Cr-Cb color space format is considered to have 12 bits.
Assume that the average number of operations to recover each missing color signal value in a particular pixel location is “K”, when using a known color interpolation technique. Then, to recover two missing color space components for each pixel location, the total number of operations to restore the full color image using this approach is 2K×(M×N), where “×” denotes multiplication. The total number of multiplications for color conversion is 9M×N, and the number of additions is 6M×N. Therefore, ignoring the computation for sub-sampling, the typical number of operations for color interpolation followed by color conversion is more than (2K+15)×M×N. Likewise, the memory buffer size employed for hardware and/or software implementation of this approach is 3M×N.
In this context, the term upscaled refers to a process applied to a color plane of a color dimension in a particular color space format whereby the color signal information in the color plane is retained, but spread or distributed over a color plane of larger dimensions. For example, for the embodiment illustrated in
To illustrate this particular embodiment in greater detail,
R(I,J)=R
G(I,J)=(G1+G2)/2
B(I,J)=B [2]
where R, G1, G2, and B are from the block B(I,J).
As a result of this transformation, a Y-Cr-Cb color image of size M/2×N/2 is obtained because each 2×2 Block produces a single pixel signal value in this color space format. The Y plane may then be upsampled to generate the M×N size Y-Cr-Cb 4:2:2 color space format image. One technique for performing this upscaling is described in greater detail in a prior patent application “A DWT-based Up-Sampling Algorithm Suitable For Image Display In An LCD panel,” by Acharya, filed on Aug. 5, 1998, U.S. patent application Ser. No. 09/129,728, now U.S. Pat. No. 6,236,765, assigned to the assignee of the present invention, although the invention is not limited in scope in this respect. As previously indicated any one of a number of techniques to perform the upscaling may be employed.
It is noted that the computational complexity of this approach and the amount of storage space utilized is significantly less than the previous approach. As discussed above, the total number of multiplications for an integrated color interpolation and color conversion is 9×(M/2)×(N/2) or (9/4)M×N and a similar number of additions and subtractions. Hence, the total number of operations is 4.5 M×N. In this particular embodiment, for an upscaling operation for the Y plane, the computational complexity is similar to that of color interpolation; however, the upscaling is applied to data of size M/2×N/2. Hence, for upscaling the Y plane, the total number of operations is K×M×N. Hence, the total number of operations for this particular embodiment is (K+4.5)×M×N, which is less than half of the approach previously described in terms of computational complexity. Likewise, the storage space employed to implement this approach is 1.5M×N or about half of the approach previously described.
As previously indicated, an embodiment of a method of converting from a sub-sampled color image in a first color space format to a full color image in a second color space format in accordance with the invention provides a number of advantages. The color interpolation approach is integrated with the color conversion methodology. As a result, a computationally complex color interpolation scheme is avoided totally with the trade off being the addition of an upscaling scheme applied to a reduced set of data. Because of this integrated approach, pipelining the imaging processing flow, as is desirable in implementing video, for example, is relative straightforward and useful for such applications. The reduction in computational complexity is more than 50%, and in memory storage space is about 50%. Therefore, this approach may be applied to virtually any digital signal processing architecture or similar computing engine and may reduce computational complexities suitable for real-time application and low power applications, such as may be desirable for handheld devices, with generally no significant loss of signal information.
It will, of course, be understood that, although a particular embodiment has just been described, the invention is not limited in scope to a particular embodiment or implementation. For example, one embodiment may be in hardware, whereas another embodiment may be in software. Likewise, an embodiment may be in firmware, or any combination of hardware, software, or firmware, for example. Likewise, although the invention is not limited in scope in this respect, one embodiment may comprise an article, such as a storage medium. Such a storage medium, such as, for example, a CD-ROM, or a disk, may have stored thereon instructions, which when executed by a system, such as a computer system or platform, or an imaging system, may result in a method of upscaling a color image in accordance with the invention, such as, for example, the embodiment previously described.
While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.
This patent application is related to U.S. Pat. No. 6,628,827, titled “Method of Upscaling A Color Image,” by Acharya, assigned to the assignee of the current invention and herein incorporated by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4642678 | Cok | Feb 1987 | A |
| 5014134 | Lawton et al. | May 1991 | A |
| 5321776 | Shapiro | Jun 1994 | A |
| 5392255 | LeBras et al. | Feb 1995 | A |
| 5398066 | Martinez-Uriegas et al. | Mar 1995 | A |
| 5406306 | Siann | Apr 1995 | A |
| 5428465 | Kanamori | Jun 1995 | A |
| 5491561 | Fukuda | Feb 1996 | A |
| 5541653 | Peters et al. | Jul 1996 | A |
| 5574572 | Malinowski | Nov 1996 | A |
| 5602589 | Vishwanath et al. | Feb 1997 | A |
| 5706220 | Vafai et al. | Jan 1998 | A |
| 5729691 | Agarwal | Mar 1998 | A |
| 5737448 | Gardos | Apr 1998 | A |
| 5832120 | Prabhakar | Nov 1998 | A |
| 5875122 | Acharya | Feb 1999 | A |
| 5892847 | Johnson | Apr 1999 | A |
| 5901242 | Crane et al. | May 1999 | A |
| 5995210 | Acharya | Nov 1999 | A |
| 6002446 | Eglit | Dec 1999 | A |
| 6009201 | Acharya | Dec 1999 | A |
| 6009206 | Acharya | Dec 1999 | A |
| 6047303 | Acharya | Apr 2000 | A |
| 6091851 | Acharya | Jul 2000 | A |
| 6094508 | Acharya et al. | Jul 2000 | A |
| 6108453 | Acharya | Aug 2000 | A |
| 6124811 | Acharya et al. | Sep 2000 | A |
| 6125201 | Zador | Sep 2000 | A |
| 6130960 | Acharya | Oct 2000 | A |
| 6151069 | Dunton et al. | Nov 2000 | A |
| 6151415 | Acharya et al. | Nov 2000 | A |
| 6154493 | Acharya et al. | Nov 2000 | A |
| 6166664 | Acharya | Dec 2000 | A |
| 6178269 | Acharya | Jan 2001 | B1 |
| 6195026 | Acharya | Feb 2001 | B1 |
| 6208350 | Herrera | Mar 2001 | B1 |
| 6215908 | Pazmino et al. | Apr 2001 | B1 |
| 6215916 | Acharya | Apr 2001 | B1 |
| 6229578 | Acharya et al. | May 2001 | B1 |
| 6233358 | Acharya | May 2001 | B1 |
| 6236433 | Acharya et al. | May 2001 | B1 |
| 6236765 | Acharya | May 2001 | B1 |
| 6259819 | Andrew et al. | Jul 2001 | B1 |
| 6275206 | Tsai et al. | Aug 2001 | B1 |
| 6285796 | Acharya et al. | Sep 2001 | B1 |
| 6292114 | Tsai et al. | Sep 2001 | B1 |
| 6301392 | Acharya | Oct 2001 | B1 |
| 6348929 | Acharya et al. | Feb 2002 | B1 |
| 6356276 | Acharya | Mar 2002 | B1 |
| 6366692 | Acharya | Apr 2002 | B1 |
| 6366694 | Acharya | Apr 2002 | B1 |
| 6373481 | Tan et al. | Apr 2002 | B1 |
| 6377280 | Acharya et al. | Apr 2002 | B1 |
| 6381357 | Tan et al. | Apr 2002 | B1 |
| 6392699 | Acharya | May 2002 | B1 |
| 6411333 | Auld et al. | Jun 2002 | B1 |
| 6535648 | Acharya | Mar 2003 | B1 |
| 6563537 | Kawamura et al. | May 2003 | B1 |
| 6563538 | Utagawa | May 2003 | B1 |
| 6570616 | Chen | May 2003 | B1 |
| 20010019630 | Johnson | Sep 2001 | A1 |
