Patent Application
20150172513
The present invention relates generally to image capture and processing. More particularly, the invention relates to improved systems and techniques for color correction in captured images.
Photography (both still picture and motion picture) has constantly developed since its first invention, becoming less expensive and simpler throughout the decades. From Niépce's photoetchings beginning in 1822 (the earliest surviving example of which is the 1826 View from the Window at Le Gras, earlier examples having been lost), through the development by Niépce and Dauguerre of photography using silver compounds, through the use of such compounds on media such as glass and celluloid, through the development of inexpensive and widely available cameras, through instant photography continuing with modern digital photography, the ability to record still and moving images has become more and more widespread until it is now nearly universal in developed societies. Many people document every aspect of their and their friends' lives and make and distribute images of every interesting event in their lives. Much photography is casual, carried out under widely varying lighting, by photographers who are uninterested in making meticulous camera adjustments. Such photographers, however, want their photographs to look attractive and to resemble the subject, particularly in terms of such properties as color balance. Color balance is a measure of the relative intensities of each color component of an image (such as red, blue, and green primary colors). Because it is particularly important that neutral colors (such as white or gray) colors appear correctly, color balance may also be referred to as white balance or gray balance. Color constancy is a characteristic of the human visual system—the human brain provides a relatively constant color perception over a significant range of wavelength variation of light illuminating the subject. Artificial mechanisms for capturing images, on the other hand, need to incorporate mechanisms for adjustment and correction in order to achieve such constancy.
In one embodiment of the invention, an apparatus comprises at least one processor and memory storing a program of instructions. The memory storing the program of instructions is configured to, with the at least one processor, cause the apparatus to at least identify a reference object appearing in a captured scene image, compute a color transform based on known color balance information for the reference object, taken under known illumination, wherein the color transform is based on a difference between the known color balance information for the reference object and the apparent color balance of the reference object as it appears in the captured image, apply the color transform to the captured image, and create a corrected image based on the application of the color transform to the captured image.
In another embodiment of the invention, a method comprises identifying a reference object appearing in a captured scene image, computing a color transform based on known color balance information for the reference object taken under known illumination, wherein the color transform is based on a difference between the known color balance information and the apparent color balance of the reference object as it appears in the captured image, applying the color transform to the captured image, and creating a corrected image based on the application of the color transform to the captured image.
In another embodiment of the invention, a non-transitory computer readable medium stores a program of instructions, execution of which by a processor configures an apparatus to at least identify a reference object appearing in a captured scene image, compute a color transform based on known color balance information for the reference object, taken under known illumination, wherein the color transform is based on a difference between the known color balance information and the apparent color balance of the reference object as it appears in the captured image, apply the color transform to the captured image, and create a corrected image based on the application of the color transform to the captured image.
The product of the spectral power of an illuminant and the reflectance of the illuminated object together determine the color stimulus captured by a camera sensor. The illuminating light, therefore, has a significant effect on the color stimulus. For example, if the color temperature of a light source is low, an image of a scene and objects within the scene that are illuminated by the source will tend toward reddish, and if the color temperature of a light source is high, the image will tend toward bluish. Automatic white balance mechanisms attempt to estimate the illumination in a scene and correct the colors in images as if they were taken under standard illumination. Such estimation is difficult because the available information does not isolate the color temperature of the illumination, but rather is a result of the illumination and the reflectance of objects within the scene. One mechanism used by professional photographers is to carry a color checker chart and to use the chart as a reference object by taking a photograph of the chart in the same location and time as the intended photograph. An image editing program can then be used to adjust the white balance used by the camera, but this approach is cumbersome and requires considerable skill. In addition, color sheets can fade so that the actual color is unknown. If the color of the reference object is unknown, color correction based on the reference object is unreliable.
These and other problems are addressed by embodiments of the present invention, which provides mechanisms for automatic correction based on a known reference object in an image. The reference object is illuminated using a known illumination and color information for the reference object is determined. In one approach, the reference object may be photographed under the known illumination to create a reference image and color information may be extracted based on the reference image. In another approach, color information for the reference object may be measured directly, without taking an image. Any suitable mechanism for obtaining color balance information for the reference object may be used.
When a photograph is taken in which the object appears, a color transform is computed based on the image of the object as it appears in the photograph and the stored information taken from photographing the object under a known illumination. The color transform is applied to every pixel in the image to produce a color correction for the entire image, applying to the entire image the correction needed to correct the apparent color of the reference object.
The location of the reference object in the image to be corrected may be determined automatically or the user may assist in locating the reference object—for example, the user may point to the object or may mark off a region in which the object is located. It will be recognized also that the reference object may be photographed under the known illumination at any time; such photographing need not be performed before the taking of the photograph that is to be corrected. Anytime the object and a known illumination source are available, the object can be photographed under the known illumination source and data captured to inform a color correction of an existing photograph in which the reference object appears.
Thus,
A reference object 124 may be placed on the platform 106, and the object 124 may be illuminated by the illuminator 108, either independently of, or under the control of, the reference processing unit 110, depending on design and configuration choices. An image may be taken of the reference object 124 and provided to the reference processing unit 110 which may then extract and store color balance information. The image may be taken automatically under control of the processor 112, or may be taken by a user or under other automatic control separate from control of the processor 112. However the image is captured, it is at some point made accessible for color balance information extraction.
In addition, or as an alternative, color balance information for a reference object may simply be measured, without an actual image being taken. Thus, as an alternative or in addition to the camera 104, a chroma sensor 125 may be used. The chroma sensor 125 measures characteristics of light being reflected from the reference object 124 as the object 124 is illuminated by the illuminator 108, and determines color balance information for the reference object 124. It will also be recognized that the chroma sensor 125 may include its own built in illuminator and also that the particular configuration of the station 102 is chosen for convenience of illustration but the practice of embodiments of the invention is not limited to fixed positioning for the camera 104, reference object 124, and chroma sensor 125, but these devices may be arranged in any desired manner and may communicate through wire or wirelessly.
Extracted color balance information may be stored in the memory 118, and may also or as an alternative be transferred to an external data repository 126, which may be accessible through a network such as the Internet 128.
In one exemplary embodiment of the invention, the reference color information station 102 may be operated by or on behalf of an enterprise. The enterprise may package a photography product such as a camera, photography capable mobile telephone, or image editing software with a reference object that can be placed in a scene to be photographed. The object may be wearable and may be, for example, a hat or badge, or may be adapted to be held by a person in a photograph, or the object may be adapted to be placed in a scene. Such an object adapted to be held or placed in a scene may, for example, take the form of a toy or stuffed animal. A user can take a picture, insuring that the packaged reference object appears in the scene, and color balance information for the reference object being photographed can be built into the photography product or, in another embodiment, made accessible to a user of a product—for example, by connection through the Internet 128 with the repository 126. As a further alternative, reference objects such as the reference object 124 might be sold together with access to color balance information, and such reference objects might take any of numerous different forms, such as clothing, decorative objects, ornaments, or toys. Such objects may be associated with identifiers that can be provided to a photography product, such as by storing the identifier in a camera or photography-enabled telephone, or embedding the identifier in a reference object, such as in the form of a bar code or radiofrequency identifier. When the identifier is read, the photography product can retrieve the proper color balance information from its own onboard memory or from the repository 126.
The system 100 has been illustrated and discussed as a separate mechanism for obtaining reference image information in order to illustrate mechanisms that may be particularly convenient for casual users. The information obtained by a system such as the system 100 may be made available to a user with very little work or consideration from the user. It will be recognized, however, that embodiments of the present invention can be devised that give the user as much control as desired, allowing a user to select his or her own reference object or to identify an object appearing in a captured image and then to capture a reference image of that object, with information obtained from the reference image then being used to color correct the existing scene image.
Thus,
When a camera function is selected for the mobile telephone 200, a color correction module 220 may be activated. The color correction module 220 may be implemented as one or more of the programs 208, and allows for identification and location of a reference object, retrieval of color balance information for the reference object, and correction of an image in which the reference object appears, based on the color balance information.
Once the color correction module 220 is activated (for example, automatically upon selection of the camera function or through an explicit user selection), the module 220 undertakes to identify the reference object or objects being used. This may be accomplished, for example, through automatic identification or through an explicit user selection. Color balance information for the reference object or objects is identified, and access to the information is established (for example, retrieving it for local storage in the telephone 200, identifying a retrieval address, or using some other suitable mechanism for making the needed color balance information accessible for use.
When a user takes a picture, an image 222 is captured to memory 206 as part of data 210. The image 222 may be presented to a user by means of a display screen 224. The color correction module 220 may process the image to perform color correction using the color balance information in order to apply to the entire image the correction needed to achieve the correct color balance for the reference object. First, the color correction module 220 identifies the reference object in the image. This may be performed automatically or with user intervention. For example, the user may be signaled with a request to indicate the position of the reference object or to specify a region in which the reference object appears. Once the reference object has been identified, the color balance of the reference object as the object appears in the image is analyzed, and a color transform 225 (which may also be referred to as a chromatic adaptation) needed to correct the apparent color of the object is computed, with the color transform being based on the difference between the apparent color of the reference object and the known, previously measured color.
The computed color transform 225 may then be applied to pixels within the image, so that the image is thus color corrected in such a way as to achieve the correct known, measured color for the reference object. In an exemplary embodiment, the color transform 225 may then be applied to pixels within the image.
A color transform, or chromatic transform, is computed that transforms the color of the selected pixel to the measured color of the reference object. One example of a chromatic transform is the von Kries transform, but any appropriate transform may be used. In one or more embodiments of the invention, a white point conversion may be performed. If the color of the illuminant (white point) of the reference object is known, then in the LMS color space, the white point conversion can be performed as follows:
L
out
=L2/L1*Lin
M
out
=M2/M1*Min
S
out
=S2/S1*Sin
where (L2, M2, S2) are the coordinates in LMS color space of the white point corresponding to a standard illuminant (such as D65) and (L1, M1, S1) are the coordinates in LMS color space of the white point of the illuminant in the image. (L1, M1, S1) is unknown, but the measured color of the object under a known illuminant (Lout, Mout, Sout) is known, as is the input image pixel value (Lin, Min, Sin) for a selected pixel (such as a user selected pixel). This information can be used to compute the ratios L2/L1, M2/M1, and S2/S1, which completely define the transform. If the spectrum of the known illumination and the standard illumination are known, a similar transform can be used to go from the known illumination to the standard illumination of D65.
A corrected image 226 may then be held in memory 206 for presentation to the user, and may also be stored in long term storage, either automatically or in response to a command from the user.
It will be recognized that more than one reference object may be used, and a captured image may include images of multiple reference objects. Multiple reference objects may be photographed under known illuminations and color information stored for each object. If multiple reference objects appear in a single image of a scene, a color transform may be computed for each object and a combined estimate may be computed for the transforms in order to bring the overall image into conformity with the color of the known reference objects. It will be recognized that an object need not be specifically identified as a reference object before an image in which the object appears is captured. If an object appearing in an image can be photographed under known illumination to create a reference image, the information obtained from the reference image can be used for color correction at any time, so long as the object has the same coloring at the time the reference image is captured and the time the scene image is captured. Thus, a reference object may be photographed and a reference image created by a photography product manufacturer or vendor, or a user may, before capturing a scene image, photograph a reference object to create a reference image. Alternatively, if a user has access to an object appearing in an image, the user may photograph that object at any convenient time in order to create a reference image, and the reference image may be used for color correction for any image in which the reference object appears. Once the reference image is captured and stored, it may be used to perform color correction for existing or subsequently captured images, with the reference object being located in an image and a color transform being performed to correct the image based on the difference between the apparent color and the measured color of the reference object.
It will be understood that a reference image station 102 is described in connection with
The system 200 may also include a self-contained reference information capture device 270, comprising an illuminator 272 and an imaging sensor 274, a chroma sensor 275, or both. The reference information capture device 270 may also comprise a processor 276, memory 278, data 280, programs 282, and an interface 284, which may suitably be a wireless interface. The reference information capture device 270 can be used to illuminate an object and sense the color of the object, providing color information to, for example, the reference station 230 or the telephone 201. The reference information capture device 270 can be used to capture true color information for reference objects conveniently, and a user can easily illuminate an object and perform color capture on the object under the known illumination before or after taking a photograph. A user might, for example, photograph a scene, observe that the color cast of the scene is unsatisfactory, and choose an object appearing in the scene to use as a reference object. The true color information can be supplied to any device used to process a captured scene image to provide a corrected image (by identifying the reference object within the captured image, computing a color transform needed to color correct the image of the reference object, and applying the color correction to every pixel within the captured scene image), and can also be stored for later use. In one approach, the device 270 may use the imaging sensor 274 to capture an image of the reference object, which may then be processed to extract color balance information. In another approach, the device 270 may use the chroma sensor 275 to measure color values of the reference object and process the measurements to determine and store color balance information. Any other suitable approach may be used to determine the reflectance of the reference object. This information can be stored in the memory 278 and provided to any desired device—for example, using the interface 284.
Any number of objects may be used as reference objects, but some particularly useful choices might be clothing. For example, a user might capture reference images of favorite clothing items, such as shirts or hats, and wearing such items in photographed scenes would make it easy to perform nearly automatic color correction. A photography-enabled device could be made aware that specified clothing items would appear in a scene, and could identify the specified items and based color correction on the known color of the items. A device such as the reference image capture device 270 would be particularly suitable for capture of such reference images, being configured to both provide a known illumination and capture color information. A device such as the device 270 would suitably be of a size to be handheld and easily operated by a user. In addition, the device 270 may, as noted above, have its own image capture capabilities, and these capabilities may be used to capture a scene. The device 270 may also use its reference object color balance determination capabilities to perform color correction for its own scene images.
Referring again to
In general, the various embodiments of the telephone 201 or the reference image capture device 270 can include, but are not limited to personal portable digital devices having wireless communication capabilities, including but not limited to cellular telephones, navigation devices, laptop/palmtop/tablet computers, digital cameras and music devices, and Internet appliances.
Various embodiments of the computer readable memories include any data storage technology type which is suitable to the local technical environment, including but not limited to semiconductor based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, removable memory, disc memory, flash memory, DRAM, SRAM, EEPROM and the like. Various embodiments of the processors include but are not limited to general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and multi-core processors.
Electronic devices implementing these aspects of the invention need not be the entire devices as depicted at
While various exemplary embodiments have been described above it should be appreciated that the practice of the invention is not limited to the exemplary embodiments shown and discussed here. Various modifications and adaptations to the foregoing exemplary embodiments of this invention may become apparent to those skilled in the relevant arts in view of the foregoing description. It will be further recognized that the various blocks illustrated in
Further, some of the various features of the above non-limiting embodiments may be used to advantage without the corresponding use of other described features.
The foregoing description should therefore be considered as merely illustrative of the principles, teachings and exemplary embodiments of this invention, and not in limitation thereof.
