Patent Application
20080303918
The present invention relates to color correcting for ambient light and, more particularly, to a device and method for adjusting the white balance point of a display based on ambient lighting conditions.
Colors appear differently to the human eye under different ambient lighting conditions. This has traditionally presented a problem for photographers and videographers. With respect to digital still cameras (DSCs), for example, a photographer may capture an image under one light source and then view a verification image on a display. Without some color correction applied to the verification image, the colors in the verification image may appear different than the colors in the image when it is later printed or developed. Similarly, different films and developing techniques may alter the appearance of the colors from the appearance of the colors viewed while the picture was originally taken. With respect to video cameras, for example, without some color correction applied to a captured video, the colors in the video may appear differently when played back on a display than they did when the videographer originally captured the video.
The appearance of colors in images typically depends on the white point of the image. The appearance of colors may be made consistent for some devices, such as the DSCs and video cameras described above, by setting a single white balance point for the device. The white balance point of an image is the definition of the color “white” for the image and is typically defined by the “color temperature” of the illuminant. One such illuminant may be daylight. The white point corresponding to daylight, for example, may be expressed according to the relative intensities of different colors of light that make up daylight or as the color temperature of daylight. For example, daylight may be expressed according the relative intensities of red, green and blue light that make up daylight. Alternatively, daylight may be expressed according to its color temperature, which is approximately 5000K.
Color temperature is a characteristic of visible light and may be determined by comparing the hue and brightness of visible light to a theoretical heated black-body radiator. The temperature in degrees Kelvin at which the black-body radiator matches the hue of the visible light is the color temperature of the visible light. For visible light that does not match the temperature of a black-body radiator, the color temperature of the visible light is referred to as the correlated color temperature of the visible light. The correlated color temperature is the color temperature of the black-body radiator that is closest to the hue and brightness of the visible light.
Conventionally, the white point of a camera may be periodically calibrated to define white relative to the appearance of a white target under ambient lighting conditions. This may be done by placing a white target in the field of view of the camera and adjusting the white point of the pixels corresponding to the target to the fixed white point. That is, the white point for the camera may be set by transforming the color temperature of the white pixel in the captured image to the color temperature of the camera's white point and adjusting the other color pixels in the image proportionately. Thus, when the video is reproduced on a standard display under ideal viewing conditions, it will appear the same no matter what ambient lighting was used during the image capture.
These and other features, aspects, and advantages of the embodiments of the present invention discussed below will become more fully apparent from the following description, appended claims, and accompanying drawings in which the same reference numerals are used for designating the same elements throughout the several figures, and in which:
Lighted displays, and especially portable lighted displays, present a different problem. These displays are viewed in ambient illumination. Thus, if a display has a fixed white point corresponding to sunlight but is viewed in an environment with fluorescent lighting, the colors on the display will appear to be different from the same colors in the ambient environment. This is especially true in brightly lighted environments. In dimly lighted environments, in which the display is significantly brighter than the environment, the color temperature of the ambient light may not be as significant a factor when viewing the display.
By way of example, suppose a photographer in the art department for IPOD takes a picture of a white IPOD in an illuminated viewing booth. The photographer subsequently displays the image on the IPOD and views the displayed image in the viewing booth under the same illumination under which the image was captured. To the photographer, the white in the image perfectly matches the white of the IPOD's casing. The next morning, the photographer carries the IPOD outside and views the image again. This time, the white IPOD in the image may appear bluer than it was in the viewing booth. The displayed image no longer matches the IPOD's actual casing. The photographer then carries the IPOD back to the office. This time, the white IPOD in the image may appear redder than it was in the viewing booth. The photographer immediately alerts the IPOD production department to let them know the IPOD display is defective.
The photographer is wrong. The display is not defective, it is just not ideal. The change in the appearance of the colors is due to differences in the illumination. For example, when outside in the morning, the illumination has relatively more red light than the illumination in the viewing booth. This causes the coloring of the case to have a higher relative red content than the image of the IPOD on the display. Because most of what the photographer sees is dominated by the surroundings and not by the small display, the display may appear slightly blue. The opposite occurs under fluorescent illumination, which has relatively more blue than the illumination in the viewing booth. This causes the coloring of the case to have a higher blue content than the image of the IPOD on the display. Again, because most of what the photographer sees is dominated by the surroundings and not by the small display, the display may appear slightly red.
The example embodiments of the present invention, described below, mitigate this problem by matching the white point of the lighted display to the sensed ambient lighting. Thus, similar colors will not appear to be different on the display and in the environment. With respect to the IPOD example, using the example embodiments of the present invention described below, the white IPOD on the display will appear the same in the viewing booth, outside, in the office, at home and anywhere else the photographer may view the image of the white IPOD displayed by the white IPOD itself.
While the examples described below concern lighted displays, it is contemplated that the invention may also be practiced with emissive displays such as organic light emitting diode (OLED) displays, plasma displays or field emissive displays (FEDs). For emissive displays, the white balance point may be set by adjusting the color processing circuitry.
In one embodiment of the present invention, the example system of
At step 200 of
Ambient light sensor 90 may be any RGB or other color sensor or imager. One suitable RGB sensor may include at least three pixels, although it may include an array of many pixels. Each pixel may include a photosensitive element and a color filter. At least one of the pixels may be a red pixel, for example, having a red filter disposed over it, another one of the pixels may be a green pixel, for example, having a green filter disposed over it and another one of the pixels may be a blue pixel, for example, having a blue filter disposed over it. The red, green and blue filters may function to pass only light having a wavelength corresponding to the assigned color and to reflect or absorb all other wavelengths. For example, the red filter may pass a band of light centered at a wavelength of 650 nm, the green filter may pass a band of light centered at a wavelength of 510 nm and the blue filter may pass a band of light centered at a wavelength of 475 nm. The passed light may enter the photosensitive element and the photosensitive element may produce a signal proportional to the intensity of the light striking the photosensitive element. The signal may then be read from each red, green and blue pixel and may eventually be converted to a digital signal representing the relative intensities of the colors red, green and blue in the ambient lighting. Using these signals, the color temperature of the ambient light may be determined. While this suitable sensor detects different colors using color filters disposed over the pixels, other sensors may separate colors using other mechanisms such as prisms or diffraction gratings. Such other sensors may also be suitable for use as ambient light sensor 90.
Avago Technologies' APDS-9002 sensor may, for example, be adapted for use as ambient light sensor 90. For example, disposing color sensors over at least three pixels of the APDS-9002 may form an excellent ambient light sensor 90 due to its responsivity being close to the response of the human eye.
At step 210 of
As described above, image data provided to the display has been transformed to a fixed white point by the camera system used to obtain the image data. The color temperature calculator determines the correction needed to transform images referenced to this fixed white point to the white point corresponding to the ambient illumination.
The reference values stored in EEPROM 110 may be stored, for example, in a lookup table (LUT). An example LUT is shown in Table 1 below. This LUT includes white point reference values in the RGB color space. It is contemplated, however, that these values may be, for example, in the CE XYZ tristimulus color space, the Long, Middle and Short (LMS) color space which mimics the cone response of the human eye, or any other color space. As shown, the LUT may include three columns, each corresponding to a separate one of the RGB coordinates. Each row in this table corresponds to a respectively different illuminant, in this example, incandescent light, moonlight and daylight. In this example, it is assumed that the fixed white-point of the image data corresponds to daylight. Thus, the values R3, G3 and B3 correspond to the white point of the received data. Where the correction is applied to a light source of a lighted display, the values from the table may be used to directly modify the Red, Green and Blue light sources. Where the correction is applied to the color signals for an emissive display or for a lighted display having a white light source, the color temperature calculator may define a transformation for the R, G and B image signals from the fixed white point to the calculated ambient white point.
One simple method for transforming an image from the daylight white point to a white point corresponding to incandescent light is to multiply the received R color signal by R1/R3, the received G color signal by G1/G3 and the received B color signal by B1/B3.
This transformation, however, may result in erroneous colors. Alternatively, the EEPROM 110 may be programmed with multiple color transformation tables, one for each of a set of fixed ambient light conditions. Each of these tables may be used to program a memory, for example, in the drive circuitry 40, which transforms the R, G and B signals provided by the processor 20 into R, G and B signals corresponding to the white point of the sensed ambient illuminant. Each of these tables may, for example, receive three 8-bit address values, corresponding to the R, G and B color values for a pixel and provide transformed 8-bit R, G and B values.
Where the ambient illuminant does not match one of the illuminants in the LUT, the appropriate color values may be interpolated. For a lighted display, the drive signals for the Red, Green and Blue light sources may be interpolated between appropriate pairs of the R, G and B values in the table. For emissive displays or lighted displays having a white light source, transformation tables may be interpolated from the appropriate transformation tables stored in the EEPROM 110.
As an alternative to using the transformation tables, the white point transformations may be accomplished using data processing circuitry in the drive circuitry 40 or processor 20. These circuits may be programmed, for example, to implement a transform from the white value of a display to the white value of the ambient light. One simple example transformation includes converting the image illuminant to a linear space, multiplying each component by the ratio of the ambient light value to the reference white value in the converted color space and then converting the converted display values back to the display's color space. For example, if the display's color space is sRGB (standard RGB), the white value of the display may first be converted to a linear space by removing the gamma correction from the sRGB signal or by converting the sRGB signals to an XYZ color space. Converting from one color space to another, such as converting from the sRGB color space to an XYZ color space, is well known in the art. Next, each component is multiplied by the ratio of the ambient light white value to the reference white value in the XYZ color space, such as by the following equations: Xdisplay=Ximage*(Xambient/Xreference); Ydisplay=Yimage*(Yambient/Yreference); and Zdisplay=Zimage*(Zambient/Zreference). This may be stated as the following matrix equation.
Display values (Xdisplay, Ydisplay, Zdisplay) corrected for viewing conditions are generated from the image values (Ximage, Yimage, Zimage) that are based on a standard reference value. If, for example, the ambient illumination and reference illumination are identical, the matrix may be an identity matrix and, accordingly, the display values may be identical to the image values.
The converted values may then be converted back to the sRGB color space. Where saturation is a concern, each of the three ratios described above may be scaled by the same factor so that the largest ratio is 1. An example scaling factor may be represented by the following equation equation: scaling factor=1/maximum (Xambient/Xreference, Yambient/Yreference, Zambient/Zreference).
While the above example is described in terms of a conversion from the sRGB color space to the XYZ color space, conversion between many color spaces are well known in the art and applicable for programming the drive circuitry. Additionally, the above example describes a simple conversion from one color space to another. More complicated conversions that may also account for brightness differences, for example, are also well known in the art. An example of such a conversion may be a von Kries transform. This transform is based on the matrix described above and may be adapted so that a scale value other than 1 may affect more than a single X, Y or Z value.
While ambient light sensor 90 detects red, green and blue light, it is contemplated that two of the three colors may be adjusted relative a stable third color to adjust the white balance point for the brightest instance of ambient light. If, however, it is desirable to adjust the white balance point for less bright instances of ambient light, a third adjustment value may be included in the LUT for adjusting the brightness of the display based on the ambient light level.
While the example LUT shown in Table 1 includes red, green and blue sensor readings and uses red and blue intensity values to adjust the white balance point, other colors may be used for this purpose. For example, sensors measuring the colors cyan, magenta and yellow may be used, although any sensor measuring any three or more colors that span a target color space may also be used. Likewise, any two or more colors may be used to adjust the white balance point of the display.
Although the memory 110 is shown as an EEPROM, it is contemplated that it may be implemented as a read only memory (ROM), flash memory or other non-volatile memory device.
After comparing measured values to reference values at step 260 of
Alternatively, in another embodiment, the example color temperature calculator 100 may compare the values detected by ambient light sensor 90 to the reference ambient lighting values stored in EEPROM 110 at step 260. At step 270, the example color temperature calculator 100 looks for a close match. If a close match is found, for example if the values of the ambient light are within five percent of the intensity values for a color temperature of 5000 K, color temperature calculator 100 may select the adjustment value(s) for the white pixels corresponding to the matching reference value. If a close match is not found at step 270, for example if the intensity values of the ambient light correspond to a color temperature of 4900 K, color temperature calculator 100 may proceed to step 290 and interpolate between the two closest color temperatures. In this example, color temperature calculator 100 linearly interpolates each of the color components between daylight at 5000 K and moonlight at 4100 K to determine interpolated adjustment values for the white pixels. In this way, the system of this embodiment may perform a more sensitive adjustment of the white point based on the ambient lighting.
Where exact color appearance is desirable, color temperature calculator 100 may be configured to calculate an adjustment value for the white point of the display exactly, based on the ambient light values sensed by ambient light sensor 90.
In the embodiment described in
The system of
As shown in
Adjusting light source 60 via controller driver 70 at step 220 of
Adjusting the display drive at step 250 of
The example system of
One example of a camera phone utilizing a system similar to that of
As shown in
In another embodiment, imager 450 may be used to capture the ambient light used for color balancing or imager 450 may be used in conjunction with any or all of ambient light sensors 420 and 440. In either scenario, when imager 450 is used as an ambient light sensor, the imager may be operable in at least two different modes. One mode may be an ambient light evaluating mode and another mode may be an image capture mode.
In ambient light evaluating mode, the imager may be exposed by opening a shutter. The shutter may be opened briefly to capture the ambient light once or for a longer period of time to capture the ambient light a number of times. During the ambient light evaluating mode, the imager may capture ambient light levels and output signals corresponding to the captured ambient light levels. As shown in
Processing in imager output processing unit 94 may be desirable when an imager is used to capture the ambient light for purposes of color balancing. This is because the imager may include a large number of different colored pixels as opposed to the example ambient light sensor 90 which uses only one pixel of each color. Processing may include, for example, averaging all or some of the pixels for each color to determine, for example, average red, green and blue values for the ambient light, selecting the brightest pixels and using the values from those pixels as the red, green and blue values for the ambient light or any other suitable processing method.
If the color balancing is used for image processing, the shutter will open a second time to capture the image and then image processing will take place using the values output by imager output processing unit 94 during the ambient light evaluating mode. Otherwise, the values output by imager output processing unit 94 will be input into color-temperature calculator 100 and the display will be color balanced according to any of the embodiments described above.
There may be advantages and drawbacks to using an imager as the ambient light sensor. One possible advantage is that for applications that already include an imager, additional components do not have to be added specifically for color balancing, thus reducing the number of parts in the device. However, imagers use more power than the example ambient light sensors disclosed above and, therefore, using the imager as the ambient light sensor may decrease battery power more rapidly than if a simpler ambient light sensor were used. Additionally, because imagers typically have many more pixels than would the typical ambient light sensor, the complexity of the processing may increase relative to the ambient light sensors disclosed above to determine, for example, red, green and blue ambient light values usable by the example color-temperature calculator 100. If the device provides for variable focusing, one possible method of reducing processing in the imager output processing unit would be to have the imager capture the image out of focus. In this way, fewer data points (pixels) may be processed to determine the ambient lighting levels.
The embodiments of the present invention may execute automatically to perform automatic white balancing of the display or may be executed manually when, for example, a user presses button 430 shown in
In one example automatic mode, ambient light sensor 90 may be configured to sample the ambient lighting once at a predetermined time. For example, ambient light sensor 90 may be configured to sample the ambient lighting after the device has been turned on and a certain period of time has elapsed. In another example automatic mode, ambient light sensor 90 may be configured to sample the ambient lighting continually and to re-calculate the color temperature upon each reading. These example automatic modes may, however, present a problem if, for example, the user is wearing a red shirt and the sensor is, for example, overly sensitive to red light. Here, the ambient light sensor may sense an exaggerated intensity of the red element in the ambient lighting if the user holds the device in such a way that the ambient light sensor is near the shirt. As a result, the white balancing may overcompensate for the red element and the colors displayed by the display may be distorted.
It is desirable for the sensor to detect the appropriate amount of red reflected from the user's shirt that will actually appear in the image. For example, if a user is holding a small white IPOD next to the user's red shirt and is looking at an image containing relatively many white pixels, the white casing of the IPOD will appear to have a red tinge. Because the embodiments described above match the white balance of the image to the white balance of the environment, the white pixels in the image would also appear to the user to have a red tinge so that the user would see a difference in color between the white in the display and the white of the IPOD casing.
If, however, the sensor is overly sensitive to red light, the sensor may detect an exaggerated amount of the red light and overcorrect for it. This problem may be resolved, for example, by configuring ambient light sensor 90 to sample continually while the device is turned on and to average each consecutive sample or to select a maximum sample. In this way, averaging the samples or selecting a maximum sample may lessen the effect of one sample taken, for example, while the user was holding the camera so that the ambient light sensor was located close to the user's shirt.
In one example manual mode, ambient light sensor 90 may be configured to sample the ambient lighting once in response to a user pushing button 430, for example, when the light sensor has a white object in its field of view. In this example, button 430 is a push-button switch for activating the white balancing operation. Button 430 may also be another kind of a switch, a touch screen operation, or any other similar mechanism. In another example manual mode, ambient light sensor 90 may be configured to sample the ambient lighting continually after the button has been depressed and until some other condition is present. For example, ambient light sensor 90 may be configured to sample continually after the button has been pressed and until the button is pressed a second time, until another button is pressed or until the device is turned off, and so on. In this example mode, ambient light sensor 90 may be configured to average together consecutive samples or to select a maximum sample and then re-calculate the color temperature in response to the resulting values.
Color sensors can measure only the light that is eventually passed to the photosensitive elements of the sensor. Accordingly, various techniques are known in the art and applicable to the present invention that may increase the scope of light that is applied to the photosensitive elements. One example technique is to place a lens over the color sensor, over each individual pixel, or both, to direct the ambient light toward the color sensor. To increase the area around the display over which the ambient light sample is taken, a fisheye lens may be placed over the sensor. For example, as shown in
As shown in
Although the invention is illustrated and described herein with reference to specific embodiments, the invention is not intended to be limited to the details shown. Rather, various modifications may be made in the details within the scope and range of equivalents of the claims and without departing from the invention.
The present invention is an apparatus and method for adjusting the color balance of a display. A sensor of the apparatus detects the color temperature of ambient light. A controller of the apparatus adjusts the color balance of the emissive display so that the white point of the display matches the white point of the detected ambient light.
While example embodiments of the invention have been shown and described herein, it will be understood that such embodiments are provided by way of example only. Numerous variations, changes and substitutions will occur to those skilled in the art without departing from the invention. Accordingly, it is intended that the appended claims cover all such variations as fall within the scope of the invention.
