Patent Application
20070279372
Novel sub-pixel arrangements are disclosed for improving the cost/performance curves for image display devices in the following commonly owned United States Patents and Patent Applications including: (1) U.S. Pat. No. 6,903,754 (“the '754 patent”) entitled “ARRANGEMENT OF COLOR PIXELS FOR FULL COLOR IMAGING DEVICES WITH SIMPLIFIED ADDRESSING;” (2) United States Patent Publication No. 2003/0128225 (“the '225 application”) having application Ser. No. 10/278,353 and entitled “IMPROVEMENTS TO COLOR FLAT PANEL DISPLAY SUB-PIXEL ARRANGEMENTS AND LAYOUTS FOR SUB-PIXEL RENDERING WITH INCREASED MODULATION TRANSFER FUNCTION RESPONSE,” filed Oct. 22, 2002; (3) United States Patent Publication No. 2003/0128179 (“the '179 application”) having application Ser. No. 10/278,352 and entitled “IMPROVEMENTS TO COLOR FLAT PANEL DISPLAY SUB-PIXEL ARRANGEMENTS AND LAYOUTS FOR SUB-PIXEL RENDERING WITH SPLIT BLUE SUB-PIXELS,” filed Oct. 22, 2002; (4) United States Patent Publication No. 2004/0051724 (“the '724 application”) having application Ser. No. 10/243,094 and entitled “IMPROVED FOUR COLOR ARRANGEMENTS AND EMITTERS FOR SUB-PIXEL RENDERING,” filed Sep. 13, 2002; (5) United States Patent Publication No. 2003/0117423 (“the '423 application”) having application Ser. No. 10/278,328 and entitled “IMPROVEMENTS TO COLOR FLAT PANEL DISPLAY SUB-PIXEL ARRANGEMENTS AND LAYOUTS WITH REDUCED BLUE LUMINANCE WELL VISIBILITY,” filed Oct. 22, 2002; (6) United States Patent Publication No. 2003/0090581 (“the '581 application”) having application Ser. No. 10/278,393 and entitled “COLOR DISPLAY HAVING HORIZONTAL SUB-PIXEL ARRANGEMENTS AND LAYOUTS,” filed Oct. 22, 2002; and (7) United States Patent Publication No. 2004/0080479 (“the '479 application”) having application Ser. No. 10/347,001 and entitled “IMPROVED SUB-PIXEL ARRANGEMENTS FOR STRIPED DISPLAYS AND METHODS AND SYSTEMS FOR SUB-PIXEL RENDERING SAME,” filed Jan. 16, 2003. Each of the aforementioned '225, '179, '724, '423, '581, and '479 published applications and U.S. Pat. No. 6,903,754 are hereby incorporated by reference herein in its entirety.
For certain subpixel repeating groups having an even number of subpixels in a horizontal direction, systems and techniques to affect improvements, e.g. polarity inversion schemes and other improvements, are disclosed in the following commonly owned United States patent documents: (1) United States Patent Publication No. 2004/0246280 (“the '280 application”) having application Ser. No. 10/456,839 and entitled “IMAGE DEGRADATION CORRECTION IN NOVEL LIQUID CRYSTAL DISPLAYS”; (2) United States Patent Publication No. 2004/0246213 (“the '213 application”) (U.S. patent application Ser. No. 10/455,925) entitled “DISPLAY PANEL HAVING CROSSOVER CONNECTIONS EFFECTING DOT INVERSION”; (3) United States Patent Publication No. 2004/0246381 (“the '381 application”) having application Ser. No. 10/455,931 and entitled “SYSTEM AND METHOD OF PERFORMING DOT INVERSION WITH STANDARD DRIVERS AND BACKPLANE ON NOVEL DISPLAY PANEL LAYOUTS”; (4) United States Patent Publication No. 2004/0246278 (“the '278 application”) having application Ser. No. 10/455,927 and entitled “SYSTEM AND METHOD FOR COMPENSATING FOR VISUAL EFFECTS UPON PANELS HAVING FIXED PATTERN NOISE WITH REDUCED QUANTIZATION ERROR”; (5) U.S. Pat. No. 7,187,353 (“the '353 patent”) having application Ser. No. 10/456,806 entitled “DOT INVERSION ON NOVEL DISPLAY PANEL LAYOUTS WITH EXTRA DRIVERS”; (6) United States Patent Publication No. 2004/0246404 (“the '404 application”) having application Ser. No. 10/456,838 and entitled “LIQUID CRYSTAL DISPLAY BACKPLANE LAYOUTS AND ADDRESSING FOR NON-STANDARD SUBPIXEL ARRANGEMENTS”; (7) United States Patent Publication No. 2005/0083277 (“the '277 application”) having application Ser. No. 10/696,236 entitled “IMAGE DEGRADATION CORRECTION IN NOVEL LIQUID CRYSTAL DISPLAYS WITH SPLIT BLUE SUBPIXELS”, filed Oct. 28, 2003; and (8) United States Patent Publication No. 2005/0212741 (“the '741 application”) having application Ser. No. 10/807,604 and entitled “IMPROVED TRANSISTOR BACKPLANES FOR LIQUID CRYSTAL DISPLAYS COMPRISING DIFFERENT SIZED SUBPIXELS”, filed Mar. 23, 2004. Each of the aforementioned '280, '213, '381, '278, '404, '277 and '741 published applications and the '353 patent are hereby incorporated by reference herein in its entirety.
These improvements are particularly pronounced when coupled with sub-pixel rendering (SPR) systems and methods further disclosed in the above-referenced U.S. Patent documents and in commonly owned United States Patents and Patent Applications: (1) U.S. Pat. No. 7,123,277 (“the '277 patent”) having application Ser. No. 10/051,612 and entitled “CONVERSION OF A SUB-PIXEL FORMAT DATA TO ANOTHER SUB-PIXEL DATA FORMAT,” filed Jan. 16, 2002; (2) United States Patent Publication No. 2003/0103058 (“the '058 application”) having application Ser. No. 10/150,355 entitled “METHODS AND SYSTEMS FOR SUB-PIXEL RENDERING WITH GAMMA ADJUSTMENT,” filed May 17, 2002; (3) U.S. Pat. No. 7,184,066 (“the '066 patent”) having application Ser. No. 10/215,843 and entitled “METHODS AND SYSTEMS FOR SUB-PIXEL RENDERING WITH ADAPTIVE FILTERING,” filed Aug. 8, 2002; (4) United States Publication No. 2004/0196302 (“the '302 application”) having application Ser. No. 10/379,767 and entitled “SYSTEMS AND METHODS FOR TEMPORAL SUB-PIXEL RENDERING OF IMAGE DATA” filed Mar. 4, 2003; (5) U.S. Pat. No. 7,167,186 (“the '186 patent”) having application Ser. No. 10/379,765 and entitled “SYSTEMS AND METHODS FOR MOTION ADAPTIVE FILTERING,” filed Mar. 4, 2003; (6) U.S. Pat. No. 6,917,368 (“the '368 patent”) entitled “SUB-PIXEL RENDERING SYSTEM AND METHOD FOR IMPROVED DISPLAY VIEWING ANGLES”; and (7) United States Patent Publication No. 2004/0196297 (“the '297 application”) having application Ser. No. 10/409,413 and entitled “IMAGE DATA SET WITH EMBEDDED PRE-SUBPIXEL RENDERED IMAGE” filed Apr. 7, 2003. Each of the aforementioned '058, '302, and '297 applications and the '277, '066, '186 and '368 patens are hereby incorporated by reference herein in its entirety.
Improvements in gamut conversion and mapping are disclosed in commonly owned United States Patents and co-pending United States Patent Applications: (1) U.S. Pat. No. 6,980,219 (“the '219 patent”) entitled “HUE ANGLE CALCULATION SYSTEM AND METHODS”; (2) United States Patent Publication No. 2005/0083341 (“the '341 application”) having application Ser. No. 10/691,377 and entitled “METHOD AND APPARATUS FOR CONVERTING FROM SOURCE COLOR SPACE TO TARGET COLOR SPACE”, filed Oct. 21, 2003; (3) United States Patent Publication No. 2005/0083352 (“the '352 application”) having application Ser. No. 10/691,396 and entitled “METHOD AND APPARATUS FOR CONVERTING FROM A SOURCE COLOR SPACE TO A TARGET COLOR SPACE”, filed Oct. 21, 2003; and (4) U.S. Pat. No. 7,176,935 (“the '935 patent”) having application Ser. No. 10/690,716 and entitled “GAMUT CONVERSION SYSTEM AND METHODS” filed Oct. 21, 2003. Each of the aforementioned '341, and '352 applications and the '219 and '935 patents is hereby incorporated by reference herein in its entirety.
Additional advantages have been described in (1) U.S. Pat. No. 7,084,923 (“the '923 patent”) having application Ser. No. 10/696,235 and entitled “DISPLAY SYSTEM HAVING IMPROVED MULTIPLE MODES FOR DISPLAYING IMAGE DATA FROM MULTIPLE INPUT SOURCE FORMATS”, filed Oct. 28, 2003; and in (2) United States Patent Publication No. 2005/0088385 (“the '385 application”) having application Ser. No. 10/696,026 and entitled “SYSTEM AND METHOD FOR PERFORMING IMAGE RECONSTRUCTION AND SUBPIXEL RENDERING TO EFFECT SCALING FOR MULTI-MODE DISPLAY” filed Oct. 28, 2003, each of which is hereby incorporated herein by reference in its entirety.
Additionally, each of these co-owned and co-pending applications is herein incorporated by reference in its entirety: (1) United States Patent Publication No. 2005/0225548 (“the '548 application”) having application Ser. No. 10/821,387 and entitled “SYSTEM AND METHOD FOR IMPROVING SUB-PIXEL RENDERING OF IMAGE DATA IN NON-STRIPED DISPLAY SYSTEMS”; (2) United States Patent Publication No. 2005/0225561 (“the '561 application”) having application Ser. No. 10/821,386 and entitled “SYSTEMS AND METHODS FOR SELECTING A WHITE POINT FOR IMAGE DISPLAYS”; (3) United States Patent Publication No. 2005/0225574 (“the '574 application”) and United States Patent Publication No. 2005/0225575 (“the '575 application”) having application Ser. Nos. 10/821,353 and 10/961,506 respectively, and both entitled “NOVEL SUBPIXEL LAYOUTS AND ARRANGEMENTS FOR HIGH BRIGHTNESS DISPLAYS”; (4) United States Patent Publication No. 2005/0225562 (“the '562 application”) having application Ser. No. 10/821,306 and entitled “SYSTEMS AND METHODS FOR IMPROVED GAMUT MAPPING FROM ONE IMAGE DATA SET TO ANOTHER”; (5) United States Patent Publication No. 2005/0225563 (“the '563 application”) having application Ser. No. 10/821,388 and entitled “IMPROVED SUBPIXEL RENDERING FILTERS FOR HIGH BRIGHTNESS SUBPIXEL LAYOUTS”; and (6) United States Patent Publication No. 2005/0276502 (“the '502 application”) having application Ser. No. 10/866,447 and entitled “INCREASING GAMMA ACCURACY IN QUANTIZED DISPLAY SYSTEMS.”
Additional improvements to, and embodiments of, display systems and methods of operation thereof are described in: (1) Patent Cooperation Treaty (PCT) Application No. PCT/US 06/12768, entitled “EFFICIENT MEMORY STRUCTURE FOR DISPLAY SYSTEM WITH NOVEL SUBPIXEL STRUCTURES” filed Apr. 4, 2006, and published in the United States as United States Patent Application Publication 200Y/AAAAAAA; (2) Patent Cooperation Treaty (PCT) Application No. PCT/US 06/12766, entitled “SYSTEMS AND METHODS FOR IMPLEMENTING LOW-COST GAMUT MAPPING ALGORITHMS” filed Apr. 4, 2006, and published in the United States as United States Patent Application Publication 200Y/BBBBBBB; (3) U.S. patent application Ser. No. 11/278,675, entitled “SYSTEMS AND METHODS FOR IMPLEMENTING IMPROVED GAMUT MAPPING ALGORITHMS” filed Apr. 4, 2006, and published as United States Patent Application Publication 2006/0244686 (“the '686 application”); (4) Patent Cooperation Treaty (PCT) Application No. PCT/US 06/12521, entitled “PRE-SUBPIXEL RENDERED IMAGE PROCESSING IN DISPLAY SYSTEMS” filed Apr. 4, 2006, and published in the United States as United States Patent Application Publication 200Y/DDDDDDD; and (5) Patent Cooperation Treaty (PCT) Application No. PCT/US 06/19657, entitled “MULTIPRIMARY COLOR SUBPIXEL RENDERING WITH METAMERIC FILTERING” filed on May 19, 2006 and published in the United States as United States Patent Application Publication 200Y/EEEEEEE (referred to below as the “Metamer Filtering application”.) Each of these co-owned applications is also herein incorporated by reference in their entirety.
Additional improvements to, and embodiments of, display systems and methods of operation thereof are described in: (1) Patent Cooperation Treaty (PCT) Application No. PCT/US 06/40272, entitled “IMPROVED GAMUT MAPPING AND SUBPIXEL RENDERING SYSTEMS AND METHODS” filed Oct. 13, 2006, and published in the United States as United States Patent Application Publication 200Y/FFFFFFF; (2) Patent Cooperation Treaty (PCT) Application No. PCT/US 06/40269, entitled “IMPROVED MEMORY STRUCTURES FOR IMAGE PROCESSING” filed Oct. 13, 2006, and published in the United States as United States Patent Application Publication 200Y/GGGGGGG; (3) Patent Cooperation Treaty (PCT) Application No. PCT/US 06/NNNNN, entitled “COLOR IMAGING BACKLIGHT FOR COLOR DISPLAY SYSTEM AND METHODS OF OPERATION” filed Jun. 6, 2006, and published in the United States as United States Patent Application Publication 200Y/HHHHHHH; (4) Patent Cooperation Treaty (PCT) Application No. PCT/US 06/NNNNN, entitled “SYSTEMS AND METHODS FOR REDUCING DESATURATION OF IMAGES REDUCED ON HIGH BRIGHTNESS DISPLAYS” filed on Sep. 30, 2006 and published in the United States as United States Patent Application Publication 200Y/IIIIIII; (5) Patent Cooperation Treaty (PCT) Application No. PCT/US 06/NNNNN, entitled “SUBPIXEL PAYOUTS AND SUBPIXEL RENDERING METHODS FOR DIRECTIONAL DISPLAYS AND SYSTEMS” filed on Feb. 13, 2007 and published in the United States as United States Patent Application Publication 200Y/JJJJJJJ; and (6) Patent Cooperation Treaty (PCT) Application No. PCT/US 06/NNNNN, entitled “HIGH DYNAMIC CONTRAST SYSTEM HAVING MULTIPLE SEGMENTED BACKLIGHT” filed on Feb. 26, 2007 and published in the United States as United States Patent Application Publication 200Y/KKKKKKK. Each of these co-owned applications is also herein incorporated by reference in their entirety.
Improved Color Gamut Mapping Systems and Methods
In the following discussion, improved color gamut mapping systems and methods will be discussed. Such a discussion may proceed by first considering the nature of color gamuts in general, and more particularly, as they pertain to subpixelated multiprimary displays.
For a general consideration of gamuts,
It will be appreciated that although the techniques described herein work well with displays comprising colored subpixels in which at least one of the colored subpixels is substantially white or grey which are wide spectrum bandpass; but the techniques herein would work on displays wherein at least one of the colored subpixels is substantially a wide spectrum bandpass—e.g. grey, cyan, yellow, magenta or the like.
Note that the maximum value of the color gamut 310, of the red/green/blue/white display, at ten units of white, is twice that of the red/green/blue display of
The present application will herein disclose various embodiments of systems and methods—for merely a few examples—in regards to
Pre-Reduction Algorithms
One possible side effect of RGB to RGBW GMAs may be to reduce the total number of color states that a display system can produce. It may be possible that as many as half of the total number of input colors are mapped onto other colors. This might tend to happen to colors that are both high-brightness and high saturation at the same time, a combination that does not typically occur in natural and unmodified images. However, it might be desirable to have a mode where the total number of colors output by our GMA matches the number of colors that arrive. As noted above, Pre-reduction is one way to accomplish this.
In the case where the RGB gamut is essentially scaled until the input RGB white point is mapped onto the RGBW white point. The high brightness+high saturation colors in the shaded areas become out-of-gamut and may be mapped to allowed RGBW values using clamp-to-black, clamp-to-luma, clamp-diagonal or other algorithms. By a pre-reduction of the input values, eventually the entire RGB gamut may fit inside the RGBW gamut. In such a case, it may obviate the step of OOG mappings. Although the resulting images may not be as bright as before, there are actually more total output states used. Not all possible W values may be used, but all possible RGB output values may be, which may not be the case when some colors are OOG.
In the case of some layouts, pre-reducing the input values by one half may make the RGB gamut fit inside the RGBW gamut. Reducing by other desired amounts may be desirable for layouts when the brightness of the W sub-pixel may not exactly equal the brightness of the other 3 sub-pixels. Also, pre-reducing by desired amounts smaller than one half may increase the total number of output states somewhat, even though some colors will still go OOG. Even in displays with the W sub-pixel brightness equal to the other three primaries, this procedure may increase the brightness of the image and be desirable.
In one embodiment, the input RGB values may be pre-reduced until no OOG values result. Then the normal RGBW GMA may be used to convert to RGBW. Finally, the W values may be scaled up by an amount that makes the largest W value (usually from bright saturated colors near white) hit the maximum value. In one case, the input RGB values have been pre-reduced by half, and then the resulting W values have been scaled up by a factor of 2. This may result in a maximum brightness of about 75% of the maximum possible. Other combinations may produce brighter maximum values, for example reducing the input by a percentage smaller than one half. However, these combinations may produce more OOG colors and decrease the total number of output states.
A Pre-reduction module may be implemented between an input gamma module and GMA module. In such an embodiment, a percentage may be stored as a fixed point binary number in a Pre-reduce register. The pre-reduce register may be 8 bits big and may store a number between 0 and 255, representing a fixed point number between 0 and approximately 0.996. Each of the input RGB values after input gamma may be multiplied by the pre-reduce value, then divided by 256 in right shift modules (e.g. >>8).
In another embodiment, instead of using a multiplier, the input values may be shifted right by different amounts, and the results are added together in different combinations to produce 100% of the input (no reduction), 75% (25% reduction), 62.5%, 50% 37.5% 25% and 12.5%. Instead of storing a fixed point binary number in the pre-reduce register, an index may be stored that selects one of the pre-calculated percentages using a MUX. This set of percentages is only one example. By adding more shifters, adders, and a wider multiplexor, any number of selectable reduction percentages can be produced.
Saturation-Based Pre-Reduction
As another alternate embodiment of pre-reduction, the input RGB values may not be reduced by a fixed amount, but instead by an amount that is a function of saturation. A function which has values near 1.0 when saturation is near zero may have the advantage of mapping the input RGB white value approximately to the output RGBW white value. This may be advantageous over the pre-reduction algorithms above where the maximum possible white value may not be achieved. In another embodiment, the maximum value may be less than 1.0 to reduce simultaneous luminance contrast error. The functions of saturation may decrease to some percentage (Pmax) when saturation is at a maximum. If this Pmax value is larger than the ratio of the brightness of W to the sum of the brightness of the R+G+B subpixels in the display, then there may be some OOG colors. Thus, a gamut clamping module as described above may still be desirable.
One possible curve for this saturation function is a Gaussian, but this may be computationally difficult to implement in hardware. A straight line may be adequate, and a piecewise linear function may also produce pleasing images. The values from this function are multiplied by the input RGB values. Thus, multiplying by 1.0 causes no reduction of the input values with low saturation and multiplying by Pmax or other fractions less than one will result in decreasing input values that have high saturation. All these multiplications by fractional values may be implemented in hardware by multiplying by fixed point binary numbers followed by appropriate right shifts. Other means of implementing multiplication by shifts and adds are also included as part of the scope of the present invention.
Saturation may be considered the perpendicular distance away from the line of grays, typically scaled to range from 0 to 1.0 at the surface of the gamut. Although any number of calculations of saturation could be used, there are approximations to calculating this number which are well known in the industry, for example;
Saturation=(max(r,g,b)−min(r,g,b))/max(r,g,b)
The resulting saturation value may then used to generate one of the curves. The piecewise linear line, for example, with a Pmax value of 0.75 may be generated by the following equation:
Pre_reduce=min(1,1−((Saturation−0.25)/(1−0.25)))
Then the input red green and blue values may be each multiplied by such a Pre_reduce value, as generated by any of the above embodiments:
Finally, these RG and B values may be run through a GMA algorithm to convert RGB to RGBW. Other embodiments of a pre-reduce function are discussed in WO 2007/047537 and incorporated herein by reference.
In yet another embodiment, the pre-reduction function may also be made a function of hue. Faces and other skin tones, for example, have a very narrow range of hue and it may be advantageous to use a different pre-reduction function on images with these features. In yet another embodiment, the pre-reduction saturation function may also be made a function of brightness. So for a given saturation value, rather than using a constant scaling value, one would scale based on the proximity to BLACK. This would act like a gamma function, and it allows one to shift the output pixel distribution closer (or farther) to the RGBW gamut hull. It should also be appreciated that the pre-reduction function could be based as a function of some combination of hue, saturation and brightness.
In the discussion above, one embodiment might have only one pre-reduction function for all primaries. It may be desirable, however, to have a separate pre-reduction function for each (or a subset) of the input R G and B primaries. This could add the ability to do color correction or adjust the white point of the display.
It is possible to place pre-reduction in many places in the image processing system, such as before an input gamma module. Because the values before input gamma typically have a smaller bit-size, this may have the advantage of decreasing the gate count of hardware based on this design. In addition, it is possible to combine the pre-reduction functions with the input gamma function, performing the gamma correction and the pre-reduction in one step. Because the input gamma function is often implemented as a look-up-table that is pre-calculated, it may then be possible to use other algorithms, such as a Gaussian curve without paying a penalty in more complicated hardware.
With reference with to
To ensure that the backlight 695 is set to the minimum brightness needed to just enclose the image color/brightness gamut hull or shape, the scaled down RGBW color data in a given frame may be surveyed to find the peak value by the Peak Survey function 670. This function detects and extracts the maximum primary value, R G B or W in the entire frame. This peak value may be used by the Last Frame Peak function 675 to calculate the backlight and normalization values to be used in the next frame period, as will be discussed in reference to
The X/XL normalized RGBW data is subpixel rendered by the SPR function 640 possibly using methods disclosed in many of the above incorporated applications. Because the target LCD 690 may have a non-linearly quantized electro-optical transfer function, the linear subpixel rendered data is non-linearly quantized to match the LCD by the Inverse Gamma function 515. This function may or may not be the inverse of the input Gamma function 610.
Additional details of the Color Peak Survey function 670 and Last Frame Peak function 675 are shown in
When the peakval is zero, the NORMVAL may be set to zero. When the NORMVAL is between 0 and one, it may be stored as a fixed point binary number. This is the case of
Because frame buffer memory requires many gates, increasing the area used in a silicon integrated circuit, possibly increasing the cost beyond what may be considered economically viable in a given product, it may be desirable to have a system that does not require a frame buffer. For another embodiment, the block diagram 602 in
Without pre-reduction or clamping, the RGBW color data may contain OOG colors whose red, green, or blue value may exceed the maximum allowable value by two times if the RGBW gamut system provides twice the brightness as an RGB display with the same backlight. Thus, the RGBW may be scaled down, for example divided in two by a simple binary right shift function 637. This value may not be a factor of two if the gain factor for a given RGBW display is other than two. In such cases, the divisor may be set to the RGBW display gain. This divide by two operation 637 may result in loss of some precision which can be avoided by doing the divide later in the pipeline, for example in the X/XL module 660. This would require storing or processing an extra bit in peak survey module 670 and last frame peak storage 675. To ensure that the backlight 695 is set to the minimum brightness needed to just enclose the image color/brightness gamut, the scaled down RGBW color data in a given frame is surveyed to find the peak value by the Peak Survey function 670. This peak value is used in the Last Frame Peak function 675 to calculate and store the backlight and normalization values to be used in the next frame period. The backlight value from the previous frame period is used by the Backlight Control function 693 to actively control the Backlight 695 brightness.
Meanwhile the scaled down RGBW may also be compensated by the normalization value from the Last Frame Peak from the previous frame period in X/XL normalizing function 660. The X/XL normalized data may still have OOG color values, including in the W, the white, channel if the peak value from the previous frame period is less than that of the current frame period. The normalized color data is clamped to the RGBW color/brightness hull by the Clamp function 635. It may be desirable to use ‘clamp to luminance’. By clamping to the closest luminance value available, the luminance error during the current frame period may be minimized, possibly at the expense of the color saturation. If the next frame period's image is the same as the current, as often happens, then the colors that are clamped during the current frame period may be substantially realized during the next frame period. Since the Human Vision System recognizes luminance faster than color saturation, the error may not be noticed by the casual viewer. The normalized and clamped RGBW data may then be subpixel rendered by the SPR function 640. Because the target LCD 690 may have a non-linearly quantized electro-optical transfer function, the linear subpixel rendered data may be non-linearly quantized to match the LCD by the Inverse Gamma function 515. This function may or may not be the inverse of the input Gamma function 610.
The performance of the systems described in the block diagrams 601 & 602 in
It may be desirable in some applications to reduce the power further by limiting the brightness of all or some of the saturated colors. The addition of a Pre-Reduce function block may be one method of achieving this. A block diagram of this embodiment is shown in
There may be some disadvantages to pre-reduction. For example, the shape of the color-brightness gamut hull may be altered, with both desirable and sometimes undesirable consequences. One of these consequences might be that the simultaneous luminance contrast between the brightest white and the brightest saturated colors, especially yellow, is increased, causing color appearance differences when images having both bright white and bright colors are displayed. The color-brightness gamut hull for a conventional RGB, red, green, blue, three-primary display has relative brightness ratios of 30:59:11 respectively. Thus the brightness ratio of white vs. any single saturated primary color would be the sum of red, green, and blue over the brightness of the primary.
However, one important metric may not be the brightness ratio, but the simultaneous luminance contrast, more specifically, the Michaelson Contrast given by the formula (Max−Min)/(Max+Min)=% contrast. Thus, to determine the simultaneous luminance contrast between white and yellow (yellow being the sum of red and green): (100−(30+59))/(100+(30+59))=11/189=5.8%. The worst case for yellow would be to decrease the brightness until it is no longer Out-Of-Gamut. However, this would increase the Michaelson Contrast to 38.4%. The relative change in contrast between the RGB and RGBW contrast is 38.4%/5.8%=6.6 times, or 560% more simultaneous luminance contrast, assuming W=RGB. To put this into perspective, comparing the worst case change of simultaneous luminance contrast between RGB and RGBW systems for the major saturated colors, the change in white to yellow is the greatest, as can be seen in table 1:
As can be seen in Table 1, the worst case relative change in luminance contrast in yellow may be far greater than for the other saturated colors. The color appearance of bright saturated yellows may be noticeably different on an RGBW display compared to an RGB display, while the color appearance of the other bright saturated colors may not be objectionably changed, especially blue. Thus, it may be desired to reduce the relatively greater change in simultaneous luminance contrast in the bright saturated yellows while maintaining the benefit of decreased power. Thus, the yellow triangle, the colors between red and green, may be reduced less than the other colors so that when these bright saturated yellow colors occur, more power is used to reproduce these colors, but the red to blue to green colors are reduced more so that when only bright saturated colors in range occur (no bright saturated yellows), the power is reduced.
Adding the Pre-Reduce function block to the system shown in the block diagram 601 in
Without complete pre-reduction or clamping, the RGBW color data may contain OOG colors whose red, green, or blue value may exceed the maximum allowable value by up to two times if the RGBW gamut system provides twice the brightness as an RGB display with the same backlight. Thus, the RGBW is scaled down, divided in two by a simple binary right shift function 637. This value may not be a factor of two if the gain factor for a given RGBW display is other than two. In such cases, the divisor may be set to the RGBW display gain. The backlight 695 may be set substantially to the minimum brightness needed to just enclose the image color/brightness gamut, the scaled down RGBW color data in a given frame is surveyed to find the peak value by the Peak Survey function 670. This peak value may be used by the Last Frame Peak function 675 to calculate and store the backlight and normalization values to be used in the next frame period. The backlight value from the previous frame period is used by the Backlight Control function 693 to actively control the Backlight 695 brightness.
Meanwhile, the scaled down RGBW may also be stored in the Frame Buffer 650 to be used in the next frame period. Thus, the image displayed may be one frame behind the current. During the current frame period, scaled down RGBW from the previous frame period may be fetched from the Frame Buffer 650 to be compensated, normalized by the value from the Last Frame Peak from the previous frame period in X/XL normalizing function 660. The X/XL normalized RGBW data may then be subpixel rendered. Because the target LCD 690 may have a non-linearly quantized electro-optical transfer function, the linear subpixel rendered data may be non-linearly quantized to match the LCD by the Inverse Gamma function 515. This function may or may not be the inverse of the input Gamma function 610.
Adding the Pre-Reduce function block to the system shown in the block diagram 602 in
This value may not be a factor of two if the gain factor for a given RGBW display is other than two. In such cases, the divisor may be set to the RGBW display gain. To ensure that the backlight 695 is set to the minimum brightness needed to just enclose the image color/brightness gamut, the scaled down RGBW color data in a given frame is surveyed to find the peak value by the Peak Survey function 670. This peak value is used by the Last Frame Peak function 675 to calculate and store the backlight and normalization values to be used in the next frame period. The backlight value from the previous frame period is used by the Backlight Control function 693 to actively control the Backlight 695 brightness. Meanwhile the scaled down RGBW is also being compensated, normalized by the value from the Last Frame Peak from the previous frame period in X/XL normalizing function 660. The X/XL normalized may still have OOG color values, including in the W, the white, channel if the peak value from the previous frame period is less than that of the current frame period. The normalized color data is clamped to the RGBW color/brightness hull by the Clamp function 635. It may be desirable to use ‘clamp to luminance’. By clamping to the closest luminance value available the luminance error during the current frame period is minimized, at the expense of the color saturation. If the next frame period's image is the same as the current, as often happens, then the colors that are clamped during the current frame period will be fully realized during the next frame period. Since the Human Vision System recognizes luminance faster than color saturation, the error may not be noticed by the casual viewer. The normalized and clamped RGBW data is subpixel rendered by the SPR function 640. Because the target LCD 690 may have a non-linearly quantized electro-optical transfer function, the linear subpixel rendered data is non-linearly quantized to match the LCD by the Inverse Gamma function 515. This function may or may not be the inverse of the input Gamma function 610.
Priority Color Gamut Mapping
When the image contains “priority” bright saturated colors or colors that are to be reproduced at greater or possibly full brightness (e.g. bright saturated cyan, magenta or yellow or any other desired color), the colors that need not be reproduced at full value may also be reproduced at substantially full brightness if the system is arranged as the block diagrams 801 and 802 in
The block diagram 801 in
Thus, the image displayed may be one frame behind the current. During the current frame period, R*G*B* from the previous frame period is fetched from the Frame Buffer 650 to be linearized by a Gamma function 810. The linear RGB may be partially scaled down, as a function of saturation and the Color Gain fetched from the Last Frame Peak 875 in the Pre-Reduce function 820. The partially pre-reduced RGB color data may be converted to RGBW color data in the RGBW GMA 830 function block. The RGBW color data is then normalized by the value from the Last Frame Peak from the previous frame period in X/XL normalizing function 860. The X/XL normalized RGBW data may be subpixel rendered by the SPR function. Because the target LCD 890 may have a non-linearly quantized electro-optical transfer function, the linear subpixel rendered data may be non-linearly quantized to match the LCD by the Inverse Gamma function 815. This function may or may not be the inverse of the input Gamma function 810.
Details of the block 870 and block 875 are shown in
If PEAKVAL register 873 is initialized to zero at the start of a frame then by the end of the frame, it may contain the largest R G B or W primary in the entire frame. Priority Color Detector 874 defects priority colors in the input RGB′ signals. Priority colors may be passed on to max comparator 872b. For example, if the test B′<max(R′,G′) is true, then the input color is yellow. Other colors may be detected in a similar manner. Max comparator 872b compares the maximum primary from block 871 with PRIORVAL register 873b and stores the larger of the two back into block 873b. If block 873b is initialized to zero at the start of each frame then at the end of the frame it will have the largest primary (R G B or W) of the priority color. Priority Color Count register 891 may be initialized to zero at the start of every frame and incremented by one every time block 874 detects a priority color. By the end of each frame, this block 891 contains the number of priority colors seen in the frame. This may be used to modify calculations. After the end of each frame the CALC 1/Xl module 876 in block 875 inverts the PEAKVAL from block 870 in to calculate the normalization value. The result may be stored in NORMVAL register 877 to be used during the next frame. In Calc Backlight module 878, the quantized output of the Calc 1/XL module 876 may be used to prevent quantization errors when calculating the backlight value. Calc Color Gain module 876b is similar to Calc 1/Xl module 876 but it calculates a normalization value from the PRIORVAL from CPSM870. This may be stored in COLORGAIN register 877b to be used in the next frame.
Because frame buffer memory requires many gates, increasing the area used in a silicon integrated circuit, possibly increasing the cost beyond what may be considered economically viable in a given product, it may be desirable to have a system that does not require a frame buffer. The block diagram 802 in
Meanwhile, the linear RGB may be partially scaled down, as a function of saturation and the Color Gain fetched from the Last Frame Peak 875 in the Pre-Reduce function 820. The partially pre-reduced RGB color data may be converted to RGBW color data in the RGBW GMA 830 function block. During the current frame period, scaled down RGBW from the current frame period may be normalized by the value from the Last Frame Peak from the previous frame period in X/XL normalizing function 860. Thus the normalization for the current frame may be calculated in the previous frame. The X/XL normalized data may still have OOG color values, including in the W, the white, channel if the peak value from the previous frame period is less than that of the current frame period. The normalized color data is clamped to the RGBW color/brightness hull by the Clamp function 635. It may be desirable to use ‘clamp to luminance’. By clamping to the closest luminance value available, the luminance error during the current frame period may be minimized, at the expense of the color saturation. If the next frame period's image is the same as the current, as often happens, then the colors that are clamped during the current frame period may be fully realized during the next frame period. Since the Human Vision System recognizes luminance faster than color saturation, the error may not be noticed by the casual viewer. The X/XL normalized and clamped RGBW data is subpixel rendered by the SPR function 840. Because the target LCD 890 may have a non-linearly quantized electro-optical transfer function, the linear subpixel rendered data may be non-linearly quantized to match the LCD by the Inverse Gamma function 815. This function may or may not be the inverse of the input Gamma function 810.
It may be desirable to survey the colors to determine if there is sufficient number of pixels of a priority color (e.g. bright saturated yellow). Priority Color Count register 881 in
Alternative Subpixel Rendering Techniques
Although all of the embodiments described herein may use any known subpixel rendering (SPR) techniques and still benefit from dynamic gamut mapping and backlight control means and mechanisms described herein, there are some alternative SPR systems and methods that might be desirable in such systems.
When applying subpixel rendering techniques to whole pixel image data, there is the possibility of error for certain image patterns, namely saturated color against unsaturated color—e.g. green and white checkerboard. This is particularly true when the subpixel repeating pattern for a display is of a RGBW format—as disclosed in many of the applications incorporated by reference above—or other multiprimary subpixel layouts in which metamers occur among the chosen primary subpixel colors. For example, the white (W) subpixel represents a metamer for red, green and blue (RGB) subpixels. In co-owned application WO 2006/12755 (the '755 publication, which is incorporated herein by reference), there are disclosed various embodiments of utilizing metamer-luminance signal based sharpening (“meta-luma”) techniques. Also in the '755 publication, there are disclosed various embodiments of combining meta-luma sharpening techniques with other SPR techniques such as area resampling and difference of Gaussian (DOG) sharpening techniques as employed on image data on a same color plane (e.g. green color plane) (“same color sharpening”) or employing different color plane image data to perform sharpening (“cross color sharpening”).
If such mixed sharpening algorithm uses same color sharpening when on—for example—the green subpixel and meta-luma sharpening when on the white subpixel; there may be a difference in luminance and color for two phases of checkerboard.
Thus, it may be desirable to determine when to use meta-luma sharpening and same color sharpening and abate or avoid any such differences. One embodiment may be to look at the saturation of pixels near the target pixel and if any are saturated beyond some threshold, then same color sharpening may be used. If the pixels are all unsaturated below some threshold, then meta-luma sharpening may be used. The following pseudo-code may be employed to implement this embodiment:
In this code, sat threshold may be set to a low value and saturation value “S” may be calculated as W/maxRGB or minRGB/maxRGB. This means that S=0 refers to unsaturated e.g. white or gray and S=1 refers to pure colors. The special case of R=G B=0 may be identified as unsaturated. “Metasharp” represents a meta luma sharpening technique. “Fullsharp” represents same color sharpening. Other filters may be used based on the results of this test.
For one possible hardware embodiment, it may be possible to use the value of S calculated in the prescaling block. If S is less than a threshold sat threshold, then a flag may be set to 0; otherwise the flag is set to 1. This information may be calculated in the GMA and may be stored through the line buffers for calculation in SPR block. This may be done by carrying one less bit for blue data in the PenTile engine and using this bit for the flag. Alternatively, the line buffers may be expanded by 1 bit.
In the SPR block, the S flag is checked for the target pixel, the pixels to right, left, up, and down. If any are set to 1, then same color sharpening is used. If same color sharpening is used, the two checkerboard phases will look the same.
An alternate embodiment is to look at all eight surrounding pixels plus the target pixel and then make a decision.
Chromatic Filtering
Since a small number of priority colors may not increase the backlight power, as well as the presence of non-priority but still bright saturated colors, it may be desirable to provide a means whereby some of these colors may still be reproduced in a manner that will reduce the color appearance shifts caused by simultaneous contrast in those cases where the backlight power has not been sufficiently increased to reproduce them directly. For example, chromatic filtering may improve, reduce, simultaneous contrast color appearance shifts for high spatial frequency image components. Consider yellow text or lines on a white background. With chromatic filtering, the total energy of the chromatic signal may be maintained even as the colored lines or text strokes are desaturated, blended to the white surround, while the white surround takes on some of the color. Blue from the surrounding white will bleed into the yellow and yellow (or more precisely, negative, or minus, blue) will bleed into the white. Both of these colors, being desaturated, will pass through the PreReduce function 820 unmolested. Because the human eye can't see high spatial frequency chromatic signals, especially those in the yellow/blue color opponent channel, the blurring of the chromatic signal will not be visible.
The block diagram 803 in
The filtered LAB′ signal may be converted to a filtered RGB′ signal by the second Color Space Transform function 824. This signal may have values that are greater than allowed since a colored signal with the same luminance as white may not be reproduced, so the filtered RGB′ signal may be clamped toward black, at the highest allowed color to maintain the same color hue and saturation by the Clamp to Black function 823. Returning to
Meanwhile, the filtered RGB′ data may be partially scaled down, as a function of saturation and the Color Gain fetched from the Last Frame Peak 875 in the Pre-Reduce function 820. The partially pre-reduced RGB′ color data may be converted to RGBW′ color data in the RGBW GMA 830 function block. During the current frame period, scaled down RGBW′ from the current frame period may be normalized by the value of the Last Frame Peak from the previous frame period in X/XL normalizing function 860. Thus the data in the current frame may be normalized by the value calculated in the previous frame. The X/XL normalized data may still have OOG color values, including in the W, the white, channel if the peak value from the previous frame period is less than that of the current frame period. The normalized color data may be clamped to the RGBW color/brightness hull by the Clamp function 635. It may be desirable to use ‘clamp to luminance’. By clamping to the closest luminance value available, the luminance error during the current frame period may be minimized, at the expense of the color saturation. If the next frame period's image is the same as the current, as often happens, then the colors that are clamped during the current frame period will be fully realized during the next frame period. Since the Human Vision System recognizes luminance faster than color saturation, the error may not be noticed by the casual viewer. The X/XL normalized and clamped RGBW′ data may be subpixel rendered by the SPR function 840. Because the target LCD 890 may have a non-linearly quantized electro-optical transfer function, the linear subpixel rendered data may be non-linearly quantized to match the LCD by the Inverse Gamma function 815. This function may or may not be the inverse of the input Gamma function 810.
Another embodiment of chromatic filtering may be accomplished as follows: first, calculate the “L”, the luminance, of the linearized RGB pixel values and store them. Secondly, filter the RGB planes with a suitable blurring filter, possibly using the diamond filter to produce filtered R′G′B′ pixel values. Since this filtering operation may also filter the “L” channel, the stored L values may be used to readjust the luminance of the filtered R′G′B′ pixel values back to their original values. This may be done by calculating the L′ of each R′G′B′ pixel value. The R′G′B′ pixel values are then adjusted using the ratio of the original L and L′ by multiplying the R′G′B′ pixel values by L/L′. Apart from a diamond filter, it may also be possible to use any number of other simple filters, such as:
With all values above divided by 12.
Other methods of adjusting the ratio of the saturated vs. desaturated colors may be employed. For example, it is possible to reduce the desaturated colors as well as the saturated colors to perform the normalization function within the pre-reduce function, eliminating the later X/XL normalization. In
Another possible function in the pre-reduce function may be to adjust the gain (scaling down) of the colors in the original RGB gamut hull 210 to the smaller one shown inside, with the colors in the triangle 1085 defined in
Perceptually quantized R*G*B* data of a given binary bit depth may be linearized by the Gamma function 910 to be a greater binary bit depth linearly encoded RGB signal. The linear RGB goes down three different paths. In a first path, the linear RGB may be surveyed in Color Peak Survey module 970. The survey results may be used at the end of each frame in Last Frame Peak function 975 to calculate and store backlight and gain values for the next frame. The two backlight values from Last Frame Peak function 975 may be fed to comparator 985 to determine which is larger and the result may be used in backlight mux 980 to select the largest of the two values. The resulting largest backlight value from the previous frame period may be used by the Backlight Control function 993 to actively control the Backlight 995 brightness.
In the second path, the linear RGB signal may be processed by up scale module 965 using the up gain value from last frame peak storage 975. In the third path, the linear RGB signal may be processed in pre-reduce module 920. Color MUX 925 may select one of these two signals based on the result from comparator 985. The selected signal, pre-reduced or up-scaled RGB color data, may be converted to RGBW color data in the RGBW GMA 930 function block. Without complete pre-reduction or clamping, the RGBW color data may contain OOG colors whose red, green, or blue value may exceed the maximum allowable value by two times if the RGBW gamut system provides twice the brightness as an RGB display with the same backlight. Thus, the selected color data may be clamped to the RGBW color/brightness hull by the Clamp function 935. It may be desirable to use ‘clamp to luminance’. By clamping to the closest luminance value available, the luminance error during the current frame period may be minimized, at the expense of the color saturation. If the next frame period's image is the same as the current, as often happens, then the colors that are clamped during the current frame period may be fully realized during the next frame period. Since the Human Vision System recognizes luminance faster than color saturation, the error may not be noticed by the casual viewer. The normalized and clamped RGBW data is subpixel rendered by the SPR function 940. Because the target LCD 990 may have a non-linearly quantized electro-optical transfer function, the linear subpixel rendered data may be non-linearly quantized to match the LCD by the Inverse Gamma function 915. This function may or may not be the inverse of the input Gamma function 910.
The details of color peak survey module 970 and Last Frame Peak module 975 are shown in
In the second path, MAX RGBW GMA output values from each input color in a frame may be sent to CALC OOG module 954. This module calculates how far the output from MAX RGBW GMA module 951 is out-of-gamut. One way to calculate this may be to subtract the maximum in-gamut value from the MAX RGBW value and clamp all negative values up to zero. If the signal is not out-of-gamut then no value is sent to the next module. Optionally, this module could also not send values to the next module if the color is not a priority color. For example the test B<max(R,G) is only true when the color is in the “yellow triangle” of a chromaticity diagram. This test may make the CALC OOG module 954 only test yellow colors for out-of-gamut. If the MAX RGBW GMA module 951 output is yellow and is out of gamut then a non-zero value may be output to MAX module 955. MAX module 955 compares the OOG value with previous OOG values in the same frame and outputs the larger of the two. OOGMAX register 956 thus acquires the largest OOG value seen in the frame and thus may be initialized to zero before the start of each frame. After all the input values from an entire frame have been seen then the Last Frame Peak module 975 calculates and stores the results. Calc up-scale module 957 uses the peakval register 953 output to calculate the Backlight 2 and up-scale values.
One way to calculate the backlight2 value may be simply to divide half the peakval by the maximum in-gamut color. This results in a value between 0 and ½ which may be scaled and quantized to the integer range of the backlight control values. The up-scale output value is the maximum in-gamut color divided by the peakval. When the peakval is zero, the backlight may also be black and zero may be used for the up-scale value. This calculation results in a value between zero and the maximum in-gamut color which may be stored in register UPSCALE 962. The division may be implemented by an inverse look-up-table (LUT). The division would ordinarily result in many non-integer values so some fixed point bit size may be chosen and values truncated to integers. This operation ordinarily results in quantization errors but those may be avoided by calculating the backlight 2 value backwards from the up-scale value. The backlight2 value in this embodiment may be half of the inverse of the up-scale value. Note that the same inversion LUT may be used for this calculation. The result is a floating point number between 0 and ½ which may be scaled and quantized to the range of the backlight control values before storing it in BACKLIGHT2 register 961. The result of calculating backlight 2 backwards from the up-scale value may be that many of the possible backlight control values may not be used, but those that may be used may not have quantization error when used with the corresponding up-scale value. In mathematical notation the formula for these two values would be:
Where MAXCOL is substantially the maximum in-gamut color and MAXBACK is substantially the maximum backlight control value. In one embodiment, Calc Color Gain module 958 may calculate the backlight1 and color-gain values in the following manner: the backlight1 value is ½ plus half of the oogmax value divided by the maximum in-gamut color. This is a value between ½ (when oogmax is 0) and 1 (when oogmax is at the maximum) and may be scaled and quantized to the integer range of the backlight controller before storing in BACKLIGHT1 register 963. The color-gain value may be one minus half of the oogmax value divided by the maximum in-gamut color. No inversion may be necessary in this calculation. However, if this is coupled with the inversion table in the pre-reduce module, quantization error may be reduced there as well. This results in a color-gain value between 1 and ½ which may be scaled and quantized to the range of values that the pre-reduce module expects and stored in COLORGAIN register 964. In mathematical notation the formula for these two values would be:
Systems with Different Backlights
It should be noted that the techniques of the present application apply whether the backlight of the system is a unitary backlight (i.e. one backlight unit that illuminates the entire display area of the LCD) or whether the backlight tends to partition the LCD display area (e.g. as is found with a low resolution LED backlight where each LED is separately controllable and addressable, as depicted in
Arrays of light emitting diodes (LEDs) are used as light emitting sources in backlit display systems. U.S. Pat. No. 6,923,548 B2 discloses a backlight unit in a liquid crystal display that includes a plurality of lamps or chips arranged such that LED chips realizing R, G, and B colors are built in the respective lamps or chips. U.S. Pat. No. 6,923,548 B2 describes the backlight unit as realizing high brightness and providing a thin backlight unit. U.S. Pat. No. 7,002,547, which is hereby incorporated by reference herein, discloses a backlight control device for a transmissive type or for a transreflective type liquid crystal display equipped with LEDs as a backlight. The backlight control device includes an LED driving circuit connected to a power supply circuit for driving the LED, and a current control device that detects brightness around the liquid crystal display for controlling the driving current for the LED according to the detected brightness. Hideyo Ohtsuki et al., in a paper entitled “18.1-inch XGA TFT-LCD with wide color reproduction using high power led-backlighting,” published in the Proc. of the Society for Information Display International Symposium, in 2002, disclose an 18.1 inch XGA TFT-LCD module using an LED-backlighting unit. Ohtsuki et al. disclose that a side-edge type backlight is applied and two LED strips are located on the top and bottom edges of a light-pipe. Each LED strip arranges multiple red, green and blue LEDs. The lights from the red, green and blue LEDs are mixed and injected into the light-pipe. The brightness of the red, green and blue LEDs can be dimmed independently by a control circuit. Ohtsuki et al. disclose that the color-filter of this LCD panel is well-tuned to get higher color saturation.
U.S. Pat. No. 6,608,614 B1 entitled “Led-based LCD backlight with extended color space” discloses a backlight for a liquid-crystal display that includes a first LED array that provides light with a first chromaticity and a second LED array that provides light with a second chromaticity. A combining element combines the light from the first LED array and the second LED array and directs the combined light toward the liquid crystal display. A control system is operationally connected to the second LED array. The controller adjusts the brightness of at least one LED in the second LED array to thereby adjust the chromaticity of the combined light.
US 2005/0162737 A1 (hereafter, “the '737 publication)”, entitled “High Dynamic Range Display Devices,” discloses a display having a screen which incorporates a light modulator and which is illuminated with light from a light source comprising an array of controllable light-emitters. The controllable-emitters and elements of the light modulator may be controlled to adjust the intensity of light emanating from corresponding areas on the screen.
For merely one example of a two dimensional array of light emitters in the backlight, consider the block diagram 1100 in
The RGBW color data is then normalized by the value from the Backlight Interpolation function 1135 in X/XL normalizing function 1130. The X/XL normalized RGBW data is subpixel rendered by the SPR function 1140. Because the target LCD 1190 may have a non-linearly quantized electro-optical transfer function, the linear subpixel rendered data may be non-linearly quantized to match the LCD by the Inverse Gamma function 1115. This function may or may not be the inverse of the input Gamma function 1110.
In one embodiment, Backlight Interpolation function may use the value of each light emitter in backlight array, as may be established in Peak Function block, to calculate the distribution of light of each color at each output pixel in display panel overlying light emitter. This distribution may be interpolated from the values of the light emitters established in Peak Function block, taking into account the point spread function (PSF) of each light emitter in array and the presence of diffusers and other optical components. This operation is an “up sampling” function, as indicated by the up arrow, and many possible “up sampling” functions may be suitable. One such function is a summation of the point sample contribution of the PSFs of the local light emitters times their values computed by the down sampling Peak function.
Table 2 provides pseudo-code for provides an example of a Peak Function, called “dopeak,” in pseudo code that uses the maximum value of the input image region to determine the value for one light emitter. For simplicity of illustration, this peak function makes the assumptions that the output display panel has a resolution of 8 times that of the backlight array, that the backlight array includes red, green and blue light emitters disposed in a rectangular (or square) array, and that the red, green and blue light emitters are coincident.
Table 2 also provides a backlight interpolation function called “dointerp.” This function fetches from a memory area called “ledbuf” (LED buffer) and writes to memory area for storing output color values called “fuzbuf” The function “dointerp” is called once for each input pixel and calculates the effect of all the surrounding backlight point spread functions to produce the color value that would be seen under the input (logical) pixel. The “dointerp” function uses a point spread function for each light emitter that assumes that each pixel can only be affected by the surrounding four light emitters.
The combination of the two functions, the “down sampling” of the Peak Function followed by the “up sampling” of the Backlight Interpolation function may retain the original resolution of the input image in terms of sample count (image size), but produce a set of output image values with lower spatial frequencies, i.e., a filtered version of the RGB input image data that approximates the distribution of light from the light emitter array. This data is then input to X/XL function. Note that some images may have regions of uniform (i.e., the same) color values. Knowledge of the location of uniform color regions in the image may be used to reduce computational load in GMA function by retaining/reusing values common to the region.
Input image RGB data is first modified by the relationship between the brightness of each incoming RGB value after input gamma function and the actual amount of RGB light available at that given pixel from backlight array, as provided by Backlight Interpolation function. This modification is accomplished in X/XL function by the ratio, X/XL, where X is the incoming value of R, G, or B. and XL is the backlight brightness value at that pixel of RL, GL, or BL. Thus, a given RGB to RGBW gamut mapping algorithm may have the input value R/RL, G/GL, B/BL. Those of skill in the art will appreciate that the use of X/XL function 136 allows for an “off-the-shelf” GMA function to be utilized (e.g., any of the gamut mapping functions disclosed in the above-referenced applications), without a modification needed to accommodate the light contributions of the light emitters in backlight array.
It should be obvious to those skilled in the art that the teachings herein may be adapted to display systems other than those used as examples here. For example, the RGBW panel need not be subpixel rendered, being whole pixel rendered instead. The panel may have more than three saturated color primaries, for example it may comprise a layout with red, green, blue, and yellow (RGBY), or red, green, blue, and cyan (RGBC), or red, green, blue, cyan, and white (RGBCW), or red, green, blue, yellow and white (RGBYW), or other multiprimary panels, subpixel rendered or whole pixel rendered. The backlight may be white or it may be some other color as needed.
In addition, it should also be noted that the backlight illumination may depend on several factors that may work either independently or in concert. For example, backlight illumination may change as a result of changing luminance values in the image data to be rendered; the mode in which the display is being operated (e.g. primarily video rendering as opposed to text rendering); or the amount of out-of-gamut colors detected in the image data. It should also be appreciated that the techniques described herein work with either type of backlight (e.g. unitary or partitioning) as well as for each possible factor that changes backlight illumination.
It is important to note; however, for a large set of natural images, a probabilistic histogram plot of color values may assume the following shape as seen in
An embodiment of a RGBW display, however, might seek to improve on its typical performance when rendering bright saturated colors. Some embodiments may dynamically alter their performance depending on a number of criteria—either singly or in combination. For example, some criteria might be user selection, application selection or analysis of image data—to mention just a few.
In
It will be appreciated in many embodiments that a discrete number, infinite or near-infinite set of curves could define the Reduced Simultaneous-Contrast Modes. In addition, the curves might assume a different shape that a piecewise linear curve in some embodiments and some curves might seek to converge to different points rather than a single point such as 1710.
One possible embodiment of operation of a system having operating curves shown in
For example, period 1802 could be a low ambient light condition, period 1804 a period of high ambient light condition (e.g. the user of a cell phone steps outside from a darker building interior) and period 1806 a period of medium ambient light condition (e.g. where the user may have stepped under an awning outside). Performance curves 1822, 1824 and 1826 could correspond to the performance of the display system respectively.
The changes in the brightness vs. saturation curve performance may occur in a variety of ways. For example, a light sensor communicating with the display device could automatically set the appropriate performance curve. Alternatively, the user could manually set the performance curve to suit the tastes of the user.
Continuing with a putative example, using a sensor to signal the intensity of ambient light, simultaneous contrast settings can be made to track the ambient light conditions. As shown in
As described above, one possible action taken when detecting out of gamut colors may be to adjust the color prescaling (either saturated or unsaturated) so that colors are more in gamut and adjust the backlight to maintain luminance. Another method to adjust the brightness of colors might be to adjust the saturated color prescaling in combination with the luminance clamping control. For example, if it is desirable to increase the brightness of saturated colors, but keep the backlight level lower (to save power), one may allow saturated colors to go out of gamut by a given percentage and then clamp the values towards the luma axis. This may tend to increase the brightness by slightly desaturating the color. Since the prescaling of each primary color can independently be set, the color desaturation can be localized.
With reference to
In another embodiment, when out of gamut mixed colors are detected, especially cyan, magenta or yellow (or some other priority color, as mentioned above), a cyan, magenta or yellow boost (CMY boost) function, as described below, may be energized to increase the brightness of those colors. In this manner, no backlight change may be necessary and power may be conserved. It should be appreciated that any combination of unsaturated color prescaling, saturated color prescaling, or CMY boost can be used to optimize the system for best performance and lowest power. It should also be appreciated that the independent control of both unsaturated and saturated color prescaling allows the independent adjustment of color points as the backlight and scaling value change. For example, if the color coordinates change if gray level, as is often the case for LCDs, then the prescaling values can be set to compensate for this change, resulting in more accurate color rendition.
CMY Boost Function
With any RGBW or multiprimary system (including not only the novel ones described in the '575 application but also in conventional ones, like RGBW quad systems), the problem of “simultaneous contrast” is an issue that arises with rendering images having pure (or highly saturated) colors rendered against a white or very light background. In fact, such saturated colors would tend to look dark against such a white or light background. This is especially evident for yellow, cyan and possibly magenta—which are bright mixed colors. This discussion provides a possible solution to the problem of displaying these bright mixed colors on a display with RGBW (or “X”) primary colors. In general the techniques disclosed herein examine the input color image data for “major colors” and a “minor color” to determine which section of the color space an input color image data value is located. For example, if the input color image data is specified as RGB data, and the R and G data values are high and the B value is low, then the color is near yellow; if R and B are high and G is low, then the color is near magenta; and if B and G are high and R is low, then the color is near cyan. When such a condition is met, the technique computes a substitute color value for the low valued color data value. In effect, the technique seeks to adjust the level of the low valued color, referred to as “boost,” in a manner that allows for smooth color transitions (i.e., the “boost” decreases smoothly) as the minor color increases or as the major colors decrease.
It will be appreciated that, even though “CMY Boost” refer to cyan, magenta and yellow, these colors are merely exemplary and any other set of suitable colors may advantageously use the techniques discussed herein.
A CMY boost block may be placed in many possible locations within an image pipeline. In these embodiments, the techniques of boost block may be placed before input gamma block, immediately after GMA block. Of course, a CMY boost block can be placed in other parts of the image processing pipeline, including before or after the output gamma block.
One embodiment of the boost function proceeds as follows: the system reads in both the input data and various operating parameters respectively. For merely one embodiment, the boost block could process the input data as red, green and blue image data to affect primarily Cyan (C), Magenta (M) and Yellow (Y). Of course, it will be appreciated other color points could be made to work as well with other mixed color points that suffer simultaneous contrast issues.
Then, a set of parameters may be read in—e.g. Ymax, Cmax, Mmax, Width and Maxcol. Parameters Ymax, Cmax, Mmax and width determine the slope and intercept of the gain curves. Maxcol is the total number of colors for a given color—e.g. 255 for 8 bit data.
The system may then apply a set of conditions. Each of these conditions tests to see if there are mixed colors that might suffer simultaneous contrast. For example, IF R,G>B tests if the color is primarily yellow; IF R,B>G tests if the color is primarily magenta; and if B, G>R tests if the color is primarily cyan. If none of the three tests is satisfied, processing may proceeds down a “No” path, and no boost is made to the input color. If, however, one of the tests is satisfied, then an appropriate change to the input image color data may be made accordingly. For example, the input RGB data values could be sorted first to directly find which of the tests is the appropriate test to apply.
For each step, there could be gain curves and an exemplary formula for processing the data. In general, the processing may selectively desaturate mixed colors (e.g. C, M and/or Y) with a prescribed function in such a way as to not introduce step artifacts. In the case of example above (i.e. three mixed colors C, M or Y), three functions may be developed that depend on the location of the “boost” function (i.e. C, M or Y respectively). If there are more mixed colors to be boosted, then other functions may appropriately be added.
As noted above, the processing looks for “major colors” and “minor color” to determine which section of color space an input color image data value (e.g., an RGB value) is located. For example, if R and G are high and B is low, then the color is near yellow; if R and B are high and G is low, then the color is near magenta; and if B and G are high and R is low, then the color is near cyan. If such a condition is met, then the system seeks to adjust the level of “boost” of the low valued color, so that the boost decreases smoothly as the minor color increases or as the major colors decrease. If R and G are high and B is low, a possible function to boost for blue (B) may be computed as:
B=B+min(min(Gain—R,Gain—G)*Gain—B,maxcol)
and R and G remain the same. If R and B are high and G is low, a possible boost for green (G) is computed as:
G=G+min(min(Gain—R,Gain—B)*Gain—G,maxcol)
and R and B remain the same. If B and G are high and R is low, a possible boost for red (R) is computed as:
R=R+min(min(Gain—B,Gain—G)*Gain—R,maxcol
and B and G remain the same. Various functions may suffice for such boost processing i.e. to decrease boost—including a linear drop, as either minor color increases or major colors decrease. The slope of the function may determine how localized the boost is.
Table 3 provides a possible embodiment of computing boost functions that work for our exemplary mixed colors of yellow, cyan and magenta, respectively:
In the above example, the functions used may be a linear ramp with a max value of redmax (for cyan boost), greenmax (for magenta boost), and bluemax (for yellow boost). “Width” is a value that determines the intercept of the boost function at the y axis. These equations create a “gain” function for each color, which is used to modify the minor color (or white).
Decay Systems and Methods for Dynamic Gamut Mapping
Temporal artifacts may be visible when large changes in the backlight brightness and compensating LCD values occur. When a given portion of an image changes brightness or saturation, from one frame to another, such that it becomes desirable to change the backlight brightness, either brighter or lower, another portion of the image may not have changed. Thus, the change in backlight brightness may be accompanied by an opposite change in the LCD value. However, although the LCD is commanded to change instantaneously, the actual response of the liquid crystal material is slow to respond. This may create an optical lag condition that may create visible bright and dark “flashes”. For example, consider the graph in
It should be noted that this potential visual effect is true whether the backlight of the system is a unitary backlight (i.e. one backlight unit that illuminates the entire display area of the LCD) or whether the backlight tends to partition the LCD display area (e.g. as is found with a low resolution LED backlight where each LED is separately controllable and addressable, as in
With continued reference to
One possible method of reducing the temporal artifact is to slow both the LCD transmissivity command 2121 and backlight brightness 2111 as shown in
Another possible method of reducing the temporal artifact is to slow only the backlight brightness 2112 as shown in
Yet another possible method of reducing the temporal artifact is to slow both the LCD transmissivity command 2121 and backlight brightness 2113, with the backlight brightness 2113 slower than the LCD transmissivity command 2121 as shown in
For many of the embodiments for dynamic gamut mapping above, once a change to the backlight luminance is calculated (e.g. either from user modes without regard to particular image data values or from regarding the image data itself), this change may be made substantially instantaneously. In some cases, however, it may be desirable to delaying or decaying the changes to the backlight luminance—e.g. either the entire change or to make the changes in multiple steps so as approach the desired change. There may be two reasons for decaying any changes in the backlight value. One may be to reduce flicker when the input image is changing rapidly. Another might be to compensate for the slow response of LCD shutters when they are changed by large amounts.
Although many possible delay or decay schemes are possible to implement (and the present application encompasses these other methods), one embodiment could be a logarithmic decay algorithm which may be simple to implement. This embodiment may take a weighted average of the previous and the next value and replaces the previous value with the result. The simplest form of this is previous=(previous+next)/2 which will converge on a new value in a maximum of 8 steps when the difference between the previous and next is an 8 bit number. This is the “binary decay” formula because it moves half of the remaining distance at every step.
Another embodiment could be a more general form is a weighted logarithmic decay: previous=(previous*(1-weight)+next*weight). If the weight value is one half, this is the same as the previous formula. In an integer (hardware) environment, it is possible to represent the weight as a fixed point binary number. For example, if this is an 8 bit value, one possible formula might be:
Previous=(previous*(256−weight)+next*weight+round)/256
Where weight may be a value from 0 to 256. If Weight=128, then this is the binary decay case. If Weight=0, then this would have the effect of ignoring the next value and sticking with the previous, and Weight=256 ignores the previous value and switches to the new value at every step (rapid decay). Of course, implementing the case of 256 may not be needed in some embodiments, in which case, it may not be necessary to implement an extra bit in the registers and multipliers to handle this 9 bit number.
If the above formula is implemented in integer arithmetic then, if the round variable has the value of zero, the formula may not converges on a constant next value that is higher than the previous value. If the round variable is 128 (one half the divisor) then the formula may not converge on a constant next value that is lower than the previous value. One embodiment might be to set the round value based on the difference between the previous and next values:
If this test is done beforehand, then the formula may converge in either direction. An 8 bit fixed point weight value might use 8*8=16 bit multipliers, with 8 bits being shifted off after the additions. Also note that in binary calculations, the value (256−weight) may be calculated by inverting every bit in the weight value.
If a lower gate count is desired, the number of bits in the weight value may be reduced. However, this may decrease the number of different decay rates that to choose from. For example, if the weight value only has 4 bits, then there may be only 16 weight values to choose from, the round value will be set to 15 for converging up, and the multipliers would be 8*4=12 bits with 4 bits shifted off after the additions.
Because LCD shutters converge to a new value at different rates when going up than when going down, it is possible in one embodiment to have two separate registers to contain the decay rate for increasing separate from decreasing. Since the round value may be already being calculated based on the direction of the change, the weight value may be selected from two different registers based on the same test result.
To implement separate decays for the LCD and the backlight (e.g. LED backlight),
It should be appreciated that the dynamic settings could be set for going lower than 50% backlight level. It could be set to go as low as 25% under normal control. However, it may be desirable to have a special case for setting the backlight to zero (0%) percent in the event of the screen going completely black to save even further on power.
As individual display system's response may be dependent upon the particular LCD response and particular backlight response, it may be desirable to tune such a system experimentally or even heuristically. Such values that work well for a system could be register settings in hardware. For example, some visual artifacts may be caused by a slow LC response. If a sudden change in the backlight value XL were to occur in response to a change in the peak color value, the LCD transmissivity change may lag. For example, when the backlight value XL goes up, the LCD drive value for a fixed input value color may go down. If the LCD is slow, the transmissivity is momentarily too high, it may cause a “flash”. When the backlight value XL goes down, the LCD drive value goes up, but with a slow LC response, the transmissivity may be momentarily too dark, causing a “dark flash”. If the backlight oscillates (without damping from decay), the lagging LCD may allow the backlight changes to be visible.
Thus, slowing down the XL value change may tend to reduce the artifact by reducing the change per frame, and thus the magnitude of the lag error. However, if the system can use two different XL values, one for the backlight and one for the X/XL calculation, then it may use slightly different logarithmic decay time constants so that the backlight follows the real LC response to further reduce the artifact. It may also be desirable to have different decay constants in each direction since the LC response is different in each direction.
Example System and Simulation Pseudo-Code
Table 4 below is one possible example system made in accordance with the present invention by describing a hardware implementation in conjunction with a computer simulation. This is written in the language Lua (see www.lua.org) which serves as a simulation language and also as pseudo-code description and implementation of one embodiment of the current invention. This embodiment is based on the design shown in
In this simulation, the backlight (in this case, an LED backlight but other backlights may be possible) is designed to normally come on at the minimum power level (e.g. above 25%) necessary for the image being displayed. The LCD values are scaled in a manner that results in power levels above 50% only when colors are out-of-gamut, (OOG). There are switches that cause only yellow OOG pixels to be counted and causes the OOG values to be scaled by a smaller amount when there are fewer pixels OOG. However, if an image has a sufficient number of bright saturated colors going out-of-gamut, (OOG) then the LED power is increased and the LCD values are decreased.
The simulation has several parameters that will probably be fixed numbers in any given design. It has a variable, GAMBITS, indicating how many bits in the gamma pipeline. If this value changes then the input and output gamma tables may be changed to match. Since the number of bits in the LED backlight controller may not always be the same there is a variable, LEDBITS, that holds this number. There are several constants based on this, LEDMAX which is the maximum LED value, LEDhalf which is the half-LED power level, and LEDquart which is the cutoff point below which the LED power will not go. There is one exception, if the image is completely black the LED power is lowered to zero.
In
r=math.floor(r*(DBLC+LEDhalf+1)/LEDMAX)
However, this involves dividing by LEDMAX which is a power-of-two-minus-one. Since the range of LED values tends to be small, this division may be replaced with a table that has each divisor inverted to a multiplier. The table may be build with the following loop:
for i=0,31 do
DLUT[i]=math.floor(2ˆ8*(i+LEDhalf+1)/LEDMAX)
end
This produces a table of 8 bit values and the formula for red above becomes:
r=math.floor(r*DLUT[DBLC]/(2ˆ8))
With a similar formula for green and blue multiplied by the same number. After DBLC correction the input values may be optionally modified in PRE SCALE module 3008. This module performs the operations described above and is not necessary for the understanding of this embodiment of the Dynamic Gamut Mapping system 3000.
Peak Survey
The peak survey module 3012 keeps statistics needed to calculate how far bright saturated colors go out of gamut. This operation is simulated in the doscan function in Table 4. The doscan function is called once for every input pixel in an entire frame. In this design the peak survey module requires un-scaled RGBW values which are not available. For this reason the peak survey module may do a separate GMA calculation to find the maximum OOG values. Fortunately, a complete RGBW GMA may not be required to calculate the maximum values. The simulation in Table 4 calls a version of the GMA (not shown) that only calculates the maximum primary (maxp) value. In addition the peak module has the following parameters, many of which will be registers in a hardware version:
yellow_only
weighted_oog
cutoff
The peak survey module collects the following statistics as it looks at every pixel in an image. These may be initialized to zero before surveying an image.
peakval
The highest value of any primary in any color in the image. This value has one more bit than values in the gamma pipeline. So it ranges from 0 to MAXCOL*2+1. Peakval is modified, if yellow_only is enabled, to count only yellow OOG colors in the survey.
numoog
The number of pixels that were OOG, clamped to cutoff. This value never becomes larger than cutoff, which is typically limited to 16 bits.
Calc, Store, & Decay LED Power Module:
During vertical retrace, once all the pixels in an image have been surveyed, the LED backlight power may be calculated. First, the peakval may be scaled based on the number of pixels OOG if the weight oog flag is set.
Next the LED power level may be calculated by scaling the peakval to the range of the LED controller, preventing it from dropping below 25% except in the case of a black image:
The LEDy value may be inverted to produce a multiplier for every pixel in the next image. Calculating the inverse requires doing a division, and there may be time in the vertical retrace to do this using an iterative algorithm for the divide. However, the number of possible LED values is small and building a table allows presetting special values in different ranges. Table 4 includes pseudo-code that generates a version of this table.
There may be an extra power of two in the denominators of the formula for that table. This may result in OOG values that range from 0 to 2*MAXCOL being scaled down by 50% to always be in gamut. This may also result in the overXL table having one less bit of precision and fewer gates. This table may typically be 9 bits wide (values between 0 and 511) when LED power is limited to 25% or higher and INVBITS=8 (INVMUL=256). With this table, inverting the LEDy value is a table lookup:
INVy=overXl[LEDy]
The resulting INVy value, when multiplied by input pixel values and divided by INVMUL, may scale images with OOG values down into gamut, scale all-in-gamut images by 1.0 and scale dim images up.
Weighted Logarithmic Decay:
Another function that Calc+Store+Decay module 3010 may do is prevent sudden changes in the LED power from being sent directly to backlight controller 3018.
previous=(previous*(WMUL−weight)+next*weight+round)/WMUL
Where weight is a value from 0 to WMUL−1. Weight=WMUL/2 is the binary decay case. The above formula may be less desirable to implement in integer arithmetic. If the round variable has the value of zero then the formula may never converge on a constant next value that is higher than the previous value. If the round variable is WMUL−1 then the formula may not converge on a constant next value that is lower than the previous value. One embodiment might be to set the round value based on the difference between the previous and next values:
If this test is done beforehand, then the formula may converge in either direction. In
previous=(previous*(WMUL−weight)+next*XMUL*weight+round)/WMUL
Previous latch 2803 may now be large enough to store the XBITS extra bits. Since the next value input does not have these bits, it may be modified by barrel shifter 2805 before comparing it with the previous latch in comparator 2801. Also and the value output to the LED backlight controller is now:
previous>>XBITS
This operation may be performed by barrel shifter 2804.
In some cases, increasing XBITS by one may add about 5 frame times to a response to a large change with a small weight. When weight=2 WBITS=4, XBITS=0, then decaying from 0 to 127 takes around 26 frame times. If XBITS=4 then the decay takes 46 frame times.
Note that there are many optimizations in the above formula. Dividing by WMUL is a right shift, shown as barrel shifter 2806. The two multiplies 2807 need to be (LEDBITS+XBITS)*WBITS in size, but since the lower bits of next*XMUL may be zero, this multiplier could be only (LEDBITS)*WBITS in size followed by a left shift shown as barrel shifter. The value (WMUL−weight) can be easily calculated by inverting every bit in the weight value in inverter 2809.
If gate count is an issue, the number of bits in the weight value may be reduced. This may decrease the number of different decay rates to choose from. For example, if the weight value only has 4 bits, then there may be 16 weight values to choose from, the round value may be set to 15 for converging up, and the multipliers may have to multiply by 4 bit values and discard 4 bits afterwards. Note that this parameter may be independent of the effect of the XBITS parameter. Both WBITS and XBITS may be chosen as fixed values in the design stage of a display.
Because LCD shutters converge to a new value at different rates when going up than when going down, it may be advantageous to have two separate registers 2810 and 2811 to contain the decay rate for increasing separate from decreasing. Which of these two weight values to use can be selected in multiplexer 2812 using the results of comparator 2801.
As discussed above, there are two reasons for decaying any changes in the backlight value. One is to reduce flicker when the input image is changing rapidly. Another is to compensate for the slow response of LCD shutters when they are changed by large amounts. To implement both,
X/Xl:
Returning to
After leaving X/Xl module 3014 the values may be piped to RGBW GMA module 3016 that may contain a gamut clamping to luminance algorithm. The resulting RGBW values may be filtered in Sub Pixel Rendering module 3001, corrected for output in Out Gamma module 3005 and sent to LCD array 3009.
This application claims the benefit of U.S. Provisional Application 60/803,855 entitled “Color Imaging Backlight for Color Display Systems and Methods of Operation” filed on Jun. 2, 2006; U.S. Provisional Application 60/827,710 entitled “Systems and Methods for Reducing Desaturation of Images Rendered on High Brightness Displays” filed on Sep. 30, 2006; U.S. Provisional Application 60/828,594 entitled “Display Systems and Methods Having Dynamic Virtual Primaries” filed on Oct. 6, 2006; and U.S. Provisional Application 60/891,668 entitled “High Dynamic Contrast Display System Having Multiple Segmented Backlight” filed on Feb. 26, 2007, which are hereby incorporated by reference herein.
| Number | Date | Country | |
|---|---|---|---|
| 60891668 | Feb 2007 | US | |
| 60828594 | Oct 2006 | US | |
| 60827710 | Sep 2006 | US | |
| 60803855 | Jun 2006 | US |
