Sub-pixel rendering system and method for improved display viewing angles

Abstract

System and methods are disclosed for improving the off-normal axis viewing angle by applying different filters if one colored sub-pixel data is driven close to 100% luminance while other colored sub-pixel data is driven close to 50% luminance values. Systems and methods for adjusting the viewing characteristics of the display system are also disclosed.

Description

BACKGROUND

In commonly owned United States Patent Applications: (1) U.S. patent application Ser. No. 09/916,232 (“the '232 application”), entitled “ARRANGEMENT OF COLOR PIXELS FOR FULL COLOR IMAGING DEVICES WITH SIMPLIFIED ADDRESSING,” filed Jul. 25, 2001, now issued as U.S. Pat. No. 6,903,754; (2) U.S. patent application Ser. No. 10/278,353 (“the '353 application”), entitled “IMPROVEMENTS TO COLOR FLAT PANEL DISPLAY SUB-PIXEL ARRANGEMENTS AND LAYOUTS FOR SUB-PIXEL RENDERING WITH INCREASED MODULATION TRANSFER FUNCTION RESPONSE,” filed Oct. 22, 2002, and published as United States Patent Application Publication No. 2003/0128225; (3) U.S. patent application Ser. No. 10/278,352 (“the '352 application”), entitled “IMPROVEMENTS TO COLOR FLAT PANEL DISPLAY SUB-PIXEL ARRANGEMENTS AND LAYOUTS FOR SUB-PIXEL RENDERING WITH SPLIT BLUE SUB-PIXELS,” filed Oct. 22, 2002, and published as United States Patent Application Publication No. 2003/0128179; (4) U.S. patent application Ser. No. 10/243,094 (“the '094 application), entitled “IMPROVED FOUR COLOR ARRANGEMENTS AND EMITTERS FOR SUB-PIXEL RENDERING,” filed Sep. 13, 2002, and published as United States Patent Application Publication No. 2004/0051724; (5) U.S. patent application Ser. No. 10/278,328 (“the '328 application”), entitled “IMPROVEMENTS TO COLOR FLAT PANEL DISPLAY SUB-PIXEL ARRANGEMENTS AND LAYOUTS WITH REDUCED BLUE LUMINANCE WELL VISIBILITY,” filed Oct. 22, 2002, and published as United States Patent Application Publication No. 2003/0117423; (6) U.S. patent application Ser. No. 10/278,393 (“the '393 application”), entitled “COLOR DISPLAY HAVING HORIZONTAL SUB-PIXEL ARRANGEMENTS AND LAYOUTS,” filed Oct. 22, 2002, and published as United States Patent Application Publication No. 2003/0090581; and (7) U.S. patent application Ser. No. 10/347,001 (“the '001 application”) entitled “IMPROVED SUB-PIXEL ARRANGEMENTS FOR STRIPED DISPLAYS AND METHODS AND SYSTEMS FOR SUB-PIXEL RENDERING SAME,” filed Jan. 16, 2003, and published as United States Patent Application Publication No. 2004/0080479, novel sub-pixel arrangements are therein disclosed for improving the cost/performance curves for image display devices and which are herein incorporated by reference.

These improvements are particularly pronounced when coupled with sub-pixel rendering (SPR) systems and methods further disclosed in those applications and in commonly owned United States Patent Applications: (1) U.S. patent application Ser. No. 10/051,612 (“the '612 application”), entitled “CONVERSION OF RGB PIXEL FORMAT DATA TO PENTILE MATRIX SUB-PIXEL DATA FORMAT,” filed Jan. 16, 2002, which was published as United States Patent Application Publication No. 2003/0034992, and is now issued as U.S Pat. No. 7,123,277; (2) U.S. patent application Ser. No. 10/150,355 (“the '355 application”), entitled “METHODS AND SYSTEMS FOR SUB-PIXEL RENDERING WITH GAMMA ADJUSTMENT” filed May 17, 2002, and published as United States Patent Application Publication No. 2003/0103058; and (3) U.S. patent application Ser. No. 10/215,843 (“the '843 application”), entitled “METHODS AND SYSTEMS FOR SUB-PIXEL RENDERING WITH ADAPTIVE FILTERING,” filed Aug. 8, 2002 and published as United States Patent Application Publication No. 2003/0085906, which are hereby incorporated herein by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in, and constitute a part of this specification illustrate exemplary implementations and embodiments of the invention and, together with the description, serve to explain principles of the invention.

FIG. 1 depicts an observer viewing a display panel and the cones of acceptable viewing angle off the normal axis to the display.

FIG. 2 shows one embodiment of a graphics subsystem driving a panel with sub-pixel rendering and timing signals.

FIG. 3 depicts an observer viewing a display panel and the possible color errors that might be introduced as the observer views sub-pixel rendered text off normal axis to the panel.

FIG. 4 depicts a display panel and a possible cone of acceptable viewing angles for sub-pixel rendered text once techniques of the present application are applied.

FIG. 5A shows one possible sub-pixel repeat grouping displaying a “white” line on a display having off-normal axis color error.

FIG. 5B shows a set of curves of brightness versus viewing angle on a LCD display depicting the performance of the image shown in FIG. 5A.

FIG. 6A shows an alternative technique of rendering a “white” line on a display with the same sub-pixel repeat grouping as in FIG. 5A but rendered with less off-normal axis color error.

FIG. 6B shows a set of curves of brightness versus viewing angle on a LCD display depicting the performance of the image shown in FIG. 6A.

FIG. 7 shows a set of curves of contrast ratio versus viewing angle.

FIG. 8 shows a laptop having a number of different embodiments for adjusting the viewing characteristics of the display by the user and/or applications.

DETAILED DESCRIPTION

Reference will now be made in detail to implementations and embodiments, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

FIG. 1 shows a display panel 10 capable of displaying an image upon its surface. An observer 12 is viewing the image on the display at an appropriate distance for this particular display. It is known that, depending upon the technology of the display device (liquid crystal display LCD, optical light emitting diode OLED, EL, and the like) that the quality of the displayed image falls off as a function of the viewing angle. The outer cone 14 depicts an acceptable cone of viewing angles for the observer 12 with a typical RGB striped system that is not performing sub-pixel rendering (SPR) on the displayed image data.

A further reduction in acceptable viewing angle for high spatial frequency (HSF) edges (i.e. inner cone 16) may occur when the image data itself is sub-pixel rendered in accordance with any of the SPR algorithms and systems as disclosed in the incorporated applications (i.e. the '612, '355, and '843 applications) or with any known SPR system and methods. One embodiment of such a system is shown in FIG. 2 wherein source image data 26 is placed through a driver 20 which might include SPR subsystem 22 and timing controller (Tcon) 24 to supply display image data and control signals to panel 10. The SPR subsystem could reside in a number of embodiments. For example, it could entirely in software, on a video graphics adaptor, a scalar adaptor, in the TCon, or on the glass itself implemented with low temperature polysilicon TFTs.

As stated in the '612 application, issued as U.S. Pat. No. 7,123,277, sub-pixel rendering (SPR), in its most simplistic implementation, operates by using the sub-pixels as approximately equal brightness pixels perceived by the luminance channel. This allows the sub-pixels to serve as sampled image reconstruction points as opposed to using the combined sub-pixels as part of a “true” pixel. By using sub-pixel rendering, the spatial sampling is increased, reducing the phase error.

A real world image is captured and stored in a memory device. The image that is stored was created with some known data arrangement (i.e., a first format). The stored image can be rendered onto a display device using an array that provides an improved resolution of color displays. The array is comprised of a plurality of three-color pixel elements (i.e., a second format) having at least a blue emitter (or sub-pixel), a red emitter, and a green emitter, which when illuminated can blend to create all other colors to the human eye.

If the arrangement of the sub-pixels is optimal for sub-pixel rendering, sub-pixel rendering provides an increase in both spatial addressability to lower phase error and in Modulation Transfer Function (MTF) high spatial frequency resolution in both axes.

Incoming RGB data is treated as three planes over lying each other. To covert the data from the RGB format, each plane is treated separately. Displaying information from the original format on more efficient sub-pixel arrangements requires a conversion of the data format via resampling. The data is resampled in such a fashion that the output of each sample point is a weighting function of the input data. Depending on the spatial frequency of the respective data samples, the weighting function may be the same, or different, at each output sample point.

To determine the values for each emitter, first one must create transform equations that take the form of filter kernels. The filter kernels are generated by determining the relative area overlaps of both the original data set sample areas and target display sample areas. The ratio of overlap determines the coefficient values to be used in the filter kernel array.

A method of converting a source pixel data of a first format for a display of a second format having a plurality of three-color pixel elements comprises determining implied sample areas for each data point of each color in source pixel data of a first format. The resample areas for each emitter of each color in the display are also determined. A set of fractions for each resample area is formed. The denominators are a function of the resample area and the numerators are the function of an area of each of the implied sample areas that at least partially overlaps the resample area. The data values for each implied sample area are multiplied by its respective fraction and all products are added together to obtain luminance values for each resample area, to produce output image data in the second format.

To render the stored image onto the display device, the reconstruction points are determined in each three-color pixel element. The center of each reconstruction point will also be the source of sample points used to reconstruct the stored image. Similarly, the sample points of the image data set are determined. Each reconstruction point is located at the center of the emitters (e.g., in the center of a red emitter). In placing the reconstruction points in the center of the emitter, a grid of boundary lines is formed equidistant from the centers of the reconstruction points, creating sample areas (in which the sample points are at the center). The grid that is formed creates a tiling pattern. The shapes that can be utilized in the tiling pattern can include, but is not limited to, squares, rectangles, triangles, hexagons, octagons, diamonds, staggered squares, staggered rectangles, staggered triangles, staggered diamonds, Penrose tiles, rhombuses, distorted rhombuses, and the like, and combinations comprising at least one of the foregoing shapes.

The sample points and sample areas for both the image data and the target display having been determined, the two are overlaid. The overlay creates sub-areas wherein the output sample areas overlap several input sample areas. The area ratios of input to output is determined by either inspection or calculation and stored as coefficients in filter kernels, the value of which is used to weight the input value to output value to determine the proper value for each emitter.

The reduction in acceptable viewing angle described herein is primarily caused by color artifacts that may appear when viewing a sub-pixel rendered image because HSF edges have different values for red, green, and blue sub-pixels. For one example using SPR on the design in FIG. 5A, black text on white background, the brightness level of the green sub-pixels will switch between 100% and 0% while the brightness level of the red and blue sub-pixels will switch between 100% to 50%.

FIG. 3 depicts the situation as might apply to sub-pixel rendered black text 30 on a white background. As shown, observer 12 experiences no color artifact when viewing the text substantially on the normal axis to the panel 10. However, when the observer “looks down or up” on the screen, the displayed data may show a colored hue on a liquid crystal display (LCD), which is due to the anisotropic nature of viewing angle on some LCDs for different gray levels, especially for vertical angles (up/down). Thus it would be desirable to perform corrections to the SPR data in order to increase the acceptable viewing angle 40 of SPR data, as depicted in FIG. 4.

For illustrative purposes, FIGS. 5A and 5B depict why these color artifacts arise. FIG. 5A shows one possible sub-pixel arrangement upon which SPR may be accomplished, as further described in the above incorporated applications. Sub-pixel repeat group 52 comprises an eight sub-pixel pattern having blue 54, green 56, and red 58 sub-pixels wherein the green sub-pixels are of a reduced width as compared with the red and blue sub-pixels (e.g. one half or some other ratio). In this particular example, a single “white” line is drawn—centered on the middle column of green sub-pixels. As measured on the normal axis, each of the green sub-pixels in the middle column are fully illuminated at 100% brightness level; the blue and the red sub-pixels are illuminated at 50% brightness. Put another way, the green sub-pixel is operating with a filter kernel of [255] (i.e. the “unity” filter, and where ‘255’ is 100% on a digital scale); while the blue and red sub-pixels have a filter kernel of [128 128] (i.e. a “box” filter—where ‘128’ is 50% on a digital scale). At zero viewing angle (i.e. normal to the display), a “white” line is shown because the red and blue sub-pixels are twice as wide as the green sub-pixels. So with G˜100, R˜50, B˜50, a chroma-balanced white is produced at 100-2×(50)-2×(50), for the case where the size ratio of red to green or blue to green is 2:1. If the size ratio is other than 2, then the multiplier will be adjusted appropriately.

FIG. 5B depicts two curves—the 100% and 50% brightness curve vs. viewing angle—as is well known for displays such as LCDs. The green sub-pixel performs as the 100% brightness curve; while the blue and red sub-pixels follow the 50% curve. At the normal axis (i.e. viewing angle at 0 degrees), the SPR works well and there is no additional color artifact. As the viewing angle increases to angle {circumflex over (-)}_UP, then the observer would view a fall-off of Δ_G in the green sub-pixel brightness—while viewing a Δ_R,B fall-off in the brightness of either the red or the blue sub-pixel brightness. Thus, at {circumflex over (-)}_UP, there is G′˜80, R′˜20, B′˜20, which results in the image of the white line assuming a more greenish hue—e.g. 80-2×(20)-2×(20). For angle {circumflex over (-)}_DOWN, the green pixels will again fall off an amount Δ_G, while the red and blue sub-pixels will actually rise an amount Δ_R,B. In this case, the white line will assume a magenta hue.

So, to correct for this color artifact, it might be desirable to drive the green sub-pixels—and possibly the red and blue sub-pixels—on a different curve so that the delta fall-off in the green vs the red/blue sub-pixels better match each other as a relative percentage of their total curve. In one embodiment, the green sub-pixels are driven with an “1×3” filter (i.e. a “tent” filter). As discussed further below, this new filter decreases the luminance of the green on high frequency edges so it is closer to the red and blue values.

One embodiment of such a correction is depicted in FIGS. 6A and 6B. In FIG. 6A, a new sub-pixel arrangement is creating the “white” line. Three columns of green sub-pixels are used—with luminances at the 12.5%, 75%, and 12.5% respectively for the left, middle and right green sub-pixel columns. The red and blue sub-pixel checkerboard columns are left at 50%. So, at normal viewing angle (i.e. θ=0), with G˜12.5+75+12.5, R˜50, B˜50, a similar chroma-balanced “white” line is produced, centered on the middle column of green sub-pixels. Stated in another way, the green sub-pixels are operating on a different tent filter of [32, 192, 32], while the red and blue sub-pixels are operating on the same filter [128 128]—as will be explained further below.

To see what the effect is off-normal axis viewing, refer to FIG. 6B. The 75% and 12.5% curves are much closer in shape to the 50% curve than the 100% curve. Thus the curves are more proportionately constant over viewing angle and the color hue will stay “white”.

It will be appreciated that other curves upon which to drive different colored sub-pixels may suffice for the purposes of the present invention. It suffices that the Δ drop in different colors match sufficiently close enough for acceptable viewing performance (i.e. no unacceptable color error at off-normal axis viewing). It will also be appreciated that the same technique of reducing color error will work for other sub-pixel repeat grouping and the discussion contained herein for the particular repeat sub-pixel grouping of FIG. 5A is also merely for illustrative purposes. For any sub-pixel repeat grouping, a set of curves should be appropriately selected to give acceptable viewing performance. Such curves might also vary depending upon the respective geometries of the different colored sub-pixels. Thus, as green sub-pixels are half the width as red and blue sub-pixels in FIG. 5A, an appropriate choice of curves should take such geometries into consideration.

Use of Adaptive Filtering and Gamma Correction

The techniques described herein may also be used in combination with—and may be enhanced by—other processing techniques; such as adaptive filtering and gamma correction, as disclosed in the '843 application and the '355 application. For example, and as previously noted, the color errors introduced by the off-normal axis viewing angles are more noticeable at regions of high spatial frequencies—such as at edges and other sharp transitions. Thus, detecting areas of high spatial frequency might be important in selectively using the techniques described above for those particular areas.

For example, at an edge transition from light to dark, the green sub-pixel value (operating with the unity filter) goes from 255 to 0 on the aforementioned digital scale. The red and blue sub-pixels (utilizing the box filter) are set to 128 each. Since the viewing angle of 255 and 128 are significantly different for twisted-nematic TN LCDs, there is a color shift. On the other hand, if the green filter is [32 191 32] then the green value goes from 255 to 224 to 32 to 0 (four successive values). The viewing angle characteristics of 224 and 32 are closer to the 128 values (than 255 or 0) of red and blue, so there is less color shift. While there is some loss of sharpness, it is not very noticeable. In addition, gamma correction could also be applied to green or red or blue to improve color matching. More generally, symmetric tent filters for green can be formulated by [f, 1−2f, f]×255. The value for “f” can be anywhere in the 0-20% of total luminance without adversely affecting the “sharpness” of high spatial frequency information, such as text. For LCDs rendering only images, such as television, “f” can be much higher with acceptable results. In addition, the tent filter can be oriented in other directions, such as vertical. In this case, the tent filter would have the values:

32
192
32

A diagonal filter could also be employed.

Other embodiments—different from the symmetric tent filter for operating the green sub-pixels—are asymmetric box filters, such as [192 63] or [63 192]. These filters also improve the sharpness, but still preserve the improved color performance vs. angle. The new values for an edge (255 to 192 to 63 to 0) are closer to the 128 values of red and blue, so the viewing angle performance may be improved. In this case, there may be an observed asymmetry to the data for left and right edges of a black stroke of a width greater than 1 pixel. In these cases, adaptive filtering can be used to detect whether the edge is “high to low” or “low to high” by looking at 4 pixels in the data set. When high to low is detected, the filter may be [63 192]; for low to high, it may be [192 63]. The adaptive filtering detection is this case is “1100” for high to low or “0011” for low to high, as is further described in the '843 application.

In either case, it is only necessary to employ the tent filter or asymmetric box filter at bright to dark transitions such as black text, where the color error is noticeable. Adaptive filtering can be used to detect light to dark transitions and apply the new filter. Several options exist; in all cases the magnitude of the “step” in brightness can be set by a separate test. The following are representative test cases:

(1) Detect white to black (black text) by looking at all three colors; if all colors change, then apply tent or asymmetric box filter to green, else apply unity filter to green and box filter for red and blue.

(2) Detect bright green to dark green transition but no red and blue transition, then use unity filter for green, box filter for red and blue. It should be appreciated that there might be no need to compensate for viewing angle in this case.

(3) Detect black to white transition (white text) then apply tent or asymmetric box filter to green and box filter to red and blue. For correct brightness, gamma should be applied.

(4) Detect dark green to bright green but no red or blue transition, then use unity filter for green, box filter for red and blue (with gamma). It should be appreciated that there might be no need to compensate for viewing angle in this case.

(5) For red and blue dark to light transitions, it may be desirable to use the standard box filter together with gamma correction. For red and blue light to dark transitions, it may be desirable to use the standard box filter without gamma correction to enhance the darkness of the text strokes.

In all of these cases where gamma is applied, the value of gamma can be selected to obtain best overall performance for that display. It may be different than the gamma of the display.

External Adjustments of Viewing Parameters for Different Viewing Conditions

SPR techniques are typically optimized for each sub-pixel layout and the values are stored in an ASIC, FPGA, or other suitable memory/processing systems. Certain tradeoffs might be desirable according to the preferences of the users. For example, the degree of sharpness of text (or other high spatial frequency information), optimal viewing angle, and color error vs. sharpness conditions are some of the viewing parameters that might be controlled either by applications utilizing the graphical subsystem or by the user itself.

The degree of sharpness may be controlled by varying the filter coefficients as follows:

No Sharpness:
0	1	0
1	4	1
0	1	0

Intermediate Sharpness:
−¼	1	−¼
1	5	1
−¼	1	−¼

Full Sharpness:
−½	1	−½
1	6	1
−½	1	−½

To control the level of sharpness, the graphic subsystem (such as one embodiment shown as subsystem 20 in FIG. 2) might contain a register containing a value corresponding with varying levels of sharpness (e.g. like the three levels shown above). Either the user could select the sharpness through a physical switch on the system (e.g. PC, or any external display) or a software switch (e.g. Control Panel setting) or an application sending image data to the graphical subsystem could automatically alter viewing settings

Alternatively, gamma table values can be adjusted under user control. For example, a low gamma value is desirable for black text; but higher values may be desired for white text. Gamma changes can be either different lookup tables or different functions applied to data. The gamma values can be either the same for positive and negative transitions, or can be different, depending on the display characteristics.

Yet another adjustment input is to adjust peak contrast ratio as a function of viewing angle. LCDs have a peak contrast ratio at a given angle that is set by the voltage applied. This voltage is typically set at the factory and cannot be adjusted by the user. However, it may be desirable to be able to adjust the peak viewing angle—e.g. for black text or high spatial frequency information.

Using the SPR data processing, the voltage corresponding to “100% ON” can be effectively changed by changing the filter coefficients—e.g. for the green sub-pixels in the repeat grouping as shown in FIG. 5A. In a display having a repeat sub-pixel grouping, such as found in FIG. 5A, the peak contrast ratio is determined mostly by the green data—red and blue data contribute but not as much. Even a 5–10% adjustment by the system or by the user would improve viewing conditions based on viewing angle. FIG. 7 depicts a series of three curves plotting contrast ratio vs. viewing angle at three levels of luminance—100%, 90%, and 80%. As may be seen, the peak contrast ratio is achieved at different viewing angles for different luminance levels. This is particularly so in the vertical axis for twisted-nematic TN LCD displays.

To adjust viewing characteristics such as contrast ratio for the particular user's viewing angle, FIG. 8 depicts a number of separate embodiments for performing such adjustments. Laptop 80 is one possible display platforms to allow such user adjustments. Other platforms might be monitors, cell phones, PDAs and televisions. A first embodiment is a manual physical switch 82 that a user would adjust to get a proper contrast ratio for the user's particular viewing angle. A second embodiment might be a switch in software (shown as a window 84) that allows the user to select a possible contrast ratio setting. Such a soft switch might be activated by individual applications (e.g. word processors, spreadsheet or the like) that access and render data on the display or by the operating system itself. A third embodiment might be automatic adjustment as performed by a switch 86 that notes the angle between the keyboard of the laptop and the display screen itself. This angle would be sufficient to infer the viewing angle of the user with respect to the screen. Based on this inferred viewing angle, the system could automatically adjust the contrast ratio accordingly. A fourth embodiment might be a eye tracking device 88 that notes the position of the user's head and/or eyes and, from that data, calculate the user's viewing angle with respect to the screen.

Claims

1. A method for sub-pixel rendering source image data onto a display, the steps of said method comprising: sub-pixel rendering said source image data; andsubstituting different filter kernels when a first colored sub-pixel data would be driven to substantially 100% luminance and a second colored sub-pixel data neighboring said first colored sub-pixel data would be driven to substantially 50% luminance such that said first colored sub-pixel data and said second colored sub-pixel data are driven to substantially closer luminance values.

2. The method as recited in claim 1 wherein the step of substituting different filter kernels further comprises: selecting a filter kernel such that the neighboring sub-pixels retain a substantially same chroma value obtained when applying an original filter kernel.

3. A display system comprising: a graphics subsystem receiving source image data and outputting display image data;a display panel coupled to said graphics subsystem; andsaid graphics subsystem further comprising a sub-pixel rendering subsystem wherein said sub-pixel rendering subsystem applies a different filter kernel to a first colored sub-pixel that would be driven to substantially 100% luminance when neighboring second colored sub-pixels would be driven to substantially 50% luminance.

4. The display system as recited in claim 3 wherein said system further comprises: means for allowing the user to adjust viewing characteristics of said system.

5. The display system as recited in claim 4 wherein said means for adjusting further comprises one of a group, said group comprising a physical switch, a software switch, a switch actuated by the angle between the display and the keyboard, and a eye tracking device.

6. The display system as recited in claim 3 wherein said display is a liquid crystal display.

7. A graphics subsystem for a display system comprising: an input to receive sub-pixel data; anda sub-pixel rendering subsystem configured to apply a different filter kernel to first colored sub-pixel data received from the input when the first colored sub-pixel data would be driven to substantially 100% luminance at the same time that neighboring second colored sub-pixel data received from the input would be driven to substantially 50% luminance.