Patent Grant
6989837
The present invention relates to the field of image processing. More particularly, the present invention relates to improving the access to a frame buffer containing 4:2:0 format video data.
The human eye has three types of photoreceptor cells called cones. Each of the three types of cones responds differently to light of different wavelengths. Accordingly, light with different wavelength content is perceived as light of different color. Color display systems have existed for a long time with knowledge of how human perception works; however, the manner of displaying colors on electronic systems continues to develop.
A basic color display system uses an RGB color system. In such a system, R corresponds to a red color source, G to a green color source and B to a blue color source. By selecting the wavelength content of the RGB color sources, a color gamut is achieved. Thus by blending or combining different amounts of either the red, green, or blue source, a wide range of colors can be achieved. From this color gamut, it is therefore important to mathematically represent a color from a set of colors. Such mathematical representation is called a color space.
RGB is an additive color system in which the three primary colors red, green, and blue are added to form a desired color. In the RGB color space, each component has a range of 0 to 255. When the three sources are at zero, the resulting color is black; when the three sources are at 255, the resulting color is white.
For an RGB color system, an RGB color space can be used. There are, however, other color spaces available including YIQ, YUV, YCbCr, and CMYK. Mathematical conversions are available to convert from one color space to another. Of particular interest in the present invention, is the YCbCr color space which is commonly used in broadcast and television systems as well as computer graphics implementations.
Different color spaces have historically evolved for different applications. In each case, a color space was chosen for reasons that may no longer be applicable. A choice may have been made on a particular color space because certain mathematical elements were simpler or faster to process. The reason for a particular choice may have been reduced memory requirements or reduced bandwidth on digital buses.
Whatever historical reasons caused color space choices in the past, the convergence of computers, the Internet, and a wide variety of video devices, all using different color representations, now forces the digital designer today to convert between them. The objective is to have a common color space to which all inputs are converted before algorithms and processes are executed. The converters are useful for a number of markets, such as image processing and filtering. Their basic function is to convert from one color space to another.
Turning to another widely used color space, the YCbCr color space was developed as part of the Recommendation ITU-R BT.601 as a worldwide digital component video standard and is used in television transmissions. YCbCr is a scaled and offset version of the YUV color space, in which Y represents luminance (or brightness), U and V components represent the color information of hue and saturation. In the YCbCr color space, the RGB color space is separated into a luminance part, Y, and two chrominance parts, Cb and Cr. A historical reason for choosing YCbCr over RGB is that YCbCr reduces storage and bandwidth requirements. In developing the YCbCr color space, it was considered that the human eye is more sensitive to a change in brightness than a change in color.
To generate the same color in the RGB format, all three-color components should be of equal bandwidth. This requires more storage space and bandwidth. Also, processing an image in the RGB space is more complex since any change in the color of any pixel requires all the three RGB values to be read, calculations performed, and then stored. However, storing color information in the luminance, chrominance format can help some processing steps to be faster. The result is that Cb and Cr provide the hue and saturation information of the color and Y provides the brightness information of the color. Y is defined to have a range of 16 to 235 and Cb and Cr have a range of 16 to 240 with 128 equal to zero. Because the human eye is less sensitive to Cb and Cr, engineers did not need to transmit Cr and Cb at the same rate as Y. Less storage and bandwidth was needed, resulting in reduced design costs.
Thus, where storage and bandwidth are a design consideration, as in many modem digital graphic systems, the YCbCr color space is the preferred space. Within the YCbCr color space there are various formats including the 4:4:4, 4:2:2, 4:1:1, and the 4:2:0 formats. These various formats provide different levels of compression with certain tradeoffs. The 4:2:0 format, in particular, provides certain compression advantages, but because of the manner in which it is implemented, memory bandwidth becomes a problem. Display systems that implement the 4:2:0 format typically use paged memory devices (e.g., SRAM or DRAM) where the Y, Cb, and Cr information are stored in different pages of memory. Moreover, a typical system retrieves information for two pixels at a time by retrieving two items of Y information, and one item each of Cb and Cr information. Thus, retrieving color information for a pair of pixels in a prior art system, requires three page breaks, each page break being the closing of a page and the opening of a different page. Page breaks are time-consuming operations.
It is therefore desirable to provide a computer display system having a memory with improved memory bandwidth. It is further desirable to provide a computer display system that reduces the number of required memory page breaks needed to display an image.
The present invention discloses a system and method for processing YCbCr video data stored in a paged memory. One embodiment of the present invention is a graphics controller for accessing a memory storing video data on different pages, where the different pages contain Y, Cb or Cr data, for accessing a FIFO storage device storing Y, Cb, and Cr data as sets of planar video data corresponding to M pixels, where M is an integer value greater than one, and for interfacing with a display device that displays YCbCr planar video data in 4:2:0 format. The graphics controller includes a means for controlling the memory, a means for entering data, retrieved from the memory, into the FIFO storage device and a means for unpacking the data. The means for controlling memory is configured to access and retrieve Y, Cb, and Cr data from the different pages of the memory in response to a request to render an image on the display device. The unpacking means is configured to unpack the data in the FIFO storage device for display on a display device.
The present invention provides for displaying of video data with reduced page breaks. Because such page breaks take longer than one clock cycle to execute, the present invention with its reduced number of required page breaks significantly improves memory bandwidth. In another embodiment of the invention, a method is disclosed for retrieving YCbCr planar video data in 4:2:0 format from paged memory. In this method of the invention, a page of the paged memory containing Y data is accessed. Y data corresponding to M pixels of video data is then retrieved, where M is a value greater than or equal to two. The retrieved Y data is then stored in a shift register. Similar steps are taken to access, retrieve and store Cb and Cr data. Within the shift register, the Y, Cb, and Cr data is stored as sets of planar video data. The Y, Cb, and Cr data is retrieved from the shift register as a series of pixel data that can be used to generate pixels on a video display unit.
In another embodiment, each type of data, Y, Cb, and Cr, are retrieved from different pages of the paged memory. In yet another embodiment, each of the accessing steps includes a page break of the paged memory. Another embodiment includes the shift register configured to store sets of information corresponding to pairs of pixels, wherein the sets of information include two items of Y data for one each of Cb and Cr data.
Another aspect of the invention is a video display system for displaying YCbCr planar video data in 4:2:0 format. The system comprises a paged memory, a memory controller, a shift register, a display request unit, a video processor and a display. The paged memory stores each type of Y, Cb, and Cr data on different pages. The memory controller is configured to access and retrieve Y, Cb, and Cr data from the different pages of the paged memory. The shift register is configured to store Y, Cb, and Cr data as sets of planar video data corresponding to M pixels, where M is a value greater than or equal to two. The display request unit is configured to make display requests of the memory controller. Responsive to the display request unit, the memory controller is configured to: access a page of the paged memory containing Y data; retrieve Y data corresponding to M pixels of video data, where M is a value greater than or equal to two; and, store the retrieved Y data in the shift register. The memory controller performs similar tasks for the Cb and Cr data. In the system of the invention, the video processor is configured to retrieve the Y, Cb, and Cr data corresponding to the M pixels from the shift register as a series of pixel data. Moreover, the Y, Cb, and Cr data is stored in the shift register as sets of planar video data. The display of the system of the invention, is configured to display the M pixels.
In another embodiment of the invention, a memory controller is configured to execute a page break to access each of the Y, Cb and Cr data. In yet another embodiment of the invention, the memory controller includes a data bus on which the Y, Cb and Cr data are retrieved from the paged memory to the memory controller. And, in another embodiment of the invention, the video processor is further configured to unpack the retrieved Y, Cb, and Cr data.
These and other features, aspects and advantages of the present invention will become better understood with regard to the following description, appended claims, and accompanying drawings where:
The present invention is particularly applicable to the YCbCr color space and certain of its formats. To fully understand the present invention, however, it is first important to understand the concepts of RGB and YUV color spaces, as well as various other YCbCr color formats. Accordingly, these will first be described in the context of the present invention.
The RGB color space is widely used in computer graphics implementations. As discussed on above, red, green and blue are the three primary additive colors. In the RGB color space, the colors are represented by a three-dimensional, Cartesian coordinate system. Being one of the earliest-used color spaces, the RGB color space is also the most prevalent in the electronics industry and most display media such as CRT displays or flatscreen displays use the RGB space. However, the RGB color space has a major deficiency when switching between different colors. Specifically, to change the intensity of a particular image or pixel, the intensity of each source, R, G, and B, must be changed.
A different color space, the YUV color space, is extensively used by the Phase Alternation Line (PAL). An important feature of the YUV color space, typically used in these video systems, is its backwards compatibility with black-and-white systems because the luminance Y and chrominance signals U, V are kept separate. Black-and-white systems used only luminance information and color systems use luminance and chrominance information. Thus, a black-and-white system can still display a normal black-and-white image and a color system can use the supplemental information to display a color image by decoding the additional U and V information. In fact, equations exist to convert between an RGB color system and a YUV color system. For example, a color image described in a gamma-corrected RGB color space can be converted to the PAL YUV color space using the following equations for the values Y, U, and V:
Y=0.299R′+0.587G′+0.114B′
U=0.492(B′−Y)
V=0.877(R′−Y)
Here, gamma correction refers to the correction for non-linearities in a display system such as the non-linearities involved in a cathode ray tube (CRT). The prime mark (i.e., the ′) indicates a gamma-corrected value. Using algebraic manipulation of the above equations, gamma-corrected RGB values can be obtained from YUV values:
R′=Y+1.140V
G′=Y−0.394U−0.580V
B′=Y+2.030U
Another color space of particular interest for the present invention is the YCbCr color space. The YCbCr color space was developed as part of Recommendation ITU-R BT.601 (formerly CCIR-601) during the development of a worldwide digital component video standard. YCbCr is a scaled and offset version of the YUV color space. Y is defined to have a nominal range of 16 to 235. Moreover, Cb and Cr are defined to have a range of 16 to 240, with the value 128 equal to zero. Importantly, there are several YCbCr sampling formats, including 4:4:4, 4:2:2, 4:1:1, and 4:2:0. A color image described in a gamma-corrected RGB color space can be converted to the YCbCr color space using the following equations for the values Y, U, and V:
Y=0.257R′+0.504G′+0.098B′+16
Cr=0.439R′−0.368G′−0.071B′+128
Cb=−0.148R′−0.291G′+0.439B′+128
Using algebraic manipulation of the above equations, gamma-corrected RGB values can be obtained from YCbCr values:
R′=1.164(Y−16)+1.596(Cr−128)
G′=1.164(Y−16)−0.813(Cr−128)−0.392(Cb−128)
B′=1.164(Y−16)+2.017(Cb−128)
Note that variations may exist for conversion formulas in order to accommodate certain considerations of interest to a system designer.
According to
As shown in
Because two different scan lines use common Cb and Cr samples, the Y, Cb, and Cr values are typically stored in different areas of a memory. When the memory is implemented as static random access memory (SRAM) or as dynamic random access memory (DRAM), the memory is either a single data rate (SDR) or double data rate (DDR) type of memory and is typically organized as paged memory. This means that the memory is typically divided into 2, 4 or 8-kilobyte pages, where the size of the page is usually determined by the addressing hardware in use. Within any particular page a certain amount of information is available for retrieval without incurring large time delays. In fact, from within any given page, information contained within its various memory locations can be retrieved at every clock cycle. Of the YCbCr formats discussed here, all but the 4:2:0 format store Y, Cb, and Cr information on the same page. Thus, a color display system not using the 4:2:0 format can quickly retrieve successive sample information because the successive information is usually contained on the same page. In these systems, page breaks are not necessary for every pixel to be rendered.
However, when using the 4:2:0 format, the situation is quite different. A typical prior art system implementing the 4:2:0 format retrieves two Y samples of information where both Y samples are contained within the same page of memory. This system then obtains access to a different page to reach another area of memory containing Cb information. From this page, the system would retrieve one item of Cb information. The system then proceeds to obtain access another page to reach the locations containing Cr information. In order to retrieve Y information for another two pixels, another access to a different page is necessary to reach the location containing Y information. Thus, for two pixels of information, a system implementing the 4:2:0 format requires separate accesses to three different pages. This is an undesirable situation which leads to poor performance when using the 4:2:0 format.
A page access typically requires an extra time of 2 latency clock cycles for what is called pre-charging. Pre-charging is a function performed by paged memory systems when closing an open page and opening a closed page. Thus, it can be seen that page accesses introduce a substantial limitation on memory bandwidth. Because the YCbCr 4:2:0 format uses so many separate page accesses, memory throughput can be a significant problem.
This embodiment of the invention, therefore, requires three separate page accesses in order to retrieve information for 8 pixels. Advantageously, this embodiment of the present invention requires only one forth of the page breaks of a prior art system and, therefore, significantly improves memory bandwidth and throughput of a display system implementing the YCbCr 4:2:0 format.
In retrieving information for 8 different pixels in the present invention, it is important to properly associate the various items of Y, Cb, and Cr information. As shown in
The memory 702 is organized into a plurality of pages and can be either SDR or DDR SRAM or DRAM. The 2D requester 704 connects to the memory controller 708. Memory controller 708 provides clock 752, control 754, and address 756 information to the memory 702. Connected to the memory controller are the 2D requester 704, a 3D requester 716, or a PCI/AGP requester 718, and the FIFO storage device 710. The Unpacking/data processing unit 712 is connected to the output of the FIFO storage device and feeds the display interface 714 to which the actual display device is connected.
The 2D requester 704 requests that display device 706 render a two-dimensional image. To achieve this, the 2D requester 704 directs the memory controller 708 to access memory 702 as previously described so that color information can be entered into the FIFO storage device 710. In response to the request, memory 702 provides Y, Cb, and Cr information through data bus 758 to memory controller 708, which then is directed to the FIFO 710 storage device. Information contained in FIFO 710 is then be read serially by unpacking/data processing unit 712. Unpacking/data processing unit 712 first unpacks information for pairs of pixels and then processes this information for transfer to the display interface 714. Display interface 714 includes any necessary specialized hardware or software for rendering an image on display device 706.
In other embodiments of the invention, display system 700 includes a 3D requester 716 that operates in a similar manner as two dimensional request unit 704. Alternatively, display system 700 includes a PCI/AGP request unit 716 that is communicatively coupled through PCI/AGP bus 718 to a CPU/Chipset 720. In this latter configuration, CPU/Chipset 720 provides the display requests to display system 700.
Although the present invention has been described in considerable detail with reference to certain preferred versions thereof, other versions are possible. For example, display systems 500 and 700 and method 600 may be modified to retrieve more or less than 8 items of Y information. In doing so, the corresponding Cb and Cr information would also be retrieved and handled appropriately. Therefore, the spirit and scope of the appended claims should not be limited to the description of the preferred versions contained herein.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5852451 | Cox et al. | Dec 1998 | A |
| 5945997 | Zhao et al. | Aug 1999 | A |
| 5977933 | Wicher et al. | Nov 1999 | A |
| 5999654 | Toujima et al. | Dec 1999 | A |
| 6058459 | Owen et al. | May 2000 | A |
| 6104416 | McGuinness | Aug 2000 | A |
| 6215822 | Bose et al. | Apr 2001 | B1 |
| 6304268 | Iourcha et al. | Oct 2001 | B1 |
| 6608625 | Chin et al. | Aug 2003 | B1 |
| 6614442 | Ouyang et al. | Sep 2003 | B1 |
| 6674479 | Cook et al. | Jan 2004 | B2 |
| 20030151610 | Kuriakin et al. | Aug 2003 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20040113913 A1 | Jun 2004 | US |
