Patent Application
20070041662
The present invention relates generally to image data processing, and particularly to methods and apparatus for efficient scaling of image data.
A mobile device may be, for example, a mobile telephone, personal digital assistant, digital camera, or digital music player. Mobile devices typically rely primarily on a battery for power. To maximize battery life in these devices, it is important to minimize power consumption.
Mobile devices commonly have graphics display systems that includes a host, a camera, a display device, and a graphics controller. The graphics controller drives the display device, and interfaces the host and the camera with one another and with the display device. The graphics controller commonly includes an embedded memory for storing image data. In graphics controllers, it is important to minimize the size of the memory, which reduces cost and also reduces power requirements. In addition, the graphics controller commonly includes logic for performing various image processing operations. One operation that graphics controllers are often called on to perform is image scaling.
Digital images are two dimensional arrays of pixels (a “frame”). The pixels in a display device are generally updated or “refreshed” in a raster scan pattern, that is, from side to side in lines from top to bottom. In addition, frames of pixels are commonly stored in memory, fetched from memory, and written to the display device in raster order.
When an image is scaled horizontally, it is typically scaled a line at a time in raster order. If the image is also to be scaled vertically, several of the scaled lines may be temporarily stored in a memory. Two of the horizontally scaled lines are subsequently used to create one or more new lines for vertically scaling the image. The memory in which the horizontally scaled lines are stored is referred to as a “line buffer.”
The line buffer requires an amount of memory which is not insignificant in a graphics controller used in a mobile device. Typically, at least two horizontally scaled lines of the image need to be stored in the line buffer. For example, for if the image is a 640×480 pixel array, the line buffer must be able to store two lines or 1280 pixels. At 24 bpp, this means that 3.75 kB of memory is needed.
It would be desirable to reduce or eliminate the memory required for the line buffer, especially in battery-powered mobile devices. In addition, power and processing time are required to use the line buffer, and it would be desirable to reduce or eliminate the processing overhead associated with the line buffer. Accordingly, a method and apparatus for efficient scaling of image data in graphics display systems would be desirable.
The invention is directed to methods and apparatus for efficient scaling of image data in graphics display systems.
In a preferred embodiment, a method for scaling an image, includes receiving a first and second groups of pixels. The first group includes at least one first pixel of a first line and at least one first pixel of a second line. The first pixel of the first line and the first pixel of the second line are vertically adjacent. The second group includes at least one second pixel of the first line and at least one second pixel of the second line. The second pixel of the first line and the second pixel of the second line are again vertically adjacent. The method includes a step of creating at least one first vertically scaled pixel from the first group, and at least one second vertically scaled pixel from the second group. At least one horizontally scaled pixel is created from the first and second vertically scaled pixels.
An alternative preferred embodiment is directed to a scaling unit for scaling an image. The scaling unit comprises a fetching unit, a vertical scaler, and a horizontal scaler. The fetching unit is for receiving a first group of pixels. The group includes at least one first pixel of a first line and at least one first pixel of a second line. The first pixel of the first line and the first pixel of the second line are vertically adjacent. The vertical scaler is for creating at least one vertically scaled pixel from the first group. The horizontal scaler is for creating at least one horizontally scaled pixel from at least two vertically scaled pixels.
In another preferred embodiment, a graphics display system includes a memory for storing an image, a graphics controller, and a bus coupling the memory and the graphics controller. The graphics controller includes a fetching unit, a vertical scaler, and a horizontal scaler. The fetching unit is for fetching a first group of pixels from the memory. The group includes at least one first pixel of a first line and at least one first pixel of a second line. The first pixel of the first line and the first pixel of the second line are vertically adjacent. The vertical scaler is for creating at least one vertically scaled pixel from the first group. The horizontal scaler for creating at least one horizontally scaled pixel from at least two vertically scaled pixels; and
Yet another is directed to a machine readable medium embodying a program of instructions executable by a machine. The program of instructions when executed by the machine preferably performs a method for scaling an image, includes receiving a first and second groups of pixels. The first group includes at least one first pixel of a first line and at least one first pixel of a second line. The first pixel of the first line and the first pixel of the second line are vertically adjacent. The second group includes at least one second pixel of the first line and at least one second pixel of the second line. The second pixel of the first line and the second pixel of the second line are again vertically adjacent. The method includes a step of creating at least one first vertically scaled pixel from the first group, and at least one second vertically scaled pixel from the second group. At least one horizontally scaled pixel is created from the first and second vertically scaled pixels.
The objectives, features, and advantages of the invention will be more readily understood upon consideration of the following detailed description of the invention, taken in conjunction with the accompanying drawings.
Preferred embodiments of the invention are directed to methods and apparatus for efficient scaling of image data in graphics display systems. Reference will now be made in detail to the present preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.
The image on a display screen is formed from an array of small discrete elements (“pixels.”) The attributes of each pixel, such as its brightness and color, are represented by a numeric value, which is typically represented in binary form. Thus, an image can also be considered as an array of binary elements of data. For convenience of explanation and in accordance with the use of the term in the art, the term pixel is used herein to refer at times to the display elements of a display device, at times to the binary elements of data that are stored and manipulated within a graphics display system and which define the attributes of such display elements, and at times to both, the appropriate sense of the term being clear from the context.
When an image is enlarged in two dimensions, pixels may be received from an image data source, such as a camera. Typically, the first line received is horizontally scaled, and temporarily stored in a line buffer until another line can be scaled. After the first line is scaled, the next lower line of the image is horizontally scaled. Horizontal scaling enlarges the image horizontally.
As pixels of the second line are scaled, two steps typically occur in parallel. The second line is stored in the line buffer for use in subsequent scaling operations, and the pixels of the second scaled line are combined with pixels from the first scaled line in an interpolation process that creates a new line of pixels. The new line is inserted between the first and second scaled lines thereby vertically enlarging the image.
An image may be vertically enlarged by inserting exactly one line between each line of the original image. In addition, an image may be enlarged in different degrees by inserting more that one new line between the first and second scaled lines. When the image is to be enlarged in greater degrees, the pixels of the first and second scaled lines are repeatedly fetched from the line buffer and combined to produce as many new lines as desired. Where one new line is produced, the first and second scaled lines are generally given equal weight in the interpolation process. Where more than one new line is produced, the first and second scaled lines are generally given different weights in each of the interpolation processes.
Generally, the above-described process for scaling an image streamed from a camera is also used when the image is stored in a memory.
As shown in
In other horizontal interpolation methods, a number of pixels other than one may be created from a group of 4 pixels, e.g., 2 pixels may be created. Further, a pixel may be created from a group having a size other than 4 pixels, e.g., groups having 3 pixels or 5 pixels.
The 2-point vertical interpolation method may be used to produce one or more new rows M/( M+1)exp for insertion between the rows Mexp and M+1exp. If only one row is inserted, the pixels of each row are typically given equal weight in the interpolation. However, if two or more rows are inserted, the pixels of each row are typically given different weights in successive iterations of the interpolation.
While
When the method shown in
In operation, a first step comprises receiving a line M, either from a camera or a memory, and feeding the fetched line through the horizontal scaler 22. A horizontally expanded line Mexp is output from the horizontal scaler 22 and stored in the line buffer 28. A line M+1 is then received and fed through the horizontal scaler 22. As the horizontally expanded line M+1exp is output from the horizontal scaler, it is sent to the vertical scaler 32 along with the horizontally expanded line Mexp, which is read out from the line buffer 28. The vertical scaler 32 thus receives groups of pixels GV which it combines to create new pixels. Once a scaled line, e.g., M/(M+1)exp has been output, the vertical scaler 32 will have output one line of vertically scaled data, which is some interpolated version of scaled lines Mexp and M+1exp. Many different interpolated versions are possible, depending on the weightings given to the pixels in the groups GH by the horizontal scaler 22, and the weightings given to the pixels the groups of pixels GV by the vertical scaler 32.
At the same time that the horizontally scaled line M+1exp is output from horizontal scaler 22 and fed to the vertical scaler 32, the scaled line M+1exp is also written to the line buffer 28. The line buffer 28 is large enough for storing two horizontally expanded lines so this does not disturb the line previously stored line. Accordingly, once the horizontally expanded line M+1exp has been completely output from the horizontal scaler 22, two horizontally expanded lines, Mexp and M+1exp, will be stored in the line buffer 28.
If a frame is being vertically up-scaled, the scaling controller 30 may determine that additional lines are required to be interpolated between scaled lines Mexp and M+1exp. If so, the scaled lines Mexp and M+1exp are read out of the line buffer 28 (as many times as is necessary) and sent to the vertical scaler 32. The multiplier coefficients used for interpolating the scaled lines Mexp and M+1exp will be different for each additional new line, so each new lines will be different from the others. Additional new lines, each with different multiplier coefficients, may be produced from the horizontally scaled lines stored in the buffer 28 until the scaling controller 30 determines that a sufficient number of lines have been created.
When the scaling controller 30 determines that a sufficient number of lines have been interpolated, the above described steps are repeated with a new line of the original image. The above described sequence of steps are repeated until the entire image has been processed. Each time the sequence is repeated, the first step comprises receiving a new line the memory. When this first step is performed, a decision must be as to which new line of the original image to use. If the frame is being vertically up-scaled, the new line will be line M+2. However, if the frame is being vertically down-scaled, this new line may be a line Y, where Y≧M+2. In this case, a line received from the camera may be discarded, or if the image is being fetched from memory, one or more lines may be skipped.
Preferred embodiments of the present invention eliminate the need for the line buffer 28, and eliminate the processing overhead associated with the line buffer. In addition, preferred embodiments are directed to enlarging an image stored in a memory. While the exemplary method scaling method and apparatus described with reference to
Referring to
In a preferred embodiment, the image or frame to be scaled is stored in a memory 46. Pixels are fetched from the memory 46 by the fetching module 26 over a bus 49, and presented to the vertical scaler 32. A scaling control module 44 directs the fetching module 26 to fetch particular lines from the memory 46. The fetching module 26 fetches groups of pixels GV. The bus 49 is preferably a 48-bit parallel bus, which is wide enough for two 22-bit pixels.
In a preferred embodiment, the vertical scaler 32 employs 2-point interpolation and therefore must be fed two lines at once. The image stored in the memory is logically divided into two fields. A first field includes all of the even numbered lines and a second field includes all of the odd numbered lines. For example, if the image is 8 lines, the first field is lines 2, 4, 6, and 8, and the second field is lines 1, 3, 5, and 7. Preferably, the fields are fetched and presented to the vertical scaler 32 in parallel. For instance, lines 1 and 2 are fetched in parallel. In addition, pixels from the same column the image are preferably fetched at the same time, or at substantially the same time, e.g., P1,1, and P2,1 of the group GV1 are fetched simultaneously.
While pixels in like columns of the first and second fields are preferably placed on the bus 49 in the same memory cycle, in an alternative embodiment, the two pixels in the group GV may be fetched sequentially in two memory cycles using a narrower bus, e.g., the bus 26, provided that the two pixels are presented to the vertical scaler 32 at the substantially the same time. For example, in an alternative embodiment, a register 46 may be provided for temporarily storing a first pixel while the other pixel in the group GV is fetched. An embodiment employing the 24-bit bus 26 would require different timing requirements from embodiments employing the 48-bit bus 49. Other embodiments may employ busses of any width, provided that two pixels of a group GV are presented in substantially simultaneously to the vertical scaler 32.
The scaling control module 44 directs the fetching module 26 to fetch pixels from the rows M and M+1. The fetched rows are fed to the vertical scaler 32 which produces a new interpolated row M/(M+1). This row M/(M+1) is fed to the horizontal scaler 22. The horizontal scaler 22 produces a horizontally expanded row M/(M+1)exp. While the horizontal scaler 22 preferably employs 4-point interpolation, this is not essential.
In operation, assuming the image is to be up-scaled, two lines are fetched in parallel from the memory 46 by the fetching unit 42 and fed to the vertical scaler 32. Preferably, the fetched lines are preferably from different fields and are adjacent to one another in the image. Two vertically adjacent pixels, e.g., a group GV, are fed to the vertical scaler 32, which outputs a new line. This new line is fed to the horizontal scaler 22 a pixel at a time. When the horizontal scaler 22 receives enough pixels to perform a horizontal interpolation, e.g., a group GH, it outputs a new pixel.
When up-scaling an image vertically, it may be desired to interpolate a pair of lines more than once in order to produce more than one new line for insertion between the two original lines. If the scaling control module 44 determines that an additional lines are required, it directs the fetching module 42 to fetch from the memory 46 the two original lines previously fetched. The two original lines are presented to the vertical scaler 32 and the vertical scaler creates new pixels from groups GV. When additional lines are produced in this manner, the vertical scaler 32 preferably gives the pixels in each of the groups GV a different weight for each successive line. Further, when additional lines are produced, the vertical scaler 32 feeds each new line to the horizontal scaler 22. For each new line fed to the horizontal scaler 22, it produces a new horizontally scaled line for insertion between the two original lines.
When down-scaling an image vertically, each original line of the image may be fetched for creating a new line which may be used to replace the two original lines. When an image is scaled by different degrees, it may not be necessary to fetch all of the original line of the image in order to create the new lines which make up the down-scaled image.
Herein, the phrase “vertically adjacent” is used to refer to pixels from the same group GV. When up-scaling an image vertically, the pair of lines from the original image are typically successive lines of the image, e.g. line M, and line M+1. When down-scaling an image vertically, the pair of lines from the original image may not be successive lines of the image. For example, the line M, and the line M+2 may be vertical interpolated. Accordingly, the phrase “vertically adjacent” is not limited to pixels in the same column in successive lines of the image, but includes pixels any lines which are not directly adjacent vertically. In addition, while the pixels in a group GV are preferably from the same column of the array, this is not essential.
Preferred embodiments are directed to a graphics controller employing the scaling module 40. Turning now to
The host 54 is typically a microprocessor, but may be a digital signal processor, a computer, or any other type of controlling device adapted for controlling digital circuits. The host 54 communicates with the graphics controller 52 over a bus 60 via a host interface 62 in the graphics controller.
The graphics controller 52 includes a display device interface 64 for interfacing between the graphics controller and the display device 56 over display device bus 66. LCDs are typically used as display devices in portable digital appliance, such as mobile telephones, but any device(s) capable of rendering pixel data in visually perceivable form may be employed. The display device 56 has a display area 56a. In an alternative embodiment, the display device 56 is a printer. The graphics controller 52 also includes a camera interface 68 (“CAM I/F”) for receiving pixel data output on data lines of a bus 70 from the camera 58.
In addition to the scaling operations performed by the scaling unit 40, a number of other image processing operations may be performed on data provided by an image data source, such as the host or the camera. Such image processing operations may be performed by units included in an image processing block indicated generally as 72. The image processing block 72 may include, for example, a CODEC for compressing and decompressing image data, and a color converter for converting the color format of the data. Image processing operations, such as cropping, compression encoding, and color converting, may be performed as the data are received, “on the fly,” or they may be performed on data that have been stored in a memory.
In a preferred embodiment, the graphics controller 52 includes a memory 46 for storing image data. Access to the memory is controlled by a memory controller 74. In other embodiments, however, the memory 46 may be remote from the graphics controller. The memory 46 is preferably an SRAM, however, any type of memory may be employed.
Image data stored in the memory 46 may fetched and transmitted through a display pipe 76, which is preferably a FIFO buffer. The output of the display pipe is transmitted through the display device interface 64 to the display device 56. If it is desired to enlarge the image stored in the memory 46, the image data may be fed to the scaling unit 40 for scaling. The scaled output may then be provided to the display pipe for subsequent display or to another module for further processing or other disposition. In one alternative embodiment, for example, the scaled output of the scaling unit 40 is stored in a nonvolatile memory, such as a flash memory card.
While the image data to be scaled by the scaling unit 40 is preferably stored in a memory, such as the memory 46, this is not essential to all embodiments of the invention. For example, if a pair of lines of an original image need to be vertically interpolated only once in order to produce a single new line for insertion between the two original lines, it is not necessary to have the image stored in memory. As another example, if the image source (other than a memory) has the ability to repeatedly feed the same line to the scaling unit 40, it is not necessary to have the image stored in a memory.
An example illustrates operation of the system 50. An image is presented to the graphics controller 52 by the camera source. Alternatively, the image may be presented by the host or another image data source. The image is stored in the memory 46. The image may be down-scaled before being stored, but this is optional. The image is subsequently fetched from the memory 46 and enlarged vertically and horizontally by the scaling unit 40 as previously described. The up-scaled image is fed to the display pipe 76 for presentation to the display device 56.
In addition, invention is directed to a machine readable medium, such as magnetic or optical disks, hard disk drives, memory chips of any type, and other similar memory devices. Preferably, the medium embodies a program of instructions that may be executed by a machine, such as a computer system. The program of instructions may be software, firmware, hardware code, or other similar program. The program of instructions, when executed by the machine, performs a method of: (a) receiving two pixels of a group GV, (b) creating a new pixel from the group of pixels GV for vertical insertion into the image. After a sufficient number of new vertical pixels are created, e.g., a four pixels forming a group of GH, (c) another new pixel for horizontal insertion into the group GH is created.
The terms and expressions that have been employed in the foregoing specification are used as terms of description and not of limitation, and are not intended to exclude equivalents of the features shown and described or portions of them. The scope of the invention is defined and limited only by the claims that follow.
