Patent Grant
6897874
1. Field of the Invention
This invention relates to computer systems, and more particularly, to methods and apparatus for providing window overlays on an output display.
2. History of the Prior Art
For some time there has been a need to provide a window on a computer output display in which information from a source other than a computer program may be displayed. For example, it is desirable to display video information provided by a source of television signals or from a digital video disk (DVD) in a window on a computer display.
To accomplish this, the prior art initially combined video signals which had been transformed into RGB format with computer graphic information in the frame buffer. Such an approach provides a window in which video images may be displayed; however, any manipulation of the video data can only be accomplished by the time consuming process of accessing and then returning the manipulated data to the frame buffer.
As the amount of video and graphics data being displayed have both increased, the need to access video data in the frame buffer has greatly slowed the display processes. For this reason, the most recent attempts to display video have concentrated on providing what is referred to as a hardware overlay that displays video information separately from the computer graphics information. The two channels of information are ultimately combined beyond the frame buffer at the end of the graphics pipeline just prior to display.
This method of displaying video information in a window on a computer graphics display performs satisfactorily so long as the manipulation of the video information is limited to color space conversion from a video format to a red/green/blue (RGB) computer graphics format and the video image is either enlarged or maintained at the size provided by the video source. However, if the video image is to be reduced from the size of the source, the data transfer rate during window scanout becomes too great for prior art overlay engines to handle.
It is desirable to provide an improved method and circuitry for furnishing video information in a window on a computer output display.
It is, therefore, an object of the present invention to provide improved circuitry and a method for furnishing video information in a window on a computer output display.
These and other objects of the present invention are realized by a circuit for providing an overlay in a window on a computer output display including scaling circuitry, storage circuitry for receiving a plurality of lines of source data, input circuitry for loading the storage circuitry in a first prefill mode and in a second low water mark mode, and circuitry for selecting a mode responsive to the characteristics of the demand for data being placed on the storage circuitry.
These and other objects and features of the invention will be better understood by reference to the detailed description which follows taken together with the drawings in which like elements are referred to by like designations throughout the several views.
In known overlay engines of the prior art, the scaling engine is capable of enlarging the video image or simply passing it through without change to the color space conversion circuitry. When the video data is passed through without change by the scaling engine, the FIFO 11 must furnish data at the same rate as the pixel clock used for clocking graphic data to the output display since each pixel of the video data appearing in a window on the display must be clocked to the screen at the same rate as the graphics data.
The overlay engine 10 is able to accomplish this by a process in which it commences to fill the FIFO 11 with video data beginning at the start of the horizontal blanking period for a scan line and ending when the FIFO has filled to some preselected level. Typically, the FIFO 11 is small compared to a scan line of video data so it fills to the selected level before the beginning of the video window on the scan line. When the video window begins, the video data is drawn from the FIFO and provided to the scaling engine. With no enlargement of the video data, the data is drawn out at the pixel clock rate for graphics data. Requests are sent by the FIFO for more data when data in the FIFO reaches a “low water mark.” The low water mark is set at a level at which the FIFO will have received sufficient data to prevent the FIFO from emptying before new data is returned. If the amount of source line data left is less than the amount that would normally be requested to cover the latency, then the smaller amount of data will be requested.
When the video image is enlarged, a lower data rate is required from the FIFO during the scanout of the window. This is true because the same number of pixels are expanded into a larger number of pixels according to some algorithmic process such as linear interpolation. As is known, typical algorithms for enlarging a scan line horizontally utilize values of adjacent pixels to generate additional pixels between those provided by the source. Vertical enlargement algorithms utilize the pixels of a plurality of usually sequential scan lines to generate scan lines in addition to those furnished by the source. To this end, scan line storage is usually provided in the scaling engine so that vertical filtering may be accomplished to generate new scan lines in enlarging video data. Because a lower pixel rate is required during the scanout of the window when enlarging the video image, the FIFO 11 is able to furnish video data to the scaling engine 12 at a rate less than the pixel clock rate of the graphics data.
A prior art overlay engine 10 is not capable of providing video data at a rate sufficient to allow a significant reduction in size of the video image which is to be displayed. This is true because the pixel rate required by a scaling algorithm during scanout of the window from the source of pixels to reduce the size of the video image is much greater than is required for a direct display or an enlargement of the video image. Thus, to produce a window image which is reduced in size, the rate of pixel data furnished by the source of video data, during scanout of the window, is much greater than required for either an unenlarged or an enlarged image. For example, to reduce the width of the video image by half, all of the pixels for a full size image are required to be manipulated by the scaling engine to generate half as many pixels as an output. However, the output furnished by the scaling engine must still be at the pixel clock rate required for display on the computer output screen. This requires that the FIFO (and the source) furnish the same amount of video data in half the time. The need for a much larger amount of video data also results from the reduction in image size in the vertical direction.
The data rate demanded from the FIFOs during the scanout of the window is simply too high to be provided by the schemes of the prior art. Moreover, the concentration of this data transfer over the graphics engine memory bus severely restricts the flow of other (graphics) data. Consequently, prior art circuitry has been unable to accomplish any significant reduction in video image size.
Even though prior art circuitry has been unable to reduce the size of video data presented in a window on a computer output display, it is desirable to be able to do so.
As has been described, prior art circuits for placing a video display on the output display of a computer have allowed the video information to be displayed at the same size as received or enlarged. Prior art circuitry has not provided for reducing the video data when displayed on a computer output display. The reason reducing has not been practiced is that it requires a significantly higher data rate from the FIFOs, during the scanout of the window, to generate a reduced image than to generate an image in the size received from the source. For example, if an image is to be placed in an area one-quarter the size of that for which it is intended by reducing its width and height each to one-half of the original, then the pixels of each scan line of pixel data must be reduced to one-half and the number of scan lines must be reduced to one-half. To reduce the number of pixels in a scan line to one-half typically requires interpolating the values of two adjacent pixels to produce a single pixel. This requires that the scaling engine of the pipeline that accomplishes the reduction manipulate twice as many pixels for a given length of a scan line in which the video window is to appear than would be handled for a one-to-one video window generation process. Moreover, to reduce the number of scan lines to one-half those of a full size video display, two or more scan lines are interpolated to combine their colors in a well known manner to provide one resulting line which replaces each two original scan lines. Thus, twice as many lines of pixel data must be manipulated by the scaling engine than would be handled for a one-to-one video window generation process. In all, four times as much pixel data is required within the time available for drawing the video window as would be needed for a one-to-one video window generation process.
The most recent versions of Microsoft Windows require that the video data placed in off-screen buffer areas and merged with graphics pixel data be capable of reduction by a factor of eight in each dimension. The rate of video data required during window scanout to accomplish an eight times reduction in the video image is much larger than the memory bus is capable of furnishing in real time. For this reason, the prior art has found it impossible to meet the requirements of the Windows system software.
The present invention overcomes these limitations by significantly modifying the manner in which video data is handled.
The three FIFOs 21-23 are connected to furnish video data to a video pipeline of the overlay engine 20. The pipeline receives the video data in appropriate order and processes that data through scaling and color space conversion stages. Video data is sent to the scaling stage of the pipeline where the image is enlarged, reduced, or kept at the same size in response to instructions furnished by a controlling process. The image in what is typically a television or DVD format which results from the scaling process is sent to the color space conversion stage which transforms the data into the RGB format used for display on a computer output display utilizing a color space conversion algorithm well known to those skilled in the prior art.
In accordance with the present invention and in contrast to prior art overlay engines, the engine 20 controls the transfer of data to each FIFO 21-23 to commence, in most cases, sufficiently early that each of the three FIFOs is prefilled before any of the data needed for processing a scan line of video data is required. For example, in one embodiment the overlay engine 20 begins to fill the individual FIFOs at the start of the horizontal blanking period which commences one full scan line before the first scan line in which the video window in displayed in the manner shown in FIG. 3. Assuming that the video window is sufficiently narrow compared with the width of the screen and assuming a system in which two scan lines of video data may be accessed at once, this allows the FIFOs 21-23 to be filled before the data is required by the scaling engine. Filling the FIFOs which are each sufficient to hold a complete scan line of unscaled source image data before the scan line within the video window is begun eliminates the problem of having insufficient bandwidth to keep up with the scanning process while downscaling the video image.
In a scaling engine which uses three source image lines to generate a single output scan line and in which the video image size is reduced vertically not more than two times, it is possible to prefill the FIFOs with the necessary video data to maintain the correct pixel clock rate. This is accomplished by starting the prefill process as soon as the video window ends and continuing until the next scan line reaches the beginning of the video window.
In cases where more video data is required than can be provided during the period allotted for prefilling, the overlay engine 20 switches to a low water mark monitoring (demand) mode in which it continues to fill the FIFOs whenever the video data remaining in any FIFO drops below a programmable level (a low water mark). The low water mark mode commences in one embodiment when all of the data requested for a FIFO have not been furnished and a video window scan line commences. The engine 20 switches to this operating state so that it requests and receives more video data as the data is drained from a FIFO down to the low water mark. When the end of the video window scan line is reached, the prefill condition begins again so that as much data as possible is placed in the FIFOs before the next scan line of the video image begins.
Switching between modes for filling the FIFOs is especially useful in a case in which a video window is as wide as the entire display (see FIG. 4). As may be seen, the time allotted from the end of the window until the window recommences is quite short compared to that allotted for the window shown in
In one embodiment, the scaling engine is adapted to switch from three scan line vertical filtering to two line vertical filtering in situations in which there is a vertical reduction in the video image of more than two or when the source image is too wide to store completely in the FIFOs. Switching to two line vertical filtering when vertical down scaling is greater than two is chosen because the embodiment is only capable of reading into two FIFOs at once while vertical down scaling of greater than two, while doing three tap vertical filtering, requires more than two new source image lines for each output scan line. If the source image is too wide for a line of it to be stored in a FIFO, then a previously read line cannot be re-used to generate the current output line (It is not stored in a FIFO). This means that two input lines must be read fresh from memory to generate every output line resulting in only two tap vertical filtering being carried out. Of course, if more FIFOs and circuitry for loading such FIFOs in parallel are available in a particular embodiment, then some greater plurality (than three taps) of vertical filtering may be utilized initially and reduced to a lesser plurality of taps for vertical filtering, equal to the number of lines which can be read in parallel, when there is a vertical reduction in the video image of some predetermined amount, or when the source image is too wide to store completely in the FIFOs, or some similar situation occurs.
A window engine 40 cooperates with the data transfer manager 35 in the operation of the overlay engine 30. The window engine keeps track of screen position and window placement to assure that the data is correctly manipulated and placed in the video window.
Ultimately, the pixel information in RGB format which results from the pipeline 38 is transferred to a multiplexor (see
In order to accomplish the transfer of video data in accordance with the present invention, the data transfer manager 35 utilizes a FIFO control circuit A, a FIFO control circuit B, and a register array 41 including registers R0 and R1.
The engine 30 functions in the following manner. Software controlling the display of a video window on the output display writes the parameters controlling the video window and the video data to be displayed to a buffer such as the buffer0 illustrated in FIG. 2. These parameters include among other things the window position and size and the scaling factor to be used. The parameters controlling where the data is to appear are also written by the software to one of the two registers R0 or R1 in register file 41. The window engine 40 keeps track of the lines being scanned to the screen and just before the end of the vertical blanking period signals the data transfer manager that the screen is about to start. The data transfer manager recognizes the screen start signal and reads the parameters in the register file 41 so that it knows which buffer in memory contains video data to be placed in the window in order to select a buffer. The data transfer manager selects the buffer designated by the parameters in the register file 41 (register R0 or R1).
When it has selected a buffer, the data transfer manager signals the window engine 40 that the particular buffer has been selected. The window engine 40 directs the pipeline 38 to obtain the parameters from the appropriate register of the register file 41 and itself reads the parameters to determine where the window is to appear on the display.
The window engine counts the lines being sent to the display until the scan line before the first scan line including the video window is reached. When the window engine has counted to the scan line which is one line before the video window is to begin, the window engine 40 resets the pipeline at the beginning of the horizontal blank for this line. This resets the vertical and horizontal scaling algorithms and directs the pipeline to fetch the first scan lines from the FIFOs. The reset also causes the pipeline to unstall.
The pipeline, which knows how many new lines it needs, generates a line request to the data transfer manager which keeps track of the particular scan lines needed for executing the vertical scaling algorithm. Once there is enough data in the FIFOs to fill the pipeline, the data transfer manager grants the request and instructs the pipeline which lines to access and in which FIFO a particular source line exists. The pipeline then passes the data down through its stages until the data arrives at the window engine. The window engine then stalls the pipeline.
The data transfer manager also keeps track of the scan lines in the FIFOs. The data transfer manager uses the FIFO control circuits A and B to fetch video data from the selected memory buffer and to place the data in the one or two of the FIFOs chosen to hold the new scan lines of data while maintaining data already in the other FIFOs to be reused in the vertical scaling calculation. The data transfer manager selects a particular FIFO control circuit to access the data and tells the FIFO control circuit which FIFO to place that data in. The data transfer manager is able to select particular lines to meet the needs of the scaling algorithm and the degree of enlargement or reduction.
When the data transfer manager has requested that the FIFO control circuits access particular lines of data from a particular buffer and place those lines in particular FIFOs, it signals the pipeline and indicates which FIFOs the scan lines reside in. The FIFO controllers are programmed to request bursts of data of a preset amount in order not to overload the memory bus. A bus arbiter causes the requests to be granted and the data to be furnished in the corresponding bursts to first one then the other of the FIFOs designated by the FIFO controllers and at a reasonable rate on the memory bus so as not to swamp it. Each of the FIFO controllers knows when it has furnished the correct amount of data to a FIFO for one scan line of the video window so that it will stop when this point is reached.
When the pipeline receives the signal that its request for data has been granted along with the order of the data in the FIFO, it starts draining the FIFOs into the pipeline 38. The window engine 40 will unstall the pipeline 38 until the data reaches the input to the window engine. The window engine will then stall the pipeline and wait for the start of the video window.
At the point at which the video window starts, the windows engine signals the FIFO controllers that the window is running. This signal causes the FIFO controllers to switch to the low water mark mode of operation. In this mode, any FIFO controller presently receiving data in response to a request to the bus arbiter finishes that request and then stops requesting until the low water mark level is reached. If data reaches that level during a scan line and more data is needed to meet the initial request for data, the FIFO controller signals the bus arbiter to furnish data using a low water mark request. The bus arbiter responds to these requests immediately so that the FIFO receives a next burst as soon as possible and the need for data can be met when it is required for a currently displaying window. Again, when the FIFO has received the entire amount of data requested initially, it stops filling.
In the meantime, the windows engine continues to check the position on the screen. When the end of a scan line within the video window is reached (see FIG. 3), the window engine resets the pipeline, causing the pipeline to request a new scan line from the data transfer manager. This in turn, causes the data transfer manager to initiate another transfer of data from the selected buffer to the FIFOs in prefill mode. This process continues until, ultimately, the buffer selected will have been scanned out and the window completed on the display. When this occurs, the next frame of the display may be filled from another buffer (buffer1) in off-screen memory thereby providing completely tear-free double buffering for the video window process.
Although the present invention has been described in terms of a preferred embodiment, it will be appreciated that various modifications and alterations might be made by those skilled in the art without departing from the spirit and scope of the invention. The invention should therefore be measured in terms of the claims which follow.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5469223 | Kimura | Nov 1995 | A |
| 5671445 | Gluyas et al. | Sep 1997 | A |
| 5694571 | Fuller | Dec 1997 | A |
| 5764201 | Ranganathan | Jun 1998 | A |
| 5767863 | Kimura | Jun 1998 | A |
| 5864512 | Buckelew et al. | Jan 1999 | A |
| 5877741 | Chee et al. | Mar 1999 | A |
| 5894312 | Ishiwata et al. | Apr 1999 | A |
| 5914711 | Mangerson et al. | Jun 1999 | A |
| 5931922 | Hough | Aug 1999 | A |
| 5953020 | Wang et al. | Sep 1999 | A |
| 5978841 | Berger | Nov 1999 | A |
| 6166748 | Van Hook et al. | Dec 2000 | A |
| 6233629 | Castellano | May 2001 | B1 |
| 6348925 | Potu | Feb 2002 | B1 |
| 6348929 | Acharya et al. | Feb 2002 | B1 |
| 6415377 | Van Der Wolf et al. | Jul 2002 | B1 |
