Patent Application
20050231643
1. Field of the Invention
The present invention relates in general to digital transformation of a video image, and more specifically a method, system and device for generating non-linear video transformations in real time.
2. Discussion of the Background
In the field of broadcasting and video production it is often desirable to add “special effects” to a production. These special effects can be as simple as a dissolve between two video images, or a complex transformation of one or more images. A certain class of effects is known in the field as Digital Video Effects (DVEs). This is a special class of effects in which elements of video are moved or transformed on the screen. Digital Video Effects (DVE) generators are devices that are used to create real-time, video transformations.
There are several classes of DVE. The simplest are two-dimensional (2D) and three-dimensional (3D) transformations. In these effects, an image is transformed linearly through two-dimensional manipulation (size and position in the horizontal and vertical axes), or three-dimensional manipulation (size, position, and rotation, in three axes with perspective) respectively. These effects are often referred to as planar effects, and are typically limited to manipulation of an image as a whole; thus multiple, independent transformations of an image requires additional DVE generators.
A more advanced class of DVE involves the use of non-linear mapping of images. It can involve such effects as curved surfaces, multiple independent transformations within one DVE generator, or other arbitrarily shaped transforms. These non-linear effects are generally known in the field as Warp effects.
The generation of planar effects involves the implementation of simple circuits to produce a linear rotation, translation, and scaling matrix. It is relatively simple using technology of today. U.S. Pat. No. 5,448,301 discloses a general design for planar effect.
Current technologies for the generation of non-linear, or Warp effects, include specialized circuit elements and/or lookup-table memory elements for each type of effect to produce various mathematical transformation functions on the video image. Examples of these approaches are presented in U.S. Pat. No. 5,233,332 to Watanabe, et al. and U.S. Pat. No. 4,860,217 to Sasaki, et al.. By sequencing and combining these specialized elements, an implementation can generate a limited number of special effects. U.S. Pat. No. 6,069,668 adds more warp look-up tables to achieve various warp effects, such as particles and bursts. U.S. Pat. No. 6,069,668 also gives a brief survey in warp-based DVE design and categorizes into warp-driven systems those that use 2D warp means for the address generator shown in
Other techniques to generate warp effects use a software-based approach. In this approach, software algorithms are used to create arbitrarily complex effects. These algorithms compute the transformations and manipulate the video. This computation is referred to as “rendering” and often is very time consuming because it uses thousands of polygons to construct arbitrarily shaped objects. It is not suitable for live broadcast applications, as these software algorithms are too slow to manipulate a video frame within that frame's allotted time interval, which is typically 16 to 20 ms. These techniques are generally used in post-production environments, where video images are assembled offline and recorded to videotape or disk for later real-time playback.
Although most of commercial DVE devices use warp-driven design due to their cost effectiveness, as mentioned in U.S. Pat. No. 6,069,668, these designs, however, still suffer from:
(i) Lack of flexibility: Each warp effect or a subset of warp effects may need particular circuits or lookup tables for a given implementation. New effects may need new hardware modules to be added for support.
(ii) Requirement of Huge external memory: Warp lookup tables are effective solutions for real-time implementation, but require a large size of physical storage (e.g., 2D implementations require lookup tables sized based on the horizontal resolution times the vertical resolution).
Therefore, there is a need for a method, system, and device that addresses the above and other problems with conventional systems and methods. Advantageously, the exemplary embodiments provide a generic way to solve the above and other problems by employing a software-based solution, wherein low cost, high-speed microprocessors, such as DSP chips, and well-developed algorithms for warp effects can be employed, allowing software to implement warp effects in real time or near real time. Accordingly, in exemplary aspects of the present invention, a method, system, and device are provided for video transformation, including generating arbitrary, non-linear video effects; generating the video effects in real-time or near real-time; providing software algorithms used to generate video transformations corresponding to the video effects; employing a microprocessor to generate an address map corresponding to the video effects; employing an interpolator to read the address map in real-time or near real-time; and manipulating via the interpolator video pixel data in real-time or near real-time to generate a desired output video image corresponding to one of the video effects.
Still other aspects, features, and advantages of the present invention are readily apparent from the following detailed description, simply by illustrating a number of exemplary embodiments and implementations, including the best mode contemplated for carrying out the present invention. The present invention also is capable of other and different embodiments, and its several details can be modified in various respects, all without departing from the spirit and scope of the present invention. Accordingly, the drawings and descriptions are to be regarded as illustrative in nature, and not as restrictive.
The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:
A method, system, and device for real-time non-linear video transformations are described. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It is apparent to one skilled in the art, however, that the present invention can be practiced without these specific details or with equivalent arrangements. In some instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.
The present invention includes recognition that current real-time Digital Video Effects (DVE) generators are limited to simple two-dimensional or three-dimensional transformations or a small number of non-linear “Warp” transformations. This is due to limitations in the hardware-centric implementation of the transform algorithms. More flexible, software-based methods are far too slow to be usable in a real-time manner. The exemplary embodiments of the invention employ a hybrid approach of software-based transformations with hardware pipelines for the video manipulation. The result allows flexible software-based transformations to be generated in real time.
Thus, the present invention describes a novel and generic method, system, and device to solve the problems posed in the prior art, and advances the state of the art by applying a software-based transformation generator to a real-time DVE generator.
In the DVE generator, transformations can be generated in one of two manners. In the first manner, for each pixel in the source video image, the transformation generator generates a co-ordinate of the desired position in the output video image. This is referred to as forward mapping and represented as:
{overscore (P)}tgr=M{overscore (E)}src (1)
where {overscore (P)}src is a source pixel position vector, {overscore (P)}tgr is a target or display pixel position vector, and M is a transformation matrix which is either spatial variant or invariant.
In the second manner, for each pixel in the output video image, the transformation generator generates a co-ordinate of the desired pixel in the source image. This is referred to as inverse mapping or reverse mapping and represented as:
{overscore (P)}src=M−1{overscore (P)}tgr (2)
where M−1 is inverse matrix of M.
Generally, M is factorized into several cascaded local transformation such that:
M=M1M2M3 . . . Mn or M−1=Mn−1 . . . M3−1M2−1M1−1 (3)
The exemplary embodiments employ a microprocessor to implement a part of cascaded transformations shown in formula 3. Advantageously, such devices can be cascaded or combined to form more complicated warp effects.
Referring to
The transformation co-ordinates can be either reverse mapping or forward mapping. In a forward-mapping system, illustrated in
The Warp Memory Controller (104) arbitrates memory accesses to the Warp Memory Buffer (107) between the Microprocessor (101) and the Warp Controller (110). Co-ordinate data received via the Microprocessor Address Bus (102) and the Microprocessor Data Bus (103) are written to the Warp Memory Buffer (107) via the Memory Address Bus (105) and the Memory Data Bus (106), as shown in
The Warp Controller (110) provides a sequence of co-ordinates. In a forward mapping system, these co-ordinates are generated sequentially and represent the pixel position of the source video image. In a reverse-mapping system, these co-ordinates are generated sequentially and represent the pixel position of the output video image. In either system, these co-ordinates are passed to the Warp Memory Controller (104) via the Warp Address Bus (108). The implementation of this bus is configured to ensure sufficient bandwidth such that the co-ordinates can be transferred at an average rate which is at least as fast as the video pixel clock rate.
The Warp Memory Controller (104) reads from the Warp Memory Buffer (107) the co-ordinate data corresponding to the Warp Address requested on the Warp Address Bus (108). This address is passed to the Warp Memory Buffer (107) via the Memory Address Bus (105). Data is returned to the Warp Memory Controller (104) via the Memory Data Bus (106). This data is then passed onto the video interpolator (111), as shown in
Different from 2D warp lookup tables previously employed, Warp Memory Buffer (107) need not be a conventional lookup table. Rather, Warp Memory Buffer (107) can function as an elastic buffer, which converts non-constant generation rate of coordinate data from Microprocessor (101) into constant generation rate of coordinate data. Therefore, when a well-developed warp algorithm generates coordinate data at rate that doesn't fluctuate too much around the average pixel rate, the Warp Memory Buffer (107) does not require a large physical memory. In an exemplary embodiment, the Warp Memory Buffer need only store several lines or several tens of lines of coordinate data, because each of the warp algorithms need only finish the calculation of a horizontal line of coordinate data roughly in a horizontal line period. In addition, since Warp Memory Buffer (107) need not employ a large sized memory, advantageously, Warp Memory Buffer (107) can be realized by using internal memory in microprocessor or FPGA devices, and the like. In an exemplary embodiment, the microprocessor (101) reads the co-ordinates via Microprocessor Data Bus (103) and can use the co-ordinates to generate co-ordinate data that is later sent to Warp Memory Buffer (107), as shown in
In an exemplary embodiment, the Warp Controller (110) also can function as an interface to any other suitable address generators. In such an embodiment, the Warp Controller need not provide a sequence of co-ordinates, but rather passes input coordinates generated by such other address generators.
The Video Interpolator (111 ) is responsible for the actual manipulation of the video pixels. The methods employed by the Video Interpolator (111 ) can vary by implementation and can include over-sampling, filtering, and video memory access. In an exemplary embodiment, such methods are made independent of the other processes of the exemplary embodiments. By contrast, a software algorithm typically is used for both transform co-ordinate generation and video pixel manipulation. However, by limiting the scope of the software algorithms to generation of transform co-ordinates, advantageously, the exemplary software algorithms are within the capabilities of a variety of microprocessor devices.
Certain microprocessor devices, known as Digital Signal Processors (DSP), are very well suited to the generation of transformation co-ordinates, and the transfer of these co-ordinates at suitable speed to meet or exceed the average pixel rate of the video image, and which advantageously allows for a small sized Warp Memory Buffer (107).
If the complexity of effects increases to a point where the algorithms cannot generate transformation co-ordinates at a rate that is at least as fast as the average pixel rate of the video system, multiple microprocessors can be employed. Similarly, if faster video pixel rates are present, multiple processors can allow a faster average rate of transformation co-ordinate generation.
An example of this, using two microprocessors, is shown in
Depending upon the bandwidth of the Microprocessor, Warp Memory Buffer and the various Address Buses and Data Buses, multiple sets of transformation co-ordinates can be generated within one frame or field. This can vary by implementation, but this method is scalable.
Video transformation effects can be greatly enhanced by the addition of lighting and shadows. These lighting and shadow elements can add greater depth and realism to the video effect. In a manner similar to the generation of the transformation co-ordinates, lighting and shadow components can be generated by the microprocessor. As shown in
As the Warp controller (310) provides co-ordinate data for the transform. The Warp Memory Controller (304) reads both transform data and lighting and shadow data from the Warp Memory Buffer (307) for the co-ordinates provided by the Warp Controller (310). The lighting data is passed to the Video Interpolator (311) via the Lighting Data Bus (312). The method for the application of the lighting and shadows to the video by the Video Interpolator can vary by implementation.
Further enhancement to the video transformation effect can be made by the addition of transparency data on a per-pixel basis. This is called an Alpha Map. The Alpha Map can be used to create semi-transparent effects for creative purposes or anti-aliasing through edge softening. Furthermore, if multiple transformation maps are being generated, these independent transforms can be layered upon each other with regions of transparency, semi-transparency, or opacity determined by the Alpha Map.
Referring to
As the Warp controller (410) provides co-ordinate data for the transform. The Warp Memory Controller (404) reads the transform data, lighting and shadow data, and the Transparency data from the Warp Memory Buffer (407) for the co-ordinates provided by the Warp Controller (410). The Transparency data is passed to the Video Interpolator (411) via the Alpha Data Bus (413). The method for the application of the Alpha Map and layering of the actual video pixels can vary by the implementation of the Video Interpolator (411). By extending the data width of Warp Memory Buffer (407), useful attributes similar to lighting data or alpha data for future warp effects implementations easily can be added.
In order to maximize the available bandwidth in the microprocessor (401), the Microprocessor Address Bus (402) and Microprocessor Data Bus (403), extensive use of a microprocessor's internal memory can be used. The use of internal memory frees the microprocessor's address and data bus for the purposes of transferring transformation co-ordinate data, lighting data, transparency data, and the like, to the Warp Memory Controller (404). Many microprocessors allow for the external bus to function independently of the internal bus through DMA allowing for data to be transferred to the external devices simultaneously with the microprocessor calculating it's warp and other data. Additionally, in many implementations, a microprocessor's internal memory provides a much faster access to program instructions and data than do external memory devices.
As detailed in
During the initialization of a software algorithm, the algorithm software is transferred from the external program memory (505), via the External Data bus (507) and DMA Controller to the Internal Memory (502). The Program Execution Units (503) are the means within the Microprocessor where the actual software algorithms are run. Software instructions are therefore fetched from the internal memory (502) and processed by the program Execution Units (503) according to the sequence in the software algorithm. Intermediate and output data (e.g., transformation co-ordinate data, lighting data, transparency data, and the like) from the algorithm can be written back to the internal memory (502). Once a pre-determined threshold quantity of such data have been written to the Internal Memory (502), the DMA controller (504) is instructed to begin the transfer of this data to the Warp Memory Controller (506) via the external data bus (507). Furthermore, the DMA Controller (504) can move data to the external data bus independently of the program execution, thereby eliminating the need for the software algorithm to wait for any external memory access or data bus activity.
The devices and subsystems of the exemplary embodiments described with respect to
As noted above, it is to be understood that the exemplary embodiments, for example, as described with respect to
The exemplary embodiments described with respect to
All or a portion of the exemplary embodiments described with respect to
While the present invention have been described in connection with a number of exemplary embodiments and implementations, the present invention is not so limited but rather covers various modifications and equivalent arrangements, which fall within the purview of the appended claims.
The present invention claims benefit of priority to U.S. Provisional Patent Application Ser. No. 60/556,506 of ROSS et al., entitled “Method, System and Device for Real-Time Non-Linear Video Transformations,” filed Mar. 26, 2004, the entire disclosure of which is hereby incorporated by reference herein.
| Number | Date | Country | |
|---|---|---|---|
| 60556506 | Mar 2004 | US |
