The following relates to generating on a display the intersection of multiple digital videos and/or multiple digital images.
An intersection effect is an image or video processing technique in which multiple planar objects that each looks 3-D meet and cross each other in space.
For example, in digital video production, an intersection effect may be created in which a first video image and a second video image are each manipulated (e.g. shrunk and rotated to look 3-D), and then intersected.
In computer graphics applications, an intersection effect may also be created in which computer digital images are interested.
Methods and systems for providing an intersection effect are disclosed herein.
For example, in one embodiment, there is provided a method of intersecting a first digital image and a second digital image. Specifically, the first and second digital images are mixed to produce a first composite image in which the first digital image is layered on top of the second digital image. The first and second digital images are also mixed to produce a second composite image in which the second digital image is layered on top of the first digital image. The first composite image and the second composite image are then mixed using a wipe pattern to produce a target digital image comprising the intersection of the first digital image and the second digital image. The wipe pattern is a function of a difference in depth between the first digital image and the second digital image.
The wipe pattern may include a set of values. The set of values may include one value for each pixel location of a plurality of pixel locations in the target digital image. In some of such embodiments, for a pixel location of the plurality of pixel locations in the target digital image, the wipe pattern has a value that is a function of a difference between a depth value associated with the first digital image at the pixel location and a depth value associated with the second digital image at the pixel location.
Mixing the first composite image and the second composite image using the wipe pattern could include: computing the difference between the depth value associated with the first digital image at the pixel location and the depth value associated with the second digital image at the pixel location to obtain a depth difference value; and as a function of the depth difference value, setting a pixel value of the target digital image at the pixel location as: a pixel value of the first composite image at the pixel location, or a pixel value of the second composite image at the pixel location, or a combination of the pixel value of the first composite image at the pixel location and the pixel value of the second composite image at the pixel location.
In some embodiments, the method may further include: modifying the depth difference value to obtain a wipe pattern value wf, where the depth difference value is modified based on at least the pixel location and at least some parameters defining a perspective transformation of an original source image; and setting the pixel value of the target digital image at the pixel location, the pixel value being set as the pixel value of the first composite image at the pixel location where wf is greater than a first threshold, the pixel value being set as the pixel value of the second composite image at the pixel location where wf is less than a second threshold, and the pixel value otherwise being set as the combination of the pixel value of the first composite image at the pixel location and the pixel value of the second composite image at the pixel location. In some such embodiments this may provide a uniform softness around an intersection line in the target digital image.
Modifying the depth difference value to obtain the wipe pattern value wf could include multiplying the depth difference value by: (i) a gain value dependent upon at least some parameters defining the perspective transformation, and (ii) another value dependent upon at least the target image coordinate values xt and yt of the pixel location and at least some parameters defining the perspective transformation. In some embodiments, the gain value may include a user-controllable parameter that defines a range of softness around the intersection line.
In some embodiments, at least one of the depth value associated with the first digital image at the pixel location and the depth value associated with the second digital image at the pixel location may be computed using: (i) at least some parameter values that define a transformation of an original source image, and/or (ii) the target image coordinate values xt and yt of the pixel location.
The target digital image may, in some embodiments, be intersected with a third digital image by: mixing the target digital image and the third digital image to produce a third composite image in which the third digital image is layered on top of the target digital image; mixing the target digital image and the third digital image to produce a fourth composite image in which the target digital image is layered on top of the third digital image; and mixing the third composite image and the fourth composite image using another wipe pattern that is a function of a difference in depth between the target digital image and the third digital image.
The third digital image could represent the intersection of further digital images.
In some embodiments the method could involve intersecting a plurality of digital images by performing operations including intersecting the target digital image with another input digital image to obtain an output digital image and then further intersecting the output digital image with a further input digital image.
In some embodiments, computing the difference between the depth value associated with the first digital image at the pixel location and the depth value associated with the second digital image at the pixel location to obtain the depth difference value may comprise either: subtracting the depth value associated with the second digital image at the pixel location from the depth value associated with the first digital image at the pixel location, or subtracting the depth value associated with the first digital image at the pixel location from the depth value associated with the second digital image at the pixel location.
In some embodiments, mixing the first digital image and the second digital image to produce the first composite image may include: for each pixel location of a plurality of pixel locations of the first composite image, multiplying a pixel value of the second digital image at the pixel location with a complementary alpha value corresponding to the first digital image to obtain an intermediary value, and then adding a pixel value of the first digital image at the pixel location to the intermediary value. In some such embodiments, the complementary alpha value corresponding to the first digital image may be complementary to an alpha value that has already been applied to the first digital image.
In some embodiments, the depth value associated with the first digital image at the pixel location is representative of how far the first digital image appears from a viewer at the pixel location, and the depth value associated with the second digital image at the pixel location is representative of how far the second digital image appears from the viewer at the pixel location.
In some embodiments, an apparatus is disclosed to intersect a first digital image and a second digital image. The apparatus may include: a first mixer to mix the first digital image and the second digital image to produce a first composite image in which the first digital image is layered on top of the second digital image; a second mixer to mix the first digital image and the second digital image to produce a second composite image in which the second digital image is layered on top of the first digital image; a third mixer to mix the first composite image and the second composite image using a wipe pattern to produce a target digital image comprising the intersection of the first digital image and the second digital image; and a wipe pattern generator to generate the wipe pattern as a function of a difference in depth between the first digital image and the second digital image.
The first mixer, the second mixer, the third mixer, and/or the wipe pattern generator could be implemented by one or more integrated circuits. The one or more integrated circuits may include at least one of a field-programmable gate array (FPGA) and/or an application-specific integrated circuit (ASIC).
In some embodiments, the apparatus may include a memory and a processor to execute instructions stored in the memory, the instructions, when executed by the processor, cause the processor to implement the first mixer, the second mixer, the third mixer, and the wipe pattern generator.
In some embodiments, the apparatus may include circuitry to: compute the difference between the depth value associated with the first digital image at the pixel location and the depth value associated with the second digital image at the pixel location to obtain a depth difference value; and as a function of the depth difference value, set a pixel value of the target digital image at the pixel location as: a pixel value of the first composite image at the pixel location, or a pixel value of the second composite image at the pixel location, or a combination of the pixel value of the first composite image at the pixel location and the pixel value of the second composite image at the pixel location.
In some embodiments, the apparatus may comprise circuitry to: modify the depth difference value to obtain a wipe pattern value wf, the depth difference value being modified based on at least the pixel location and at least some parameters defining a perspective transformation of an original source image; and set the pixel value of the target digital image at the pixel location, the pixel value being set as the pixel value of the first composite image at the pixel location where wf is greater than a first threshold, the pixel value being set as the pixel value of the second composite image at the pixel location where wf is less than a second threshold, and the pixel value otherwise being set as the combination of the pixel value of the first composite image at the pixel location and the pixel value of the second composite image at the pixel location.
The apparatus may, in some embodiments, include a multiplier to modify the depth difference value to obtain the wipe pattern value wf by multiplying the depth difference value by: (i) a gain value dependent upon at least some parameters defining the perspective transformation, and (ii) another value dependent upon at least the target image coordinate values xt and yt of the pixel location and at least some parameters defining the perspective transformation.
The apparatus may, in some embodiments, include circuitry to compute at least one of the depth value associated with the first digital image at the pixel location and the depth value associated with the second digital image at the pixel location using: (i) at least some parameter values that define a transformation of an original source image, and (ii) the target image coordinate values xt and yt of the pixel location.
In some embodiments, the apparatus may further include: a fourth mixer to mix the target digital image and a third digital image to produce a third composite image in which the third digital image is layered on top of the target digital image; a fifth mixer to mix the target digital image and the third digital image to produce a fourth composite image in which the target digital image is layered on top of the third digital image; and a sixth mixer to intersect the target digital image with the third digital image by mixing the third composite image and the fourth composite image using another wipe pattern that is a function of a difference in depth between the target digital image and the third digital image.
In one embodiment there is provided an apparatus to intersect more than two digital images, the apparatus including a plurality of identical computation circuitry, each one of the plurality of identical computational circuitry comprising the apparatus above to receive as an input a pair of input digital images and to produce as an output a target digital image comprising the intersection of the pair of input digital images. The plurality of identical computational circuitry may be arranged so that an output target digital image of one of said plurality of identical computational circuitry is used as an input digital image of another of said plurality of identical computational circuitry. In some embodiments, two of said plurality of identical computational circuitry may be arranged in parallel to each other, with the output of each of said two of said plurality of identical computational circuitry fed to an input of another of said plurality of identical computational circuitry.
Other aspects and features of embodiments of the present disclosure will become apparent to those ordinarily skilled in the art upon review of the following description.
Embodiments of the present application will be described, by way of example only, with reference to the accompanying figures wherein:
Like reference numerals are used in different figures to denote similar elements.
For illustrative purposes, specific example embodiments will now be explained in greater detail below in conjunction with the figures.
The embodiments set forth herein represent information sufficient to practice the claimed subject matter and illustrate the best way of practicing such subject matter. Upon reading the following description in light of the accompanying figures, those of sufficient skill will understand the concepts of the claimed subject matter and will recognize applications of these concepts not particularly addressed herein. It should be understood that these concepts and applications fall within the scope of the disclosure and the accompanying claims.
Moreover, it will be appreciated that any module, component, or device exemplified herein that executes instructions may include or otherwise have access to a non-transitory computer/processor readable storage medium or media for storage of information, such as computer/processor readable instructions, data structures, program modules, and/or other data. A non-exhaustive list of examples of non-transitory computer/processor readable storage media includes magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, optical disks such as compact disc read-only memory (CD-ROM), digital video discs or digital versatile disc (i.e. DVDs), Blu-ray Disc™, or other optical storage, volatile and non-volatile, removable and non-removable media implemented in any method or technology, random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology. Any such non-transitory computer/processor storage media may be part of a device or accessible or connectable thereto. Any application or module herein described may be implemented using computer/processor readable/executable instructions that may be stored or otherwise held by such non-transitory computer/processor readable storage media.
Turning now to the figures, some specific example embodiments will be described.
Digital video comprises a series of digital images displayed in rapid succession. Each digital image includes a picture portion, which is what is displayed, and typically some ancillary information (e.g. audio, metadata, etc.).
The “Y” in “Y(x,y)” in general may be (and is) in some portions of the description below replaced with the letter designating the image being discussed. For example, an image F has a pixel value F(x,y) at pixel location (coordinate) (x,y).
Also,
Digital Video Effects and Video Mixing
A digital video may be manipulated to produce a digital video effect such as resizing, rotation, translation, and/or distortion of the video. Such manipulation may be performed by a digital video effects (DVE) device. A “DVE device” is sometimes also referred to as a “DVE system” or just a “DVE”.
A digital video effect is a visual effect which provides comprehensive video image manipulations in 3-D space, primarily dealing with resizing, rotation, translation or distortion of a source visual image. For example,
One way to perform a digital video effect is to apply a geometric transformation (or mapping) of each digital image of the digital video. Such geometric transformations are also referred to as geometrical coordinates mapping. A geometrical mapping is called forward mapping if coordinates of a source video image are transformed into coordinates of a target video image. Conversely, a geometrical mapping is called an inverse mapping if a transform starts from a target video image and generates a source video image.
A video source undergoing digital video effects in a DVE device may be further layered over one or more other video images using video mixing.
Video layering is a process that combines a sequence of separate video sources into a single video. One or more of these separate source videos may have been previously manipulated via a digital video effect, although this is not necessary. Video layering may provide a visual effect separate from (outside of that) provided by a DVE device, and/or video layering may be performed by a DVE device as part of the digital video effect.
Video layering is a type of video mixing, and may also be referred to as video compositing or video keying in production switchers.
For example,
Video layering may be performed by a mixer that mixes the video images together to generate the layered image. Such a mixer may use a method that is sometimes referred to as “A/B mix effect”. Such a video mixing method implements the following function:
O
mix
=αA+(1−α)B, where αε[0,1]. Equation (1)
In Equation (1), the parameters “A” and “B” each represent a video image. The video image “A” is layered on top of the video image “B”. The parameter Omix is the layered video image. The parameter “a” represents a keying alpha. The keying alpha may be independent of the videos A and B. Alternatively, the keying alpha may be associated with video image A. The keying alpha “α”, and its role in Equation (1), is explained in detail below.
Some digital video images have an alpha associated with them, the purpose of which is to “blank out” (i.e. make transparent) the entire video image, except for an area or item of interest. Since a digital video comprises a series of digital video images, there may be an alpha associated with each video image in the video. These associated alpha values may be transmitted in an “alpha channel” associated with the video, which is a series of associated alphas, one for each video image. Sometimes such an alpha channel is referred to as an “alpha channel against normal RGB or YC components”. However, in the remainder of this document, such an alpha channel will be referred to as an “associated alpha channel” that is associated with a particular video. Also, in the explanation below, often a digital video image is discussed (since the processing occurs on an image-by-image basis), and so “alpha channel” may not be used, but instead “associated alpha” may be used, it being understood that the associated alpha is associated with the particular digital video image, and the associated alpha may be part of an associated alpha channel.
The alpha α 142 associated with the first video image 140 comprises a set of alpha values α(x,y), one for each pixel coordinate (x,y) in the first video image 140. The associated alpha α 142 is for making the entire first video image 140 transparent, except for around an object of interest, which in this example is the car 143. In one embodiment, this is achieved as follows: for a given pixel coordinate (x,y) in the first video image 140, the associated alpha value α(x,y) is set to 1 if the pixel coordinate falls within the area of interest (i.e. within the car); otherwise the associated alpha value α(x,y) is set to 0. This is illustrated in
One way to apply the associated alpha α 142 to the first video image 140 is to multiply each pixel value Y(x,y) in the first video image 140 with its corresponding associated alpha value α(x,y). The examples illustrated herein use this method, which is one reason why α(x,y)=0 is used to denote full transparency of the first video image pixel and α(x,y)=1 is used to denote full opaqueness of the first image pixel. Specifically, when α(x,y)=0, then Y(x,y)×α(x,y)=Y(x,y)×0=0, which means that the image pixel displays no value and hence is fully transparent, and when α(x,y)=1, then Y(x,y)×α(x,y)=Y(x,y)×1=Y(x,y), which means the image pixel displays its full value. In some embodiments, the pixel coordinates on the border of the object of interest (i.e. on the edge of the car 143) may have an associated alpha value α(x,y) equal to a number between zero and one for semi-transparency to try to achieve a smoother transition between the object of interest (the car 143) and any background image over which the object of interest will be layered.
In the illustrated example, it is assumed that video image 144 has no associated alpha, or that it has already been applied to video image 144 (i.e. video image 144 has been “pre-shaped” by its associated alpha), such that video image 144 as illustrated is the area of interest for this image.
The mixer 146 implements equation (1), i.e.: Omix=αA+(1−α)B.
The mixer 146 multiplies each pixel value Y(x,y) in the first image 140 with its corresponding associated alpha value α(x,y). The complementary shaping alpha value, that is 1−α, is applied to the second image 144 in a similar manner, so that the pixels of the second image 144 are completely transparent when the corresponding pixels of the first image 140 are completely opaque, and vice versa, and so that a complementary amount of semi-transparency is applied to each second image pixel corresponding to a semi-transparent first image pixel.
More specifically, in operation, the “Port A” receives two inputs: the first image 140 and the associated alpha value α 142, and the “Port B” receives a single input: the second image 144. The associated alpha value α 142 is applied to the first image 140 by multiplying the first image 140 with the associated alpha value, as at multiplier 152. More specifically, each pixel value Y(x,y) in the first image 140 is multiplied with its corresponding associated alpha value α(x,y). The complementary associated alpha value 1−α is applied to the second image 144 in the same manner, as at multiplier 154. Then the two images are added, at adder 156, to form the layered output image 148. Thus, in this way, the mixer 146 in this example embodiment implements the function Omix=Aα+B*(1−α) (i.e. Equation (1)), where A is the first video image 140 and B is the second video image 144.
In the embodiment described above with reference to
O
mix
=wA+(1−w)B, where wε[0,1]. Equation (2)
Note that this is the same as Equation (1), except that the notation w is used instead of α to clarify that the alpha w is a wipe alpha defining the transition effect and is independent of the video images A and B.
For example,
With reference to
In some embodiments, it may be the case that the value of the wipe pattern w around the transition between w=0 and w=1 (i.e. at the seam 156 between the first and second video images) is instead between zero and one in order to provide some “fuzziness” or “softness” around the seam 156 with the goal of improving the look of the seam 156 compared to a hard switch. This “softness” is produced by the mixing of semi-transparent pixels from both layers.
To wipe from the first video image 140 to the second video image 144, the wipe pattern w changes over time (e.g. every frame or field or under the control of a user in some embodiments), to reveal more of the second image 144 and show less of the first video image 140 until the wipe from the first video image 140 to the second image 144 is complete. Such a wipe may be performed, for example, in a production switcher.
A wipe transition is a spatial-variant pattern in that for a given wipe pattern w applied to a pair of images, the specific value of w for each coordinate, i.e., w(x,y), will vary. For example, in the wipe pattern w 150 shown in
Wipe transitions are discussed in more detail and used herein, and therefore they deserve some additional attention.
As discussed above, a wipe transition may be defined by Equation (2): Omix=wA+(1−w)B, where wε[0,1] is the wipe pattern. Such a wipe transition may be used in television broadcast industries, for example, in production switchers.
In some embodiments it may be desirable that a wipe transition have what is referred to as a “strong rule of seamless switching”. What this means is that if a wipe transition occurs between two videos A and B, and if it turns out that A and B are identical videos, then the mix output Omix=wA+(1−w)B should be identical to A and B, regardless of what type of wipe pattern w is being applied. That is, if videos A and B are identical, then a viewer should not even know there is a wipe between the two videos and should not see the seam between the two videos, regardless of the specific type of wipe pattern w being applied. This is:
If A=B, Omix=wA+(1−w)B=A=B. Equation (3)
This “strong rule of seamless switching” may be used as a test or check to see if equipment, such as a production switcher and/or a DVE device, implements wipe transitions of acceptable quality. If the “strong rule of seamless switching” is met, then the implementation of the wipe transitions may be considered of acceptable quality.
Such a “strong rule of seamless switching” may be achieved if using a mixer to implement the wipe using an A/B mix Omix=wA+(1−w)B. Problems may occur in meeting the “strong rule of seamless switching”, for example, if a mixer such as A/B mix Omix=wA+(1−w)B is not used for a particular transition effect, but instead an “ad hoc” approach is used to try to deal with a more complicated wipe transition (e.g. between three or more video sources when some of the video sources have transparent or semi-transparent areas and some or all of the video sources are intersected).
Wipe Patterns and Associated Alphas
In a production switcher, a wipe pattern w may be used to implement a wipe transition to seamlessly overlay a video image A over another video image B via an A/B mix. Generally, video image B is the background in the A/B mix effect. It may be assumed to be solid without any transparent or translucent areas. In other words, video image B is assumed not to be associated with any alpha (or at least if video image B had an associated alpha, it has already been applied so that video image B represents the area of interest). Video image A is assumed to have an associated alpha α that defines transparency or semi-transparency in video image A.
As an example, the video image B may be what is currently shown on air, and it may be desired to show on air video image A layered over video image B. For example, video image A may be a network logo to be layered over the image B on air. In such a situation, the current output on air is
O
pgm
=B,
where Opgm stands for program output, and the desired end product is the preview:
O
pvw
=αA+(1−a)B.
Using the example above, Opgm=B may be what is currently shown on air, video image A may be a network logo to be displayed having an associated alpha α that makes all of image A transparent except for the logo, and Opvw=αA+(1−α)B is what is to be transitioned to (i.e. the network logo on top of what is currently shown on air). A transition is applied to switch from Opgm (i.e. what is currently on air) to Opvw (i.e. the desired end result in which the logo is on top of what is currently on air). Using the A/B mix effect to apply a wipe transition from Opgm to Opvw:
As can be seen from Equation (4), the wipe transition can be applied by using an A/B mix in which the alpha input is wα (i.e. the alpha α associated with image A modified by the wipe pattern w).
Shaping Alpha
In the description thus far, two types of alphas have been discussed:
(1) an alpha α associated with a video image, which defines transparency or semi-transparency in the video image and is used, for example, to highlight an object of interest in the video image and suppress or “blank out” (i.e. make transparent) the other areas of the image;
(2) a wipe alpha used for defining a transition effect between two video images, such as a wipe pattern w, which may be independent of the content of the video images themselves.
When a video image is subject to a digital video effect (e.g. 3-D manipulations such as resizing, rotation, and/or distortion), for example in a DVE device in a production switcher, another type of alpha may be generated that is associated with that transformed video image. This is referred to as a “shaping alpha” αshp. Specifically, the shaping alpha αshp defines the resultant boundaries of a target video (i.e. the transformed source video) and helps layering the target video image on the top of other video images, after a source video undergoes a series of geometrical transformations. Values of the shaping alpha αshp may also range from zero to one, where for a given pixel coordinate a zero indicates that the target image (transformed source image) is out of source bounds, and where one indicates that the target image is within the source bounds.
For example, with reference to
In operation, the DVE device 170 receives as an input the source video image 172, and the DVE device 170 outputs the manipulated source video image 173 in a target video image 174. The mixer 175 layers the target video image 174 on top of the background image 176 to produce the output video image 179.
When the source video image 172 undergoes manipulation in the DVE device 170, a shaping alpha αshp is also created, which defines the resultant boundaries of the transformed source image in the target image. The shaping alpha αshp is particularly useful for situations like that illustrated in
As shown in
Therefore, for a source image 172, the DVE device 170 creates a corresponding target image 174 that includes an object 173 representing the transformed source image 172. In general, some of the pixels of the target image 174 will be within source bounds (i.e. within object 173) and others will be outside source bounds (i.e. in area O). This scenario is illustrated in
The shaping alpha αshp created by the DVE device 170 comprises a plurality of values, each one of the values of the shaping alpha αshp corresponding to a respective pixel in the target image 174. Each one of the shaping alpha values of the shaping alpha αshp will therefore be denoted as αshp(xt,yt). As mentioned above, each value αshp(xt,yt) in the shaping alpha may range from zero to one (i.e. 0≦αshp(xt,yt)≦1). A value of zero indicates that the corresponding target image pixel is out of bounds of the source image (i.e. the pixel of the target image 174 is in area O), and therefore when the target image 174 is layered on top of the background image 176, that pixel of the target image will be completely transparent, such that only the background 176 is seen. On the other hand, a shaping alpha value αshp(xt,yt) of one indicates that the corresponding target image pixel is within the bounds of the source image (i.e. the pixel of the target image 174 is within the area defined by L, R, T, and B), and therefore when the target image 174 is layered on top of the background 176, that pixel of the target image will be completely opaque in the sense that it will not show any of the background image and only show the target image. A shaping alpha value αshp(xt,yt) between zero and one indicates a boundary condition (a border area of the object 173) in which partial transparency (semi-transparency) is present to allow for a smooth transition between the edge of the object 173 and the background image 176. The closer the shaping alpha value αshp(xt,yt) is to one, the more the object 173 will show, and the closer the shaping alpha value is to zero, the more the background image 176 will show. By using the shaping alpha αshp, the transformed source image (i.e. the object 173) may be made to fade out gradually into the background image 176 at edge boundaries L, R, T, and B, which is typically more aesthetically pleasing to a human eye.
The shaping alpha αshp corresponding to the target image 174 is shown as 178 in
As a simple example to further help illustrate the principle of the shaping alpha αshp, consider
For example, target image pixel (103,74) is outside the boundary L and specifically in area O of the target image 174. Therefore, this pixel has a corresponding shaping alpha value of zero (αshp (103,74)=0). This means that this pixel will be completely transparent in that it will only show the background image 176 at this pixel location when the background image 176 is layered underneath the target image 174. That is, no matter what the target image pixel value Y(103,74) is, it will not be displayed. Instead, only the pixel value of the background image 176 layered underneath the target image 174 will be displayed. On the other hand, target image pixel (106,74) is within the boundaries of the source image and has a corresponding shaping alpha value of one (αshp (106,74)=1). This means that this pixel will be completely opaque in that it will only show the target image 174 at this pixel location when the background image 176 is layered underneath the target image 174. That is, Y(106,74) will be displayed with no corresponding background image pixel being shown. Target image pixel (105,74) is on the boundary L (specifically in the softness range) and therefore has a corresponding shaping alpha value that is between zero and one and specifically 0.8 in this example (αshp(105,74)=0.8). This means that this pixel of the target image 174 will have semi-transparency in that it will show some of the background image 176 at this pixel location when the background image 176 is layered underneath the target image 174. That is, no matter what the target image pixel value Y(105,74), the pixel value of the background image 176 layered underneath the target image 174 at this point will still also be partially displayed. Similarly, target image pixel (104,74) is also on the boundary L and therefore has a corresponding shaping alpha value that is between zero and one and specifically 0.5 in this example (αshp(104,74)=0.5). This means that this pixel of the target image 174 will also have semi-transparency in that it will show some of the background image 176 at this pixel location when the background image 176 is layered underneath the target image 174. Since 0.5 is closer to 0 than 0.8, the pixel value Y(104,74) will be more transparent than the pixel value Y(105,74). As mentioned above, a primary purpose of the shaping alpha is to cause the transformed source image (i.e. the object 173 in the target image 174) to fade out gradually into the background image 176 at edge boundaries of the object 173 (e.g. at boundary L), which is typically more aesthetically pleasing to a human eye.
Returning now to
Note that if there is also an associated alpha α that is associated with the source video image 172, then this associated alpha α is also manipulated by the DVE 170 in the same way the source video image 172 is manipulated, and it becomes the alpha associated with the target image 174. It may be applied to the target image 174 by, for example, multiplying it by the shaping alpha prior to the mixer 175, i.e. α=α×αshp.
Perspective Transformation and Image Depth
As discussed above, a DVE device may manipulate a video image to provide an effect such as resizing, rotation, translation, or distortion. This is shown, for example, in
Assume a source image comprising a grid of 2-D pixels (like that illustrated in
To obtain the corresponding y-coordinate of the source pixel (i.e. ys) for a given target pixel (xt,yt), the following computation is performed by the perspective transform engine:
Equations (5) and (6) are based on mapping of a target pixel coordinate to a corresponding source pixel coordinate using the following inverse mapping function (geometric transformation):
where M−1 is a homogeneous matrix, inverse to a forward mapping M, where a 4×4 homogeneous matrix represents a geometrical transformation. The value f represents a focus distance between the viewer and the target screen.
In this example, Ta, Tb, Tc, Td, Te, Tf, Tg, Th, Pa, Pb, Pc, Pd, and f are parameters set based on the specific effect being applied to the source image. For example, if it was desired to rotate the source image by 15 degrees and shrink the image to half its size, then the corresponding values of the parameters Ta, Tb, Tc Td, Te, Tf, Tg, Th, Pa, Pb, Pc, Pd, and f would be computed.
One way to compute the parameters is as follows. A matrix M=Mt×My×Mz is computed, where
and represents the translation or shift in 3-n space by (x,y,z), where
and represents the rotation at angle β around the y-axis, and where
and represents a rotation at angle φ around the z-axis. M−1 is then the inverse of M. Therefore, for a given effect, the translation and rotation angles around the y and z axis are determined. Then, M=Mt×My×Mz is computed, which gives the effect of rotation around the z-axis first, then rotation around the y-axis, and then translation. Either in parallel to computing M (or from M), the inverse matrix M−1 is computed to determine parameters Ta, Tb, Tc, Td, Te, Tf, Tg, Th, Pa, Pb, Pc, Pd. Also, the focus f is determined based on the effect. Then, Equations (5) and (6) above may be computed for each target pixel coordinate (xt,yt).
Equations (5) and (6) relate to the parameters from matrix M−1 as follows. Subjected to perspective projection, the depth of a point in 3-D space may be expressed in terms of a target pixel coordinate (xt,yt) such that
where
Equations (5) and (6) may also be rearranged to instead be in terms of unified source coordinates
For a given source video, the parameters above (i.e. the parameters in matrix M−1, and the value of f) are computed on an image-by-image basis, although this need not be the case (particularly if the effect does not change between video images).
In the matrix M−1, the parameters in the first two rows (Ta, Tb, Tc, Td, Te, Tf, Tg, Th) generally relate to providing the transform, whereas the parameters in the third row (Pa, Pb, Pc, Pd) generally relate to providing the perspective.
The effect applied to the source image is 3-D in the sense that the target image can appear in 3-D space even though the screen is 2-D. For example, with reference to
where
As also referenced above, the pixel coordinate in the source image (xs,ys) corresponding to the target pixel coordinate (xt,yt) is:
x
s
=
and
y
s
=
Intersection Effect
As discussed above, a DVE device may manipulate a video image to provide an effect such as resizing, rotation, translation, or distortion. An intersection effect is another digital video effects feature that may be performed by a DVE device in which multiple of planar objects meet and cross each other in 3-D space. For example,
In operation, the first perspective engine 182 receives first image 180 and subjects it to a transformation to produce a first transformed image 184, which is still a 2-D image, but because of the transformation has the look of being 3-D (i.e. having a depth). Similarly, the second perspective transform engine 188 receives second image 186 and subjects it to another transformation to produce a second transformed image 190, which is also still a 2-D image, but because of the transformation has the look of being 3-D (i.e. having a depth). The intersection effect system 192 receives as its input each transformed image 184 and 190 and intersects the two images to produce a composite image 194 that displays the intersection effect, i.e. the intersection of images 184 and 190 at their middle areas.
Specifically, to perform the intersection in this embodiment, the composite image 194 is generated as follows: for each pixel location, if the second transformed image 190 has an associated depth value that exceeds the associated depth value of the first transformed image 184 (i.e. the first transformed image 184 appears closer to the viewer), then the pixel value of the first transformed image 184 is displayed. On the other hand, if the first transformed image 184 has an associated depth value that exceeds the associated depth value of the second transformed image 190 (i.e. the second transformed image 190 appears closer to the viewer), then the pixel value of the second transformed image 190 is displayed. The pixel locations at which transformed images 184 and 190 have the same associated depth value (i.e. appear equal distance from the viewer) are the intersection line.
Therefore, the intersection effect performs swapping of two images across their intersection line. Such a swap depends on the difference in depth between the two images.
As is clear from the two paragraphs above, creating the intersection of two transformed images takes into account the associated depth value of each pixel of each transformed image. This associated depth value is a value representative of how far away the image appears to the viewer due to the 3-D projection perspective effect. The depth value can be computed, for example, using the parameters defining the perspective transformation, e.g. as per Equation (8) above.
The description above with respect to
As described above, one example way to produce an intersection effect is to use a depth difference to switch visibility between two video images. This depth-based method for visibility may be referred to as “Z-Buffer”, particularly in computer graphics applications. The measurement of depth difference between two transformed images is explained below with reference to
Turning to
In
The z-axis is representative of the perceived depth of each image A and B. Specifically, for a given target coordinate Vtgt on the viewer's screen, there is a depth value ZA associated with that pixel coordinate on the video image A and another depth value ZB associated with that pixel coordinate on the video image B. These points on images A and B are respectively shown as VA and VB in
That is, the target pixel Vtgt displayed on the screen is the corresponding pixel in image A (i.e. VA) if the depth ZB of image B exceeds the depth ZA of image A (i.e. image A appears closer to the viewer), and the target pixel Vtgt displayed on the screen is the corresponding pixel in image B (i.e. VB) if the depth ZA of image A exceeds the depth ZB of image B (i.e. image B appears closer to the viewer).
In Equation (11), the difference in depth between image A and image B at target pixel coordinate Vtgt is computed by subtracting the depth value ZB of image B from the depth value ZA of image A (i.e. ΔZ=ZA−ZB). This is because a convention is being assumed in which the z-axis intersects the screen 200 at z=0, and that positive values of the z-axis are in front of the screen 200 between the screen 200 and the viewer 202, and that negative values of the z-axis are behind the screen 200, such that the farther away an image appears from the viewer, the larger the negative number of the associated depth. For example, in
If it is the case that for a given target pixel coordinate Vtgt on the screen 200, the associated depth value of each of images A and B is the same (i.e. ΔZ=0), then Equation (11) indicates that image A is displayed. The value ΔZ=0 represents a point at which images A and B intersect. Having image A displayed on the intersection line is implementation specific. It could be the case that instead image B is displayed, or that a combination of both are displayed.
When creating an intersection effect such as that described above, for example, in a DVE device in a production switcher, transparent areas in video images may require special consideration. For example,
In
One way to address this is to not allow the intersection effect to be performed when the transformed images have transparent or semi-transparent areas. However, this limits the availability of the intersection effect. Another method is to develop an “ad hoc” or custom approach in which the transparent areas are identified and transformed video images A and B, as well as their associated alphas, are swapped at the transparent areas and layered, e.g. through Equation (1). However, such a method may not be able to be propagated into intersection effects in which more than 2 layers are intersected. Also, such an approach may not meet (or only barely meet) the “strong rule of seamless switching” described above in situations in which the intersection is occurring as part of a wipe transition from one video image to another.
One possible method to deal with multiple transparent or semi-transparent video images being intersected is to use a concept in computer graphics referred to as an A-buffer, which is described for example in the following references:
In addition to the issue of dealing with the intersection of video images having transparent (or semi-transparent) areas, the intersection method described above with reference to
The following provides alternative systems and method for performing an intersection effect in which transparent and/or semi-transparent areas in one or more of the images may be accommodated, and/or in which a uniformly smooth intersection between images may be obtained. Specifically, in the description below, first methods and systems for performing an intersection effect using a wipe-based mix are discussed, and then methods and systems for aiming to provide a uniformly smooth transition across the intersection are discussed.
Using a Wipe to Perform the Intersection Effect
As is clear from the above, an intersection effect performs swapping of images across an intersection line. Such a swap depends on the difference in depth between the images.
In embodiments discussed below, a wipe pattern based on depth difference is applied to multiple video layers for a general wipe transition that provides an intersection effect.
First consider two images to be intersected: image F1 having associated alpha α1, and image F2 having associated alpha α2. More specifically, assume the images are pre-shaped by their associated alpha, as described below, prior to their intersection, so that the output is the intersection of the pre-shaped images. That is, F1 is pre-shaped by its associated alpha α1 to result in {circumflex over (F)}1=α1F1, and F2 is pre-shaped by its associated alpha α2 to result in {circumflex over (F)}2=α2F2, and the output of the intersection effect is an image displaying the intersection of {circumflex over (F)}1 and {circumflex over (F)}2.
On the left hand side of
Another image F2 is also illustrated. The pre-shaped version is not used since, as is clear from the below, its associated alpha α2 is instead applied by A/B mixer 240. In general, F2 is also a transformed source image that may have a transparent or semi-transparent area. To show this generality, in
Mixer 240 layers F2 on top of {circumflex over (F)}1 to produce composite image 250. That is, the output of mixer 240 is: (1−α2){circumflex over (F)}1+α2F2={circumflex over (F)}1α2c+{circumflex over (F)}2, where α2c=(1−α2). Thus, the output of mixer 240 (i.e. composite image 250) is the resulting image when {circumflex over (F)}2 is mixed with the background {circumflex over (F)}1. The image {circumflex over (F)}2 is on top, and the transparent area 248 in {circumflex over (F)}2 correctly shows the image {circumflex over (F)}1.
On the right hand side of
Thus, the output of mixer 242 (i.e. composite image 252) is the resulting image when {circumflex over (F)}1 is mixed with background {circumflex over (F)}2. The image {circumflex over (F)}1 is on top, and the transparent area 246 in {circumflex over (F)}1 correctly shows the image {circumflex over (F)}2.
Composite images 250 and 252 are then input into further A/B mixer 244, which implements a wipe pattern w1,2 to produce the intersection of images {circumflex over (F)}1 and {circumflex over (F)}2, as shown in composite image 254, which is the output of the mixer 244. Specifically, by using a wipe pattern w1,2 based on a difference in depth between images {circumflex over (F)}1 and {circumflex over (F)}2 at each pixel location (i.e. pixel coordinate), the wipe pattern w1,2 may be used to construct the composite image 254 showing the intersection. One way to achieve this is as follows. For each pixel location:
(1) if the depth of image {circumflex over (F)}2 at the pixel location exceeds the depth of image {circumflex over (F)}1 at the pixel location (i.e. if image {circumflex over (F)}1 appears closer to the viewer), then the pixel value of the composite image 254 at the pixel location is set as the pixel value of the composite image 252 at the pixel location (i.e. the image where {circumflex over (F)}1 is on top of {circumflex over (F)}2);
(2) if the depth of image {circumflex over (F)}1 at the pixel location exceeds the depth of image F2 at the pixel location (i.e. if image {circumflex over (F)}2 appears closer to the viewer), then the pixel value of the composite image 254 at the pixel location is set as the pixel value of the composite image 250 at the pixel location (i.e. the image where {circumflex over (F)}2 is on top of {circumflex over (F)}1).
By this method the intersection effect is created, and the transparent areas 246 and 248 are correct, as each possible layering of the transparent areas are first created in composite images 250 and 252, and then the correct one chosen depending on whether {circumflex over (F)}1 appears closer to the viewer or {circumflex over (F)}2 appears closer to the viewer.
The output of mixer 244 is:
O
mix
=K
1,2({circumflex over (F)}2α1c+{circumflex over (F)}1)w1,2c+({circumflex over (F)}1α2c+{circumflex over (F)}2)w1,2. Equation (12)
The intersection occurs when {circumflex over (F)}1 and {circumflex over (F)}2 appear equally close to the viewer (i.e. the depth difference between images {circumflex over (F)}1 and {circumflex over (F)}2 is equal to zero). On the intersection line (i.e. when the depth difference is zero), the wipe pattern w1,2 may be implemented, for example, to display either composite image 250 or composite image 252. Alternatively, in other embodiments, to try to allow for a smoother transition across the intersection line, the wipe pattern w1,2 may be implemented to apply some “fuzziness” or “softness” in a softness range around the intersection line through the mixing of pixel values from each of composite image 250 and 252 on and around the intersection line. One example of such a wipe pattern may be as follows. For each pixel location:
(1) if the depth of image {circumflex over (F)}2 at the pixel location exceeds the depth of image {circumflex over (F)}1 at the pixel location (i.e. if image {circumflex over (F)}1 appears closer to the viewer) by a first threshold amount, then the pixel value of the composite image 254 at the pixel location is set as the pixel value of the composite image 252 at the pixel location (i.e. the image where {circumflex over (F)}1 is on top of {circumflex over (F)}2)
(2) if the depth of image {circumflex over (F)}1 at the pixel location exceeds the depth of image {circumflex over (F)}2 at the pixel location (i.e. if image {circumflex over (F)}2 appears closer to the viewer) by a second threshold amount, then the pixel value of the composite image 254 at the pixel location is set as the pixel value of the composite image 250 at the pixel location (i.e. the image where {circumflex over (F)}2 is on top of {circumflex over (F)}1).
(3) otherwise, the pixel value of the composite image 254 at the pixel location is set as combination of the pixel value of the composite image 252 at the pixel location and the pixel value of the composite image 250 at the pixel location.
The first threshold amount and the second threshold amount may be equal to each other or different from each other.
One example way to implement the wipe pattern above is as follows:
where Δf is a number representing the softness range, which in some embodiments may be controlled by a user (e.g. through a user interface), and ΔZ is the depth difference between images {circumflex over (F)}1 and {circumflex over (F)}2. The “first threshold” mentioned in the paragraph above may be +Δf and the “second threshold” mentioned in the paragraph above may be −Δf (or vice versa).
In this wipe pattern, when ΔZ=0 (i.e. on the intersection line), then w1,2=0.5, which means that both the pixel value from composite image 252 and the pixel value from composite image 250 contribute equally to the pixel value on the intersection line in composite image 254. More generally, in Equation (13), when the absolute value of the depth difference between the images {circumflex over (F)}1 and F2 is less than Δf (i.e. on or around the intersection line), then both the pixel value from composite image 252 and the pixel value from composite image 250 contribute to the pixel value in the composite image 254. This and other wipe patterns will be discussed in more detail later.
The embodiments described above with reference to
In operation, the pre-shaper 360 multiplies input image F1 with its associated alpha α1 at multiplier 368 to obtain pre-shaped image {circumflex over (F)}1=α1F1. The pre-shaper 360 also computes the complementary of associated alpha α1, i.e. α1c=1−α1, by subtracting α1 from one at subtractor 366. The pre-shaper 362 operates in the same way as pre-shaper 360, except that the input is image F2 and associated alpha α2. Specifically, the pre-shaper 362 multiplies input image F2 with its associated alpha α2 at multiplier 372 to obtain pre-shaped image {circumflex over (F)}2=α2F2. The pre-shaper 362 also computes the complementary of associated alpha α2, i.e. α2c=1−α2, by subtracting α2 from one at subtractor 370.
The video-mixer 364 creates composite images 250 and 252 and mixes them using a wipe pattern w1,2. Specifically, composite image 252 is created as follows: {circumflex over (F)}1 and {circumflex over (F)}2 are mixed to produce composite image 252 in which {circumflex over (F)}1 is layered on top of {circumflex over (F)}2 by multiplying α1c with {circumflex over (F)}2 at multiplier 374, and then adding this to {circumflex over (F)}1 at adder 382 to obtain {circumflex over (F)}2α1c+{circumflex over (F)}1. Composite image 250 is created as follows: {circumflex over (F)}1 and {circumflex over (F)}2 are mixed to produce composite image 250 in which {circumflex over (F)}2 is layered on top of {circumflex over (F)}1 by multiplying α2c with {circumflex over (F)}1 at multiplier 376, and then adding this to {circumflex over (F)}2 at adder 384 to obtain {circumflex over (F)}1α2c+{circumflex over (F)}2. The composite images {circumflex over (F)}2α1c+{circumflex over (F)}1 and {circumflex over (F)}1α2c+{circumflex over (F)}2 are then mixed using wipe pattern w1,2 to produce the intersection of {circumflex over (F)}1 and {circumflex over (F)}2 as follows: w1,2 is subtracted from one at subtractor 388 to obtain w1,2c=1−w1,2c, and then w1,2c is multiplied by {circumflex over (F)}2α1c+{circumflex over (F)}1 at multiplier 378 to obtain ({circumflex over (F)}2α1c+{circumflex over (F)}1)w1,2c and w1,2 is multiplied by {circumflex over (F)}1α2c+{circumflex over (F)}2 at multiplier 380 to obtain ({circumflex over (F)}1α2c+{circumflex over (F)}2)w1,2, and then ({circumflex over (F)}2α1c+{circumflex over (F)}1)w1,2c is added to ({circumflex over (F)}1α2c+{circumflex over (F)}2)w1,2 at adder 386 to obtain Equation (12):
K
1,2=({circumflex over (F)}2α1c+{circumflex over (F)}1)w1,2+({circumflex over (F)}1α2c+{circumflex over (F)}2)w1,2.
In view of the description above in relation to
In step 392, the first digital image (e.g. {circumflex over (F)}1) and the second digital image (e.g. {circumflex over (F)}2) are mixed to produce a first composite image (e.g. {circumflex over (F)}2α1c+{circumflex over (F)}1) in which the first digital image (e.g. {circumflex over (F)}1) is layered on top of the second digital image (e.g. {circumflex over (F)}2).
In step 394, the first digital image (e.g. {circumflex over (F)}1) and the second digital image (e.g. {circumflex over (F)}2) are mixed to produce a second composite image (e.g. {circumflex over (F)}1α2c+{circumflex over (F)}2) in which the second digital image is layered on top of the first digital image.
In step 396, the first composite image (e.g. {circumflex over (F)}1α2c+{circumflex over (F)}1) and the second composite image (e.g. {circumflex over (F)}1α2c+{circumflex over (F)}2) are mixed using a wipe pattern (e.g. w1,2) to produce a target digital image (e.g. K1,2 defined in Equation (12)) comprising the intersection of the first digital image (e.g. {circumflex over (F)}1) and the second digital image (e.g. {circumflex over (F)}2).
Because the intersection effect described above uses a wipe-based mix for performing the intersection of two images, it may be possible to achieve the “strong rule of seamless switching” mentioned earlier.
By way of further example,
On the left hand side of
On the right hand side of
Composite images 314 and 316 are then input into a further mixer 356, which implements a wipe pattern w1,2 to produce the intersection of {circumflex over (F)}1 and {circumflex over (F)}2, as shown in composite image 322, which is the output of the mixer 356. Specifically, by using a wipe pattern w1,2 based on a difference in depth between {circumflex over (F)}1 and {circumflex over (F)}2 at each pixel location (i.e. pixel coordinate), the wipe pattern w1,2 may be used to construct the composite image 322 showing the intersection. Specific examples of wipe patterns to achieve this are described above and also later.
The output of the mixer 356 is Equation (12):
K
1,2=({circumflex over (F)}2α1c+{circumflex over (F)}1)w1,2c+({circumflex over (F)}1α2c+{circumflex over (F)}2)w1,2.
This is then mixed with background image B via multiplier 358 and adder 359 as follows: the background image B is multiplied by α1,2c=α1c×α2c at multiplier 358, and then that intermediary value is added to K1,2 at adder 359. The output of adder 359 is:
O
mix
=Bα
1,2
c
+K
1,2. Equation (14)
where α1,2c=α1c×α2c, and K1,2=({circumflex over (F)}2α1c+{circumflex over (F)}1)w1,2c+({circumflex over (F)}1α2c+{circumflex over (F)}2)w1,2 (i.e. Equation (12)), and where α1c=1−α1, α2c=1−α2, w1,2c=1−w1,2, {circumflex over (F)}1=α1F1, and {circumflex over (F)}2=α2F2.
Just like the embodiment in
By way of further example, embodiments will now be described in which four different transformed source video images are intersected over a background B. That is, with reference to
In the example embodiments described below, different variations of how to provide the intersection effect are also discussed (e.g. parallel or cascade implementation). The specific variation used may be chosen based on the particular design resources and environment available.
Also, by way of summary, in the example embodiments below, the following notation will also be used, most of which has already been introduced earlier:
The example embodiments below assume a general process of video layering known as a mix tree. In a mix tree, there may be a current output on air, which is called the program branch and is denoted as Opgm. For example, if it were the case that the current output on air (the Opgm) was the composite image 316 in
In a mix tree, there may also be a branch that prepares the next output ready-to-be-used, which is called a preview branch and is denoted as Opvw. A wipe is then used to transition between Opgm and Opvw.
Using the mix tree notation of Opgm and Opvw, and assuming it is desired to intersect video images {circumflex over (F)}1 and {circumflex over (F)}2, then let Opgm=(Bα1c+{circumflex over (F)}1)α2c+{circumflex over (F)}2 (i.e. {circumflex over (F)}2 layered on top of {circumflex over (F)}1), and Opvw=(Bα2c+{circumflex over (F)}2)α1c+{circumflex over (F)}1 (i.e. {circumflex over (F)}1 and {circumflex over (F)}2 swapped so that {circumflex over (F)}1 is layered on top of {circumflex over (F)}2), and then the intersection of the two is:
(i.e. the same as Equation (14), where K1,2=({circumflex over (F)}2α1c{circumflex over (F)}1)w1,2c+({circumflex over (F)}1α2c+{circumflex over (F)}2)w1,2).
Assume though that two more video layers {circumflex over (F)}3, and then {circumflex over (F)}4, are being layered on top of the intersection of {circumflex over (F)}1 and {circumflex over (F)}2 in the mix tree. Then:
That is, the two video layers {circumflex over (F)}1 and {circumflex over (F)}2 in Equation (16) are assumed to be swapped to form Equation (17). The program branch defines {circumflex over (F)}2 on the top of {circumflex over (F)}1, while the preview branch swaps the order of {circumflex over (F)}1 and {circumflex over (F)}2. A wipe pattern w1,2 is used to mix Opgm and Opvw, or to swap the order of {circumflex over (F)}1 and {circumflex over (F)}2, and therefore define the intersection effect, i.e.:
Equation (18) defines a pair of video layers ({circumflex over (F)}1 and {circumflex over (F)}2) swapped through a wipe pattern w1,2 to produce intersected image K1,2 (i.e. an intersection of images {circumflex over (F)}1 and {circumflex over (F)}2), with image {circumflex over (F)}3 layered over K1,2, and then image {circumflex over (F)}4 layered on top of that.
The follow observations will be made.
First, the notation {circumflex over (F)}1=Fiαi, represents video image Fi in its shaped form (i.e. the associated alpha already applied). Some video sources already provide their video in shaped form (i.e. with the associated alpha already applied). If the video layer Fi is not pre-shaped by its associated alpha, then this may be done by computing {circumflex over (F)}i=Fiαi (i.e. for each pixel in video layer Fi multiplying the pixel value with the corresponding associated alpha value).
Also, K1,2 represents a basic swap operation on a pair of video layers with the property that:
The letter “K” is used in K1,2 because K1,2 can be thought to represent a mix (i.e. video keying). The “K” is for “keying”. The value α1,2c can be thought of as a complementary composite alpha associated with K1,2, which makes K1,2 behave as a new video source layer in a mix tree. The value w1,2 is the complementary expression to the wipe pattern w1,2.
Intersecting with Images F3 and F4: A Parallel Implementation
Assume video images {circumflex over (F)}3 and {circumflex over (F)}4 are also being intersected with images {circumflex over (F)}1 and {circumflex over (F)}2 to result in the intersection of four images as shown in
O
pgm=((Bα1,2c+K1,2)α3c+{circumflex over (F)}3)α4c+{circumflex over (F)}4 Equation (20)
(i.e. image {circumflex over (F)}3 on top of intersected images {circumflex over (F)}1 and {circumflex over (F)}2 and then image {circumflex over (F)}4 layered on top of that).
O
pvw=((Bα1,2c+K1,2)α4c+{circumflex over (F)}4)α3c+{circumflex over (F)}3 Equation (21)
(i.e. images {circumflex over (F)}3 and {circumflex over (F)}4 swapped: that is, image {circumflex over (F)}4 on top of intersected images {circumflex over (F)}1 and {circumflex over (F)}2, and then image {circumflex over (F)}3 layered on top of that).
To obtain images {circumflex over (F)}3 and {circumflex over (F)}4 intersected and layered on top of intersected images {circumflex over (F)}1 and {circumflex over (F)}2, a mix through a wipe pattern w3,4 is applied such that
where K3,4=({circumflex over (F)}3α4c+{circumflex over (F)}4)w3,4c+({circumflex over (F)}4α3c+{circumflex over (F)}3)w3,4.
Equation (22) defines a parallel structure for two pairs of wipe-based swap: {K1,2, K3,4}. Similar to α1,2c, the value α3,4c can be thought of as a complementary alpha associated with K3,4, which makes the second pair behave as a layer in a mix tree.
To now intersect the intersected video images {circumflex over (F)}1 and {circumflex over (F)}2 with the intersected video images F3 and {circumflex over (F)}4, a further swap between K1,2 and K3,4 may be performed as follows.
O
pgm=(Bα1,2c+K1,2)α3,4c+K3,4, Equation (23)
(i.e. intersected images {circumflex over (F)}3 and F4 layered over top of intersected images {circumflex over (F)}1 and {circumflex over (F)}2), and define Opvw as:
O
pvw=((Bα3,4c+K3,4)α1,2c+K1,2 Equation (24)
(i.e. a swap such that intersected images {circumflex over (F)}1 and {circumflex over (F)}2 are layered over top of intersected images {circumflex over (F)}3 and {circumflex over (F)}4).
To obtain images {circumflex over (F)}3 and {circumflex over (F)}4 intersected with intersected images {circumflex over (F)}1 and {circumflex over (F)}2, a mix through a wipe pattern w1:4 is applied such that
where K1:4=(K3,4α1,2c+K1,2)w1:4c+(K1,2α3,4c+K3,4)w1:4.
Note that α1:4c=α1,2cα3,4c=α1cα2cα3cα4c. However, the notation w1:4 does not indicate any sort of multiplication. This notation is just to designate the wipe pattern is one that intersects layers 1 to 4 together.
The value α1:4c may also be thought of as a complementary composite alpha that is associated with K1:4.
Generating the intersection effect between the four video images in the manner explained above may be considered a parallel implementation in that images {circumflex over (F)}1 and {circumflex over (F)}2 are intersected, images {circumflex over (F)}3 and {circumflex over (F)}4 are intersected (which may be done in parallel), and then the two intersected images are intersected. Specific example hardware for such an implementation is described later.
Intersecting Using a Cascade Implementation
Equation (15) above generates the intersection of video images 1 and 2: Omix=Bα1,2c+K1,2. Assume video images {circumflex over (F)}3 and {circumflex over (F)}4 are also being intersected with images {circumflex over (F)}1 and {circumflex over (F)}2 to result in the intersection of four images as shown in
Specifically, to first intersect image {circumflex over (F)}3 with intersected images {circumflex over (F)}1 and {circumflex over (F)}2, define the Opgm as:
O
pgm=((Bα1,2c+K1,2)α3c+{circumflex over (F)}3)α4c+{circumflex over (F)}4 Equation (26)
(i.e. image {circumflex over (F)}3 on top of intersected images {circumflex over (F)}1 and {circumflex over (F)}2, and then image {circumflex over (F)}4 layered on top of that).
O
pvw=((Bα3c+{circumflex over (F)}3)α1,2c+K1,2)α′4+{circumflex over (F)}4 Equation (27)
(i.e. a swap such that intersected images {circumflex over (F)}1 and {circumflex over (F)}2 are layered over top of image F3).
To obtain image {circumflex over (F)}3 intersected with intersected images {circumflex over (F)}1 and {circumflex over (F)}2, a mix through a wipe pattern w1:3 is applied such that
where, K1:3=({circumflex over (F)}3α1,2c+K1,2)w1:3c+(K1,2α3c+{circumflex over (F)}3)w1:3, and α1:3c=α1,2cα3c=α1cα2cα3c.
The value K1:3 can be thought of as representing a composite video image produced by combining an early or previous composite video image K1,2 with image {circumflex over (F)}3.
To now intersect the image {circumflex over (F)}4 with intersected video images {circumflex over (F)}1, {circumflex over (F)}2, and {circumflex over (F)}3, a further swap between {circumflex over (F)}4 and K1,3 may be performed as follows.
O
pgm=(Bα1:3c+K1:3)α4c+{circumflex over (F)}4 Equation (29)
(i.e. image {circumflex over (F)}4 layered on top of intersected images {circumflex over (F)}1, {circumflex over (F)}2, and {circumflex over (F)}3).
O
pvw=(Bα4c+{circumflex over (F)}4)α1:3c+K1:3 Equation (30)
(i.e. a swap such that intersected images {circumflex over (F)}1, {circumflex over (F)}2, and {circumflex over (F)}3 are layered on top of {circumflex over (F)}4).
To obtain image {circumflex over (F)}4 intersected with intersected images {circumflex over (F)}1, {circumflex over (F)}2, and {circumflex over (F)}3, a mix through a wipe pattern w1:4 is applied such that
where K1:4=({circumflex over (F)}4α1:3c+K1:3)w1:4+(K1:3α4c+{circumflex over (F)}4)w1:4, and α1:4c=α1:3cα4c=α1cα2cα3cα4c.
Producing the intersection effect between video images {circumflex over (F)}1 to {circumflex over (F)}4 in the manner explained above may be considered a cascade implementation in that images {circumflex over (F)}1 and {circumflex over (F)}2 are intersected, then image {circumflex over (F)}3 is added (i.e. intersected with already intersected images {circumflex over (F)}1 and {circumflex over (F)}2), and then image {circumflex over (F)}4 is added (i.e. intersected with already intersected images {circumflex over (F)}1, {circumflex over (F)}2, and {circumflex over (F)}3). Specific example hardware for such an implementation is described later.
It will be appreciated that whether using the parallel implementation described above (i.e. with the final intersection effect produced via Equation (25)) or whether using the cascade implementation described above (i.e. with the final intersection effect produced via Equation (31)), the end result should be the same: images {circumflex over (F)}1, {circumflex over (F)}2, {circumflex over (F)}3, and {circumflex over (F)}4 intersected over a background B, e.g., as shown in
Dynamic Pairing to have Images of Adjacent Depths Intersect
In the examples described above in which images {circumflex over (F)}1, {circumflex over (F)}2, {circumflex over (F)}3, and {circumflex over (F)}4 are intersected, it may be the case that visual mistakes might occur depending upon the order in which the different layers are intersected. To help understand this, consider
Now, if composite image 404 in
The possible visual error discussed above with reference to
With reference to
The sorter 502 comprises three levels (which may also be called stages or layers): level 0 (which is indicated by reference character 504), level 1 (which is indicated by reference character 506), and level 2 (which is indicated by reference character 508). Level 0 comprises comparators 510, 512, 514, and 516, and multiplexers 518, 520, 522, and 524. Level 1 comprises shift register banks 532, 534, 536, and 538, inverters 542 and 544, comparators 550 and 552, and multiplexers 554 and 556. Level 2 comprises shift register banks 560 and 562, inverter 570, comparator 572, and multiplexer 574. Shift register bank 580 receives and holds the output of level 2.
As can be seen in
Each comparator receives two inputs (generally designated “A” and “B” in
Each multiplexer receives two inputs (generally designated as “T” and “F” in
Each inverter is a logical inverter (e.g. a NOT Gate), which inverts the output signal of a comparator to indicate the opposite (e.g. if the comparator output is “T” (truth, which may be represented by the number “1” for example) it is inverted by the inverter to “F” (false—e.g. “0”), and vice versa). The logical inverters are used so that in the merging process only one of two banks of shift registers are shifted, the one shifted being chosen by the output of the comparator.
Each bank of shift registers can shift data to the left each time a shift enable (“en” in
In operation, the incoming eight video images are (perhaps arbitrarily) paired in index orders: {F1, F2}, {F3, F4}, {F5, F6}, and {F7, F8}, with respective depth orders {Z1, Z2}, {Z3, Z4}, {Z5, Z6}, and {Z7, Z8}. At level zero, each pair is sorted within itself and output into shift register pairs in level 1: {L11, L12}, {L13, L14}, {L15, L16}, and {L17, L18}, in order such that for each bank of shift registers 532, 534, 536, and 538, the larger magnitude depth value is in the front (i.e. on the left most side of the shift register bank). As an example to help visualize this, let us say, for example, that Z1=5, Z2=−3, Z3=−4, and Z4=−6. Then comparator 510 would indicate that Z1 is a larger magnitude depth than Z2 (i.e. Z1 is a larger negative number than Z2), and Z1 would be chosen by multiplexer 518 to be shifted into L12 of the shift register bank 532. Subsequently, the value Z1=−5 would be shifted into L11 of the shift register bank 532 and the depth value Z2=−3 shifted into L12. Similarly, comparator 512 would indicate that Z4=−6 has a larger magnitude depth than Z3=−4 (that is, Z4 is a larger negative number), and Z4 would be chosen by multiplexer 520 to be shifted into L14 of the shift register bank 534. Subsequently, the value Z4 would be shifted into L13 of the shift register bank 534 and the depth value Z3=−4 shifted into L14. A similar process would occur for Z5, Z6, Z7, and Z8.
At level 1, two neighboring pairs are grouped together for comparison. The larger magnitude depth value of a pair (i.e. the greatest negative number) is output into the shift registers of level 2 and shifted to the left so that the bank of shift registers 560 and the bank of shift registers 562 each hold depth values from largest magnitude to smallest magnitude, with the largest magnitude depth value on the left and the smallest magnitude depth value on the right. The shifting process is continued until all data in the registers of level 1 are shifted into level 2. In this way, in level 1, two pairs of data are sorted and merged into one quad of data. The inverters 542 and 544 ensure that the correct depth values are compared for the merge. To help visualize this, let us return to the example in the paragraph above in which Z1=−5, Z2=−3, Z3=−4, and Z4=−6. At level 1, register L11=Z1=−5, register L12=Z2=−3, register L13=Z4=−6 and register L14=Z3=−4. The comparator 550 would first select the largest magnitude of the two depth values in registers L11 and L13, which is Z4=−6, and this would be shifted into register L24 in the bank 560 of shift registers. Subsequently, since the larger magnitude depth value was from the bank 534 of shift registers, this bank would shift to the left and not the bank 532 of shift registers (due to the inverter 542). This would then cause Z3=−4 to be compared to Z1=−5, and Z1=−5 would be chosen by the multiplexer 554 since it has the largest magnitude depth value. Subsequently, since the larger magnitude depth value was from the bank 532 of shift registers, this bank would shift to the left and not the bank 534 of shift registers (due to the inverter 542). The value Z2=−3 would therefore be compared to Z3=−4. The comparison and shifting would continue until the four values shifted into bank 560 of shift registers were: L21=Z4=−6, L22=Z1=−5, L23=Z3=−4, and L24=Z2=−3.
At level 2, the values in the two banks of shift registers 560 and 562 are further grouped together for comparison and merged in the same manner described above with respect to level 1 to result in the eight depth values in bank 580 of shift registers in order from greatest magnitude depth (the left most register) to smallest magnitude depth (the right most register).
Note that as each value is shifted to the left in the register banks 532, 534, 536, 538, 560, and 562, if there is no subsequent value to replace the value shifted out of a right-hand register, a predetermined large magnitude number may be placed in that right-hand register. For example, using the example above, once Z1=−5 is shifted into layer 2 and Z2=−3 shifted into L11, a predetermined large magnitude number (out of the depth range) may be placed in register L12 to ensure continued correct sorting.
The sorting described above occurs on a pixel-by-pixel basis (since each pixel location has (in general) new associated depth values).
Since each level of the sorter 502 merges two sub-branches to form an input for the following level, the calculation speed may be improved (e.g. doubled) compared to a more straight-forward sorting method (e.g. using brute force).
As mentioned above, merge sort is only one example sort method that may be used. Other sorting method may be used instead. Examples include quick sort, insertion sort, bubble sort, and selection sort.
The image depth sorting discussed above may address a visual error problem (e.g. described above with reference to
Aiming to Provide Uniform Softness Around an Intersection Line
A method is presented above for providing an intersection effect using a wipe-based mix. For example, using the mix tree notation of Opgm and Opvw described above, if it is desired to intersect video images {circumflex over (F)}1 and {circumflex over (F)}2, then let Opgm=(Bα1c+{circumflex over (F)}1)α2c+{circumflex over (F)}2 (i.e. {circumflex over (F)}2 layered on top of {circumflex over (F)}1), and Opvw=(Bα2c+{circumflex over (F)}2)α1c+{circumflex over (F)}1 (i.e. {circumflex over (F)}1 and {circumflex over (F)}2 swapped so that {circumflex over (F)}1 is layered is on top of {circumflex over (F)}2), and then the intersection of the two is Equation (14) above:
where w1,2 is the wipe pattern, and where Omix will be referred to as the “target image”.
More generally, for a wipe pattern w to produce an intersection between images Opgm and Opvw via an A/B mix Omix=(1−w)Opvw+wOpgm in the manner described above, the wipe pattern may, for each pixel location (i.e. each pixel coordinate) in the target image (Omix):
(i) choose the pixel value of image Opgm at the pixel location when the depth value Zpvw of Opvw at the pixel location exceeds the depth value Zpgm of Opgm (i.e. when Opgm appears closer to the viewer at that pixel location); and
(ii) choose the pixel value of image Opvw at the pixel location when the depth value Zpgm of Opgm at the pixel location exceeds the depth value Zpvw of Opvw (i.e. when appears closer to the viewer at that pixel location).
An example of a wipe pattern to implement this is:
However, with such a wipe pattern there is a hard switch (or “swap”) on the intersection line (ΔZ=Zpgm−Zprv=0). This may result in visible jagged or stepped edges along the intersection line, which may be further exacerbated by depth accuracy errors due to calculation truncation or calculation rounding. This was discussed earlier with reference to
One method to try to smooth a jagged or stepped intersection line is to apply “fuzziness” or “softness” around the intersection line, for example, by implementing the following wipe pattern:
where ΔZ=Zpgm−Zprv and where Δf is a number representing the softness range. In this wipe pattern, when the absolute value of the depth difference between the images is less than Δf (i.e. when |ΔZ|<Δf, which occurs on and around the intersection line), then the pixel value from both images contribute to the displayed pixel value. For example:
(1) At pixel locations at which both images are equally far away from the viewer, then |ΔZ|=0 and thus w=0.5, such that both video images contribute to the displayed pixel value equally.
(2) At pixel locations at which Opgm is slightly closer to the viewer than Opvw, then 0<ΔZ<Δf, and thus 0.5<w<1. This means that both video images still contribute the displayed pixel value, but Opgm contributes more than Opvw.
(3) At pixel locations at which Opgm is slightly farther from the viewer than Opvw, then −Δf<ΔZ<0, and thus 0<w<0.5. This means that both video images still contribute the displayed pixel value, but Opvw contributes more than Opgm.
(4) At pixel locations at which Opgm is significantly closer to the viewer than Opvw (e.g. not immediately adjacent to the intersection line), then ΔZ≧Δf, and thus w=1 such that only Opgm is displayed.
(5) At pixel locations at which Opgm is significantly farther away from the viewer than Opvw (e.g. not immediately adjacent to the intersection line), then ΔZ≦−Δf, and thus w=0 such that only Opvw is displayed.
It will be appreciated that the larger the value of Δf chosen, the greater the “softness” range around the intersection line. In some embodiments, the value of Δf may be chosen to be a constant number (i.e. does not change spatially along the intersection line). However, this may cause non-uniform softness along the intersection line when there is a perspective projection effect in which the perspective transform emphasizes the difference between the two ends of the intersection line. To help understand this, please consider
Embodiments are described below in which a wipe pattern is constructed that may provide uniform softness spatially along the intersection line.
With reference to
In this embodiment, it is desired to have a uniform softness around the intersection line 702, as shown at 704, to improve the aesthetic quality of the intersection. In this embodiment it will be assumed that the wipe pattern to produce the intersection pattern is Equation (33), i.e.:
but the softness range Δf is spatially varied (i.e. a function of the target pixel coordinate (xt,yt)) with the aim of producing a uniform softness for the pixels neighbouring the intersection line 702 (in the softness range 704).
Consider a small square pixel area X on the screen having dimensions dxt=dyt. Each corner of the square pixel area X is projected into 3-D space on either image A or image B as shown in
A small change in depth Z, i.e. dz, may be expressed in terms of the total differential:
where dxt is the small change in the x direction in the image 700, and where dyt is the small change in the y direction of the image 700.
Assume depth Z in the image on the screen 700 is computed on a pixel-by-pixel basis as described earlier via Equation (8):
then applying Equation (34) to the Equation above (Equation (8)) produces
where ΣP=Paxt+Pb yt+Pcf. Note that ΣP is an intermediate calculation for computing unified depth
The softness range Δf may be considered the absolute value of the small change in depth z, i.e.,
Δf=|d
By setting the small change in the x direction dxt to be one pixel, i.e. dxt=1, and by setting the small change in the y direction dyt to be one pixel, i.e. dyt=1, then the small pixel area X in the target image is one pixel and Equation (36) reduces to
Δf=|Pd(Pa+Pb)|/(ΣP)2 Equation (37)
More generally, assuming a small change in the x direction equal to τ and also assuming a small change in the y direction equal to τ (i.e. dxt=dyt=τ), then Equation (35) reduces to
Note that τ is a positive number, which is why it may be factored out of the absolute value brackets in Equation (38). In some embodiments, τ may be a user-controllable parameter, e.g. controlled through a user interface, to allow the user to control how much softness is around the intersection line 702 (i.e. how big softness area 704 is, which provides a visual effect).
Therefore, considering the softness around the intersection line to be
from Equation (33), then substitute in
(from Equation (38)) to obtain
where
The value G may be considered a gain value. It is dependent upon the parameters defining the transformation (and specifically in this embodiment Pd, Pa, and Pb) and the (possibly user-controllable) parameter τ.
The wipe pattern to use for the intersection effect is therefore (in this embodiment):
where wf=(G(ΣP)2Δ
When computing wf in implementation, note that:
(1) the term |Pd(Pa+Pb)| is constant for a given transformation and is therefore only modified when the transformation applied to the source image is modified, which in general may happen every video field or frame. These parameters Pd, Pa, and Pb are calculated as part of the transformation and so are generally pre-calculated prior to the intersection effect.
(2) As mentioned above, the term τ may be user-controllable to achieve more visual effects (specifically, to control the amount of softness around the intersection). Mathematically and physically, τ=1 may cover two cases: w=0˜0.5 if ΔZ is negative and w=0.5˜1 if ΔZ is positive. Therefore, when τ=1, the edges from one image to the other (i.e. the softness range) is two pixels wide, where one pixel produces a positive ΔZ and the other pixel produces a negative ΔZ. More generally, when a user sets a specific value for τ, the softness area around the intersection will be 2τ pixels wide.
More generally, from the discussion above in relation to Equations (39) and (40), it will be appreciated that a depth difference value ΔZ may be modified to obtain a wipe pattern value wf, where the depth difference value ΔZ is modified based on at least the pixel location (xt,yt) and at least some parameters defining the perspective transformation. A pixel value of the target digital image may be set as the pixel value of the first image at the pixel location when wf is greater than a first threshold (e.g. wf≧1), or the pixel value may be set as the pixel value of the second image at the pixel location when wf is less than a second threshold (e.g. wf≦10), or otherwise the pixel value may be set as the combination of the pixel value of the first image at the pixel location and the pixel value of the second image at the pixel location (e.g. the specific combination determined by the value of wf between 0 and 1). Modifying the depth difference value to obtain the wipe pattern value wf may involve multiplying the depth difference value by a gain value G dependent upon at least some parameters defining the perspective transformation, and another value (ΣP)2 dependent upon at least the target image coordinate values xt and yt of the pixel location and at least some parameters defining the perspective transformation. The gain value G may depend upon or include a user-controllable parameter.
Some specific hardware implementations will now be described. These are meant by way of illustration only.
Turning first to
The device 1000 comprises perspective transform processing systems 1002a to 1002n and an intersection effect system 1004. The letter “n” is used in this figure (and the other figures) to represent an arbitrary number. It will be appreciated that
The device 1000 receives as input n sources video images V1 to Vn, each having a respective associated alpha αV1 to αVn. Each source video image and respective associated alpha undergoes respective perspective transform processing. In particular, as illustrated in
More specifically, the source video image V1 comprises a 2-D grid of source image pixels, each source pixel having a source pixel coordinate (xt,yt) and associated source pixel value V1(xs,ys). The associated alpha αV1 also comprises a plurality of values, one associated alpha value αV1(xs,ys) for each source pixel coordinate (xs,ys). The pre-shaped target video image {circumflex over (F)}1 is also 2-D grid of image pixels, which will be referred to as target image pixels. Each target pixel in {circumflex over (F)}1 has a target pixel coordinate (xt,yt) and associated target pixel value {circumflex over (F)}1(xt,yt). The associated complementary alpha α1c also comprises a plurality of values, one alpha value α1c(xt,yt) for each target pixel coordinate (xt,yt). There is also a depth value {circumflex over (Z)}1(xt,yt) corresponding to each target pixel coordinate (xt,yt), which is representative of the depth of the target image at that target pixel coordinate, or in other words, how far the target image appears from the viewer at that pixel location of the target image. There is also an associated perspective parameter value ΣP1(xt,yt) corresponding to each target pixel coordinate (xt,yt), which in this embodiment is computed as per above: ΣP1(xt,yt)=Paxt+Pbyt+Pcf.
Perspective transform processing systems 1002b to 1002n work in a similar manner.
The intersection effect system 1004 receives as an input the output of each of the perspective transform processing systems 1002a to 1002n and generates an intersection effect comprising a single video image Vout displaying an intersection of pre-shaped target video images {circumflex over (F)}1 to {circumflex over (F)}n.
With reference to
In operation, the perspective transform processing system 1002 receives as its input the source video image V1 and associated alpha αV1 and stores them in the memory 1020. In some embodiments, the source video image V1 and/or alpha αV1 may be subject to pre-filtering (not shown). The raster-scan counter 1022 may be used to write the source video image V1 and associated alpha αV1 into the memory 1020 on a pixel-by-pixel basis.
A corresponding target image F1, representing a transformed version of source video image V1 (i.e. source video image V1 having undergone a 3-D perspective effect), and a corresponding transformed associated alpha αF1, are then generated. The corresponding target image F1 comprises a 2-D grid of target image pixels, each target image pixel having a target pixel coordinate (xt,yt) and associated target pixel value F1(xt,yt). The transformed associated alpha αF1 also comprises a plurality of values, one alpha value {circumflex over (α)}V1(xt,yt) for each target pixel coordinate (xt,yt).
The target image F1 and corresponding transformed associated alpha αF1 are generated on a pixel-by-pixel basis using an inverse mapping function as follows. The raster-scan counter 1022 begins at (xt,yt)=(0,0) and counts up to (xt,yt)=(m,n) (assuming the target image F1 comprises an (m+1)×(n+1) grid of pixels). For each target pixel coordinate (xt,yt), the target coordinate values xt and yt are input into the perspective transform engine 1024, which computes the corresponding source pixel coordinate (xs,ys) of the source video image V1 stored in the memory 1020. The source pixel coordinate (xs,ys) corresponding to the target pixel coordinate (xt,yt) is a function of the manipulation being applied by the digital video effects and is defined by parameters 1034. In this embodiment, to obtain the corresponding x-coordinate of the source pixel (i.e. xs) for a given target pixel coordinate (xt,yt), the following computation is performed by the perspective transform engine 1024:
To obtain the corresponding y-coordinate of the source pixel (i.e. ys) for a given target pixel (xt,yt), the following computation is performed by the perspective transform engine 1024:
As discussed earlier, Equations (5) and (6) are based on mapping of a target pixel coordinate to a corresponding source pixel coordinate using the following inverse mapping function:
In this example, Ta, Tb, Tc, Td, Te, Tf, Tg, Th, Pa, Pb, Pc, Pd, and f are the parameters 1034, and their specific values are computed based on the specific effect requested by the user. For example, if a user indicated via a user interface (not shown) that it was desired to rotate the source video image V1 and by 15 degrees and shrink the source video image V1 and to half its size, then the user interface would electronically forward this request to the perspective transform processing system 1002a, which would compute the corresponding values of the parameters 1034, and forward these specific parameter values 1034 to the perspective transform engine 1024.
In general, the target pixel value F1(xt,yt) for the target image F1 will be set as the corresponding source pixel value V1(xs,ys), which is stored in and retrieved from memory 1020. However, the corresponding source pixel coordinate (xs,ys) computed by the perspective transform engine 1024 for a given target pixel coordinate (xt,yt) will (in general) not be an integer, but will be a fractional value. For example, if (xt,yt)=(12, 14) is input into the perspective transform engine 1024, then the corresponding source pixel coordinate (xs,ys) may be, for example, (xs,ys)=(23.63,17.27). Therefore, to compute the target pixel value F1(xt,yt), interpolation is used. In one embodiment, bilinear interpolation is used as follows: assume a source pixel coordinate of (xs,ys)=(i+u,j+v) where (i,j) is the integer and (u,v) is the fraction. For example, in the example above of (xs,ys)=(23.63,17.27), then i=23, u=0.63, j=17, and v=0.27. The source pixel values neighbouring (xs,ys)=(i+u,j+v) are therefore V1(i,j), V1(i+1,j), V1(i,j+1), and V1(i+1,j+1). The target pixel value F1(xt,yt) is then computed as F1(xt,yt)=Yj×(1−v)+Yj+1×v, where Y1=(i,j)×(1−u)+V1(i+1,j)×u and where Yj+1=V1(i,j+1)×(1−u)+V1(i+1,j+1)×u. To perform interpolation of this nature, the perspective transform processing system includes the interpolator 1026. It will be appreciated that another type of interpolation could be performed instead (e.g. a higher order of interpolation that uses a higher order of polynomial in the x-axis and y-axis).
The associated alpha αV1 is subject to the same perspective transform as the source video image V1. In the example implementation shown in
As shown in
The process above is repeated for each target image coordinate (xt,yt) such that the target image F1 and associated transformed alpha αF1 is created on a pixel-by-pixel basis.
The target image F1 and associated transformed alpha αF1 are fed to the pre-shaper, which operates in the manner described later.
As shown in
The perspective transform engine 1024 further outputs associated depth values
and the associated perspective parameter values are computed as: ΣP1(xt,yt)=Paxt+Pbyt+Pcf. An example of circuitry to compute the associated depth values and perspective parameter values is explained later.
The perspective transform processing system 1002a also includes the shaping alpha generator 1030, which computes the shaping alpha αshp for the target image F1 in the manner described in more detail below. The shaping alpha αshp comprises a set of shaping alpha values {αshp(xt,yt)}, i.e. a set comprising one shaping alpha value αshp(xt,yt) for each pixel coordinate (xt,yt) in the target image F1. As shown in
The perspective transform processing system 1002a further includes the pre-shaper 1032, which receives as its input the shaping alpha αshp, the target image F1, the associated transformed alpha αF1, and the associated depth values
As discussed above, the parameters 1034 include Ta, Tb, Tc, Td, Te, Tf, Tg, Th, Pa, Pb, Pc, Pd, and f. These are computed in real-time before the target counts (of the raster scanner 1022) starts. These parameters may be re-calculated per video field or frame, repeatedly, or at least every time the effect being applied is modified.
In operation of the perspective transform engine 1024, the parameters Ta, Tb Tc, and f are combined with target image coordinates xt and yt to compute ΣX=Taxt+Tbyt+Tcf, as per computational block 1102. The parameters Te, Tf, Tg, and f are combined with target image coordinates xt and yt to compute ΣY=Text+Tfyt+Tgf, as per computational block 1104. The parameters Pa, Pb, Pc, and f are combined with target image coordinates xt and yt to compute ΣP1=Paxt+Pbyt+Pcf, as per computational block 1106. The parameter Pd is divided by ΣP=Paxt+Pbyt+Pcf to generate
as per divider 1108.
This value is the associated depth, as per Equation (8),
The value
is then multiplied with ΣX=Taxt+Tbyt+Tcf at multiplier 1110 and this result subsequently added to parameter Td at adder 1112 to generate
(i.e. Equation (6)). Similarly, the value
is multiplied with ΣY=Text+Tfyt+Tgf at multiplier 1114 and this result subsequently added to parameter Th at adder 1116 to generate
(i.e. Equation (6)).
In the illustrated embodiment, the perspective transform engine 1024 additionally includes overflow/underflow circuitry 1118, which detects any overflow or underflow or data out-of-range conditions when the computations of the perspective transform engine 1024 are performed and indicates this via output signal “ovfl_unfl”. An example of overflow/underflow conditions that may be checked are as follows: (1) determine whether ΣP1=Paxt+Pbyt+Pcf is equal to zero, and if it is equal to zero, then indicate an overflow/underflow condition; and/or (2) determine whether
is less than zero, and if it is less than zero, then indicate an overflow/underflow condition; and/or (3) check whether the image coordinates xs and ys are out of bounds, and if one or both of them are out of bounds then indicate an overflow/underflow condition.
The shaping alpha generator 1030 and pre-shaper 1032 are also illustrated in
for example, using a comparator to evaluate whether α′L(xs,ys) is less than zero, between zero or one, or greater than zero. The softness range gain computational circuitry 1212 determines or computes a gain value GL. The specific circuitry of softness range gain computational circuitry 1212 is implementation specific and dependent upon the shaping alpha softness effect desired, and the gain value GL may be constant or dynamic. For example, in one embodiment the softness range gain computational circuitry 1212 may compute the gain value GL as the absolute value of ΣP1 multiplied by a scaling parameter S, in which case the softness range gain computational circuitry 1212 would comprise digital logic for determining the absolute value of ΣP1, followed by a multiplier for multiplying the absolute value of ΣP1 with the scaling parameter S. This method uses the intermediary value ΣP1, and it is assumed that a method such as this is used in the
The shaping alpha generator 1030 further includes fuzzy logic AND blocks 1220, 1224, and 1222, as well as multiplexer 1226. The fuzzy logic AND blocks may be implemented using computational circuitry configured to implement the fuzzy logic AND operations described herein (e.g. as described in more detail below with respect to
The shaping alpha generator 1030 of
The source image V1 is defined by four boundaries: left, right, top, and bottom. More generally, given that cropping can be applied to the source image V1, the source image can more generally be considered to be defined by four cropping lines (left, right, top, and bottom), which will be designated as left cropping boundary line CL, right cropping boundary line CR, top cropping boundary line CT, and bottom cropping boundary line CB. If there is no cropping, then these cropping lines are just the boundary lines. A cropping line can be denoted as:
Therefore, the four cropping values CL, CR, CT, and CB are known as they define the boundaries of the source image V1.
The shaping alpha defines the softness around these four cropping boundaries (i.e. the border of the source image). The amount of “softness” (i.e. the value of the shaping alpha) may be determined based on a distance of the inverse mapped source pixel (i.e. (xs,ys) in
In some embodiments, the cropping boundaries may be defined in higher precision (e.g. 6-bit fraction) to achieve sub-pixel movement. That is, sub-pixels (fractions of pixel coordinates) may be used in some embodiments, and in such embodiments the cropping boundaries may therefore be defined on a higher precision basis. Any movement of a cropping boundary on a sub-pixel basis may have a smoother look to a viewer.
Each processing path may compute an initial shaping alpha value as:
α′i=GiΔDi+α0, Equation (43)
Note that in this context α′i is referring to an initial shaping alpha value, not to a complementary alpha value. The α0 value is referred to as an “initial offset value”. It is optional and is discussed later.
Thus, an initial shaping alpha value α′i be computed for each border (boundary) of the source image, each initial shaping alpha value α′i representing the distance of the (inversed mapped) source pixel coordinate from that border, the distance being measured along the coordinate axis perpendicular to the border.
The initial shaping alpha value α′i may be truncated or clipped to unit range:
to obtain boundary shaping alpha value αi(xs,ys), and then each boundary shaping alpha value αi(xs,ys) for i=L, R, T, B may be combined using a fuzzy logic AND:
to result in shaping alpha value αshp(xt,yt)
The computations above describe one way to compute the shaping alpha αshp, and in particular these are the computations performed by the shaping alpha generator 1030 illustrated in
Right alpha processing path 1204, top alpha processing path 1206, and bottom alpha processing path 1208 are of the same structure as left alpha processing path 1204, except that different parameter inputs are used to respectively compute right boundary shaping alpha value αR(xs,ys), top boundary shaping alpha value αT(xs,ys), and bottom boundary shaping alpha value αB(xs,ys) as per the equations above.
The shaping alpha generator 1030 then implements Equation (45) as follows. The left and right boundary shaping alpha values αL(xs,ys) and αR(xs,ys) are used as the set of inputs for a first fuzzy logic AND
as at block 1220, and the top and bottom boundary shaping alpha values αT(xs,ys) and αB(xs,ys) are used as the set of inputs for a second fuzzy logic AND
as at block 1222, and then these two results are used as the set of inputs for a third fuzzy logic AND, as at block 1224, to generate the shaping alpha value αshp(xt,yt) for the pixel coordinate (xt,yt) of the target image. This can be considered parallel or cascade merging of multiple binary operation fuzzy logic ANDS to construct a fuzzy logic AND having more than 2 operands.
Then, as shown at 1226, the shaping alpha value αshp(xt,yt) is multiplexed with the value “0”, with the overflow/underflow choosing “0” (that is, overflow/underflow=“true”) if there was an overflow/underflow error in the computations. This multiplexer implements a switch that can switch between “0” and the shaping alpha value αshp(xt,yt). By choosing “0”, this results in the pixel being fully transparent if the overflow/underflow condition becomes true.
One example way to implement the fuzzy logic AND operations is illustrated in
αshp(xt,yt)=min{αL(xs,ys),αR(xs,ys),αT(xs,ys),αB(xs,ys)}. Equation (46)
This is considered a fuzzy logic implementation using an arithmetic minimum.
Another example way to implement the fuzzy logic AND operations is illustrated in
αshp(xt,yt)=αL(xs,xt)×αR(xs,xt)×αT(xs,xt)×αB(xs,ys). Equation (47)
This is considered a fuzzy logic implementation using arithmetic multiplication.
The
As explained above with reference to Equation (43), each processing path may compute an initial shaping alpha value α′i as α′i=GiΔDi+α0, where α0 is an initial offset value. The initial offset value α0 is optional, but if used, it takes on a value of between 0 and 1. In some embodiments, the exact value of the initial offset value α0 may be controlled by the user, e.g., through a user interface (not shown). The initial offset value α0 is an effect. Specifically, it geometrically shifts where the softness area of the shaping alpha is located relative to the boundary of the target object. The higher the initial offset value α0, the more outwardly extended the softness area is. It will be appreciated that in some embodiments, a different initial offset value α0 may be chosen for each boundary, if desired.
Turning now to
The video modifier 1304 comprises multipliers 1310 and 1312, as well as a subtractor 1314. The multipliers 1310 and 1312 each may be implemented, for example, by digital logic for multiplying two inputs signal. The subtractor 1314 may be implemented, for example, by digital logic for subtracting two input signals.
The depth modifier 1032 modifies each associated unified depth value
In one implementation, a predetermined number is used to represent “infinity”.
Equation (48) is implemented by the depth modifier 1032 in operation as follows: the comparator 1306 compares the shaping alpha value αshp to zero for each target pixel coordinate. If αshp=0, then the multiplexer 1308 selects “infinity” as the depth value {circumflex over (Z)}1. Otherwise, the multiplexer 1308 selects the associated depth value
The video modifier 1304 performs a function that is different from the depth modifier 1302. Specifically, the video modifier 1304 pre-shapes the target video image F1 to produce pre-shaped target video image {circumflex over (F)}1, and the video modifier 1304 also generates associated complementary alpha α1c. In operation, the multiplier 1310 of the video modifier 1304 multiplies the transformed associated alpha αF1 with the shaping alpha αshp, and then this is multiplied by the target video image F1 at multiplier 1312 to produce pre-shaped target video image {circumflex over (F)}1. The complementary alpha α1c associated with the pre-shaped target video image {circumflex over (F)}1 is obtained by computing α1c=1−αF1αshp at subtractor 1314.
As alluded to earlier, if the original video source image V1 does not have an associated alpha αV1, then one may be generated in which each value of the associated alpha αV1 is set to 1. This will then allow for the generation of transformed associated alpha αF1 and ultimately pre-shaped target video image {circumflex over (F)}1 and complementary alpha α1c.
Note that pre-shaped target video image {circumflex over (F)}1 is called “pre-shaped” because this shaping occurs before any wipe-based mix occurs.
Example implementations for the intersection effect system 1004 of
Recall from the discussion of intersection effect above, if it is desired to intersect two transformed video images F1 and F2, then a wipe-based mix is performed via Equation (14) to obtain Omix=Bα1,2c+K1,2, where
K
1,2=({circumflex over (F)}2α1c+{circumflex over (F)}1)w1,2c+({circumflex over (F)}1α2c+{circumflex over (F)}2)w1,2. Equation (49)
K1,2 may be called a “key”, and in this case K1,2 represents a wipe between sources {circumflex over (F)}2α1c+{circumflex over (F)}1 and {circumflex over (F)}1α2c+{circumflex over (F)}2. More specifically, K1,2 represents the intersection of images {circumflex over (F)}1, and {circumflex over (F)}2 (since {circumflex over (F)}2α1c+{circumflex over (F)}1 is image {circumflex over (F)}1 layered on top of {circumflex over (F)}2, {circumflex over (F)}1α2c+{circumflex over (F)}2 is image {circumflex over (F)}2 layered on top of image {circumflex over (F)}1, and the wipe pattern chooses with one of {circumflex over (F)}2α1c+{circumflex over (F)}1 and {circumflex over (F)}1α2c+{circumflex over (F)}2 to display based on the difference in depth between {circumflex over (F)}1 and {circumflex over (F)}2).
More generally, for two transformed video images Fi and Fi+1, then the key Ki,i+1 is:
K
i,i+1=({circumflex over (F)}i+1αic+{circumflex over (F)}i)wi,i+1c+({circumflex over (F)}iαi+1c+{circumflex over (F)}i+1)wi,i+1. Equation (50)
Also recall from the discussion of intersection effect above that to obtain images {circumflex over (F)}3 and {circumflex over (F)}4 intersected and layered on top of intersected images {circumflex over (F)}1 and {circumflex over (F)}2, a wipe-based mix is performed via Equation (25) to obtain Omix=Bα1:4c+K1:4, where
K
1:4=(K3,4α1,2c+K1,2)w1:4c+(K1,2α3,4c+K3,4)w1:4. Equation (51)
The key K1:4 represents a wipe between previous keys K1,2 and K3,4.
It will be noted that Equations (50) and (51) have a similar form: in Equation (50) the input are source images (i.e. Fi and Fi+1), whereas in Equation (51) the images are keys (i.e. K1,2 and K3,4). Mixes among keys (e.g. Equation (51)) may be referred to as “inter-pair mixes”, whereas mixes among source videos (e.g. Equation (50)) may be called intra-pair mixes”. In both cases they always output a shaped key K.
It will also be noted from the discussion of intersection effect that:
α1,2c=α1cα2c, Equation (52)
and
α1:4c=α1,2cα3,4c. Equation (53)
Turning now to
The operation of the alpha processing path 1351 is as follows. An input complementary alpha αic is multiplied via multiplier 1354 with input complementary alpha αi+1c to obtain αi,i+1c=αicαi+1c. Therefore, the alpha processing path 1351 may compute Equation (52) or (53).
The video processing path 1352 may compute Equation (50) (if the input is {circumflex over (F)}i and {circumflex over (F)}i+1) or Equation (51) (if the input are keys Ki=K1,2 and Ki+1=K3,4) as follows. The complementary alpha αic is multiplied by {circumflex over (F)}i+1 (or Ki+1) at multiplier 1356 and the result is added to {circumflex over (F)}1 (or Ki) at adder 1364 to obtain {circumflex over (F)}1+1αic+{circumflex over (F)}1 (or Ki+1αic+Ki). The complementary alpha αi+1c is multiplied by {circumflex over (F)}1 (or Ki) at multiplier 1358 and the result added to {circumflex over (F)}1+1 (or Ki) at adder 1366 to obtain {circumflex over (F)}iαi+1c+{circumflex over (F)}i+1 (or Kiαi+1c+Ki+1). The wipe value wi,i+1 is received at subtractor 1370, which subtracts wi,i+1 from the value one to obtain wi,i+1c=1−wi,i+1. The output of adder 1364 is then multiplied by wi,i+1c at multiplier 1360, and the output of adder 1370 is multiplied by wi,i+1 at multiplier 1362, and the outputs of multipliers 1360 and 1362 added together via adder 1368 to obtain Equation (50): Ki,j+1=({circumflex over (F)}i+1αic+{circumflex over (F)}i)wi,i+1c+({circumflex over (F)}iαi+1c+{circumflex over (F)}i+1)wi,i+1, or (if the input were keys Ki=K1,2 and Ki+1=K3,4 and complementary alpha values αic=α1,2c and αi+1c=α3,4c) to obtain Equation (51).
The video mixer 1350 may be considered to be blending the two input video images {circumflex over (F)}i and {circumflex over (F)}i+1 (or Ki and Ki+1) using wipe-based pattern wi,i+1.
An example implementation for producing the wipe pattern of Equation (40) will now be described.
Recall that the wipe pattern of Equation (40) is:
where
w
f=(G(ΣP)2ΔZ+1)/2, Equation (54)
and where
The wipe pattern w is constructed based on a difference in depth between two images, which will be referred to as “image A” and “image B”. The values of the wipe pattern w range from 0 to 1, where for a given pixel location in the target image, a value of w=0 indicates that the target pixel value at the target pixel location is the pixel value from a composite image A+B (i.e. A layered over B), a value of w=1 indicates that the target pixel value at the target pixel location is the pixel value from composite image B+A (i.e. B layered over A), and values between 0 and 1 introduce “softness” into the target display by having the target pixel value be a combination of the pixel value from composite image A+B and the pixel value from composite image B+A.
Turning now to
As shown in
In operation, the comparator 1406 compares the depth value {circumflex over (Z)}A of image A and the depth value {circumflex over (Z)}B of image B and the smaller of the two depth values is chosen by multiplexer 1414 and output as depth value {circumflex over (Z)}X. Note that these unified depth values are non-negative numbers. Otherwise, the underflow/overflow flag would be “true”, indicating an underflow/overflow condition. The signal from the comparator 1406 is also used as the selection signal for multiplexers 1416 and 1418 to respectively select the gain value GX and perspective parameter value ΣPX corresponding to the image with the smallest magnitude depth. The perspective parameter value ΣPX is multiplied with itself at multiplier 1412 to obtain (ΣPX)2, and then (ΣPX)2 is multiplied with the gain value GX at multiplier 1410 to obtain GX (ΣPX)2. The subtractor 1404 subtracts depth value {circumflex over (Z)}B of image B from depth value {circumflex over (Z)}A of image A to obtain a difference in depth Δ
The output multiplier 1413, i.e. wf, is then fed to a clipper or truncator 1422, which implements Equation (40), i.e.:
The clipper 1422 may be implemented, for example, using one or more comparators and/or multiplexers to compare the value wf to 1 and 0 and output the appropriate value depending upon whether wf≦0 or wf≧0 or 0<wf<1.
In the implementation described above with respect to
Turning now to
Example parallel mix structures are described in U.S. patent application Ser. No. 14/516,596, filed Oct. 17, 2014, entitled “PARALLEL VIDEO EFFECTS, MIX TREES, AND RELATED METHODS”, and incorporated herein by reference.
The structure of each wipe-based mixer is described below.
The notation “Vi” (i.e. “V1”, “V2”, . . . “V16”) represent a set of values. Specifically, the notation “Vi” represents the output of a perspective transform processing system in
where wf=(G(ΣP)2Δ
The other wipe-based mixers in
Turning now to
A possible drawback of the cascade implementation of
Recall that, as discussed earlier with reference to
Results of Experiments
Systems and methods in accordance with embodiments of the present disclosure were tested to perform an intersection effect, and specifically (in one testing example), the intersection of four images. This is discussed below in relation to
(1) “Wipe 1,2”, which is the wipe pattern for intersecting images 1 and 2. For this particular wipe pattern there is no line of intersection because the image 2 always has a depth that exceeds the depth of image 1 (i.e. image 1 always appears closer to the viewer than image 2).
(2) “Wipe 3,4”, which is the wipe pattern for intersecting images 3 and 4.
(3) “Wipe 1:4”, which the wipe pattern for intersecting K1,2 (i.e. the intersection of images 1 and 2) and K3,4 (i.e. the intersection of images 3 and 4).
That is, images 1 and 2 are paired (Wipe 1,2), images 3 and 4 are paired (Wipe 3,4), parallel to images 1 and 2, and then the two pairs are paired for the output.
The wipe patterns are generated based on Equation (40). The black in each wipe pattern in
As mentioned above, for Wipe 1,2 in
As can be seen from Wipe 3,4 in
As can be seen from Wipe 1:4 in
The implementations above assume a field programmable gate array (FPGA) or application specific integrated circuit (ASIC) implementation in that separate computation circuitry (e.g. adding/multiplication circuitry) is illustrated. In an alternative embodiment, the computations above, such as the computations performed by the perspective transform processing systems 1002a to 1002n and the intersection effect system 1004 may instead be performed by a general processor having the relevant parameters stored in memory and instructions stored in memory that, when executed, cause the processor to perform the computations. An example of such a computational device (e.g. a computer) is illustrated in
A possible benefit of a field programmable gate array (FPGA) or application specific integrated circuit (ASIC) implementation is that it may execute the computations faster, making it more desirable for real-time processing, such as when producing a live video (live production). The general processor implementation shown in
In the embodiments described above, it is assumed that the images are digital video images, e.g., each image a field or frame of digital video, where the digital video comprises a series of digital images displayed in rapid succession. In such embodiments, the intersection effect described above may, for example, be a DVE device feature in a DVE device embedded in a production switcher in a video application. However, the methods above may also be applied to a non-video environment. Specifically, in a non-video environment, one could perform the methods above with digital images (e.g. digital pictures) to perform an intersection effect for two or more digital images. This may be useful, for example, in a computer graphics application.
Various embodiments have been described above, including embodiments in which an intersection effect is created using a wipe-based mix. Since the intersection effect is created using the wipe-based mix in the manner described above, it may meet the “strong rule of seamless switching” discussed above and therefore be considered to be a high quality intersection effect. In some embodiments, a depth difference based wipe pattern that aims to provide uniform softness along an intersection line to achieve anti-alias effects is also described. In some embodiments, a video mixer and a wipe pattern generator are disclosed for performing the intersection effect. In some embodiments, additional system components are disclosed, such as a perspective transform engine, an alpha shaping generator (which may produce an anti-alias shaping alpha), and a pre-shaper. In some embodiments, different implementation structures for performing the intersection effect among multiple video layers are disclosed, including a parallel structure (with or without pixel-by-pixel depth based sorting for dynamic pairing of video layers to try to eliminate potential visual mistakes), and a cascade structure (with or without pixel-by-pixel depth based sorting for dynamic pairing of video layers to try to eliminate potential visual mistakes).
In some embodiments described above, the general layering process for multiple video sources and the rule of seamless switching are discussed and used as a starting point, and a wipe-based mix for intersection effect is derived, and then softness is introduced into a wipe pattern to mix intersected video layers. The wipe pattern of such embodiments is based upon depth differences, but dynamically modulated (i.e. scaled) by parameters from perspective transforms. In some embodiments, the wipe pattern(s) is/are constructed using image depth difference, and the wipe pattern(s) is applied to multiple video layers for a general wipe transition.
In some embodiments, a wipe-based mix between two video layers is provided that may meet the “strong rule of seamless switching” discussed above. The two video layers may each have an alpha component or channel for transparency (e.g. an “associated alpha”), as discussed earlier. A wipe pattern is also provided to mix the two layers in a way that may provide uniform softness (or “fuzziness”) along an intersection line on the target display screen. The wipe pattern may be produced from a depth difference between two video layers, the depth difference modified (or scaled or modulated) by perspective data dynamically, as described above.
In some embodiments, a parallel or cascade structure is disclosed to connect multiple wipe-based mixes. A dynamic pairing function may sort depths from all video layers in order (e.g. descending order). The sorting may be on a pixel-by-pixel basis and may be any effective sorting method. Merge sort, described earlier, is only one possible sorting function.
In some embodiments, the system for producing the intersection effect receives as data inputs perspective data parameters and depth data (computed as part of the perspective transformation), shaping alpha data, video data that was subject to the 3D manipulation (e.g. from an interpolation function in a perspective transform engine), and associated alpha data subject to the 3D manipulation (e.g. from an interpolation function in a perspective transform engine).
In some embodiments, the wipe-based mix circuitry may include a video path, which has as an input two video data paths plus a wipe pattern, and has as an output a composite of video data (e.g. video processing path 1352 in
In some embodiments, the circuitry for computing the wipe pattern may include two groups of inputs, each input including a perspective data parameter ΣP, pre-shaped depth value {circumflex over (z)}, and gain parameter G (e.g.
In some embodiments, the gain parameter G may be computed using parameters in a homogeneous matrix that are determined by a homogeneous matrix (e.g. parameters from Equation (7)) and a user-controllable parameter (e.g. Σ) to achieve a visual effect.
As described above with respect to
With regard to
Although the foregoing has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the scope of the claims appended hereto.