Analytical Motion Blur Rasterization With Compression

Abstract

A rasterizer, based on time-dependent edge equations, computes analytical visibility in order to render accurate motion blur. An oracle-based compression algorithm for the time intervals lowers the frame buffer requirements. High quality motion blurred scenes can be rendered using a rasterizer with rather low memory requirements. The resulting images may contain motion blur for both opaque and transparent objects.

Description

BACKGROUND

This relates to digital image processing and, particularly, depictions of motion blurring and compression techniques.

Motion blur is generated when the shutter of the camera is open for a finite time, and some relative motion appears inside the field of view of the camera. It is an effect that is important for offline rendering for feature films, since the frame rate is rather low (˜24 frames per second). With motion blur in the rendered images, apparent jerkiness in the animation can be reduced or removed entirely. However, motion blur is also becoming an important visual effect for real-time rendering, e.g., for games. In order to get good performance, various rather crude approximations, that may or may not apply in all cases, are used.

In general, motion blur rendering can be divided into two parts, namely, visibility determination and shading computations. Most solutions that converge to a correctly rendered image are based on point sampling. The more samples that are used the better image is obtained, and at the same time, the rendering cost goes up. In many cases, one can obtain reasonable results with rather few shader samples compared to the number of visibility samples. For example, RenderMan uses only a single shader sample for motion-blurred micro polygons.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a moving triangle projected in screen space;

FIG. 2 is a graph of visibility function multiplied by the depth function over time for the sample point shown in FIG. 1;

FIG. 3 is a three dimensional depiction with side surfaces of each edge rendered as a bilinear patch;

FIGS. 4 and 5 include depictions, at the top, of side surfaces rendered as patches with the corresponding depth function, plotted underneath;

FIG. 6 depicts interval insertion over four cases;

FIG. 7 depicts merging two depth functions with similar depth functions, showing results with and without compression;

FIG. 8 is a flow chart for one embodiment; and

FIG. 9 is a hardware depiction for one embodiment.

DETAILED DESCRIPTION

The motion blur visibility problem may be solved analytically in order to avoid sampling noise. One embodiment involves rasterization of motion-blurred triangles with analytical visibility.

Assume that the entire transform matrix, including projection, for a vertex is called M. A vertex in homogeneous clip space is then obtained as |p=M p, where p is the vertex in three-dimensional object space, and p is the resulting four-dimensional vertex in homogeneous clip space, i.e., before division by the w-component. To simplify the derivation, we use the following notation: {circumflex over (p)}=(p_x,p_y,p_w), which is a scaled and translated version of the point in camera space. This can be confirmed by looking at the definition of the projection matrix in OpenGL and DirectX.

The standard (no motion) edge function, in homogeneous form, through two vertices, say and {circumflex over (p)}⁰ and {circumflex over (p)}¹, is:

e(x, y,w)=({circumflex over (p)}¹×{circumflex over (p)}⁰)·(x, y, w)=ax+by+cw.   (1)

A sampling point, (x,y,w), is inside the triangle if e_i(x,y,w)≦0 for i∈{0,1,2}, i.e. for the three edges of the triangle. Next, this is extended with a time dimension.

Assume that the vertices move linearly from the beginning of a frame, at t=0, to the end for a frame, at t=1. At t=0, we denote the vertices as qⁱ, and we call them rⁱ at t=1. Since there is no bar nor a hat on the vertices, all qⁱ and rⁱ are in homogeneous clip space. A linearly interpolated vertex is then given as:

p ⁱ(t)=(1−t)qⁱ+trⁱ,   (2)

for a certain instant t∈[01]. The coefficients of a time-dependent edge equation are given by:

(a,b,c)=({circumflex over (p)}¹×{circumflex over (p)}⁰)=((1−t){circumflex over (q)}hu 1+t{circumflex over (r)}¹)×((1−t){circumflex over (q)}⁰+t{circumflex over (r)}⁰)=t²f+tg+h,   (3)

where:

h={circumflex over (q)} ¹ ×{circumflex over (q)} ⁰,

k={circumflex over (q)} ¹ +{circumflex over (r)} ⁰ +{circumflex over (r)} ¹ ×{circumflex over (q)} ⁰,

f=h−k+{circumflex over (r)} ¹ ×{circumflex over (r)} ⁰,

g=−2h+k.   (4)

Each edge equation is now a function of time consisting of three functions: (a(t), b(t), c(t)), where, for example, a(t)=f_xt²+g_xt+h_x. Finally, the entire time-dependent edge equation is:

e(x, y,t)=a(t)x+b(t)y+c(t),   (5)

where we have set w=1 since rasterization is done in screen space (x,y).

For now, we assume that each pixel has a single sample point at (x₀,y₀). Extensions to multi-sampling and super-sampling just increase the sampling rate. If we consider a particular pixel, then (x₀,y₀) are constant. In this case, the time-dependent edge function becomes a function of time, t, alone:

e(x₀,y₀,t)=e(t)=a(t)x₀+b(t)y₀+c(t).   (6)

This expression can be expanded using Equation 3:

$\begin{array}{cc}\begin{array}{c}e\left(t\right)=t^2\left(f_xx_0+f_yy_0+f_z\right)+t\left(g_xx_0+g_yy_0+g_2\right)+\\ \left(h_xx_0+h_yy_0+h_z\right)\\ =\alpha t^2+\beta t+\gamma ,\end{array}& \left(7\right)\end{array}$

where (α,β, γ) are constants for a certain sample point, (x₀,y₀). Hence, each edge equation is a quadratic function in t. Next, we introduce a binary inside-function, i(t), as:

$\begin{array}{cc}i\left(t\right)=\{\begin{array}{c}1,\mathrm{when}e\left(t\right)\le 0\\ 0,\mathrm{elsewhere},\end{array}& \left(8\right)\end{array}$

i.e, i(t)=1 for all t∈[0,1] when (x₀,y₀) is inside the corresponding time-dependent edge equation. Note that the inside functions, i_k(t), can be computed analytically by solving the second-degree polynomial in Equation 7.

For a moving triangle, we have three time-dependent edge functions, e_k(t), where k∈{0,1,2}. The point (x₀,y₀), will be inside the moving triangle when all inside functions are positive at the same time. This visibility function can be expressed as:

v(t)=i₀(t)i₁(t)i₂(t),   (9)

i.e., the multiplication of all three inside functions.

We derive the equation for the depth during the time span where the sample point, (x₀,y₀), is inside the moving triangle. Perspective-correct interpolation coordinates (u,v), can be used to interpolate any attribute per vertex. This is done as:

s(u,v)=(1−u−v)p⁰+up¹+vp²,   (10)

where p^k are the attribute vectors at the three vertices, and s(u,v) is the interpolated attribute vector. Edge equations, e_k, can be used to compute (u,v):

$\begin{array}{cc}\left(u,v\right)=\frac{1}{e_0+e_1+e_2}\left(e_1,e_2\right).& \left(11\right)\end{array}$

Note that u, v, and all e_k are functions of (x₀,y₀), but this was left out to shorten notation. Equation 11 also holds when time-dependent edge equations are used.

The depth buffer stores interpolated depth values. Assuming that p^k=(p_x^k,p_y^k,p_z^k,p_w^k), k∈{0,1,2}, are the triangle vertices in clip space (before division by w), one first uses Equation 10, and then computes the depth as d=s_z/s_w for a particular fragment with perspective-correct barycentric coordinates, (u,v).

When we turn from static triangles to moving triangles, p^k are functions of time (Equation 2), and so are the edge equations. Let us first take a look at one of the texture coordinates, u (see Equation 11):

$\begin{array}{cc}\begin{array}{c}u=\frac{e_1}{e_0+e_1+e_2}\\ =\frac{{\alpha }_1t^2+{\beta }_1t+{\gamma }_1}{\left({\alpha }_0+{\alpha }_1+{\alpha }_2\right)t^2+\left({\beta }_0+{\beta }_1+{\beta }_2\right)t+{\gamma }_0+{\gamma }_1+{\gamma }_2},\end{array}& \left(12\right)\end{array}$

where the three time-dependent edge equations are: α_kt²+β_kt+γ_k (Equation 7). The texture coordinate, u, becomes a rational polynomial of degree two in t. The major difference, when compared to static triangles, is when the entire depth function is put together,

$\begin{array}{cc}d\left(t\right)=\frac{s_z}{s_w}=\frac{\left(1-u-v\right)p_z^0+{\mathrm{up}}_z^1+{\mathrm{vp}}_z^2}{\left(1-u-v\right)p_w^0+{\mathrm{up}}_w^1+{\mathrm{vp}}_w^2},& \left(13\right)\end{array}$

where all pⁱ and p_wⁱ are functions of time according to Equation 2, and u and v are functions of time (Equation 12) as well. When these expressions replacing the corresponding terms in Equation 13, we arrive at a cubic rational polynomial for the depth function for a certain sample point, (x₀,y₀):

$\begin{array}{cc}d\left(t\right)=\frac{m_3t^3+m_2t^2+m_1t+m_0}{n_3t^3+n_2t^2+n_1t+n_0},& \left(14\right)\end{array}$

Two of the advantages of using d=s_z/s_w include the fact that the depth is in the range [0,1] due to the way the projection matrix is set up and that depth buffer compression can therefore be implemented efficiently since the depth will be linear over a triangle. Also, d=s_z can be used, which will generate the same images, but the depth will now range between the near and the far plane: [Z_near, Z_far]. This simplifies the depth function for moving triangles. It will still be a rational function in t with degree three in the numerator, but the degree in the denominator will be reduced to two, that is:

$\begin{array}{cc}d\left(t\right)=\frac{m_3t^3+m_2t^2+m_1t+m_0}{n_2t^2+n_1t+n_0},& \left(15\right)\end{array}$

In theory, the intersection of three inside functions of the visibility function can result in at most four disjoint time spans where the resulting function, v(t), is positive. This is because each inside function can consist of two disjoint positive parts. In practice, we have only encountered three intervals when you consider front-facing triangles for any value of t. Most commonly, only a single interval is generated for most triangles and samples, however.

The term “interval” denotes a range in the time dimension, together with the color and depth for that time range. An interval is denoted by Δ. In practice, the third-degree rational depth function (Equation 15), is approximated by a linear function. The motivation for this is that the depth function rarely varies much beyond such an approximation within a pixel, and it makes computations much faster. In addition, we have good experiences with this approximation.

Given the depth function approximation, each interval stores the following parameters:

t_i^s: time at the beginning of the interval

t_i^e: time at the end of the interval

z_i: depth at the beginning of the interval

k_i: slope of the depth function

c_i: color of the interval   (16)

Our interval is analogous to a fragment in rendering without motion blur, and an example of an interval is shown in FIGS. 4 and 5. In FIG. 4, the sample point 22 is inside the moving triangle in two disjoint time intervals, as shown by the depth function below. In FIG. 5, the sample point 24 is inside in three disjoint time intervals. In general, all intervals belonging to a pixel are simply stored in an interval list in that pixel, similar to how order-independent transparency is done with DX11 using graphics processing units. A couple of optimizations that can be used for opaque rendering will be described below. As a triangle that covers a pixel is rendered, one or more intervals are added to that pixel's interval list. When rendering starts, each pixel is initialized with an interval with background color and depth equal to the far plane.

Interval insertion is illustrated in FIG. 6. New intervals 26 are inserted into the existing interval list 28. At 30, clipping the interval 26 would result in the existing interval 28 and two intervals 26, one on each side of interval 28. At 32, for intersections, the new interval is also added to the pixel's interval list. At 34 and 36, optimizations valid only for opaque rendering with a LESS_EQUAL depth test are shown.

Our approach is based on trying to keep the number of intervals stored per pixel small, and to facilitate compression when possible. When two intervals intersect, one can use clipping to create non-overlapping (in time) intervals. However, that can generate up to four intervals, which is undesirable. An intersection can also generate only two intervals, but in such cases, we also refrain from clipping since our compression mechanism works better with unclipped intervals. Note that using non-clipped intervals requires a slightly more complex resolve procedure. For opaque rendering, simple depth test optimizations can be included in this process as well, and this is shown in the bottom two illustrations 34 and 36 of FIG. 6, where we have assumed that a LESS_EQUAL depth test is used. It is, however, straightforward to adapt to any depth function. In the second illustration from the bottom, the interval 10 is clipped in time since it is occluded and because the clipping only generate one interval, i.e., it does not require more storage than simply storing the unclipped interval.

Note that to facilitate depth testing, we keep the intervals sorted on t_sⁱ per pixel. This can be done during interval insertion using insertion sort, for example.

A rasterization with compression sequence, shown in FIG. 8, begins by rasterizing a triangle (block 42) and then the intervals are computed per sample or pixel (block 43). Then, using an oracle function and interval merging, temporal pixel compression may be done in some embodiments (block 44). If the image is done, the flow moves to the resolve pass. Otherwise, it iterates back to rasterizing another triangle in block 42.

After rendering all moving and non-moving triangles, as indicated in FIG. 8 at block 42, we have a list of possibly overlapping (in time) intervals, Δ_i, per pixel. Recall that we keep the intervals sorted, i.e., t_sⁱ≦t_sⁱ⁺¹,∀i, since this is part of the interval insertion step described above.

The resolve pass (blocks 46, 47 and 48) processes a pixel independently of other pixels, and sweeps the sorted intervals in a pixel from time t=0 to t=1. During the sweep, we maintain a list, called Active List, per pixel of all intervals overlapping the current time of the sweep, as indicated in block 46 of FIG. 8. For example, when the sweep starts, at t=0, the Active List is initialized to hold all intervals that overlap t=0. Note that the Active List is also kept sorted on depth. As an interval start-point, t_i^s, or end-point, t_i^e, or an intersection point between two intervals is encountered, the following action is taken:

• • Interval-Start: insertion and sort (on depth) new interval into Active List.
  • Interval-End: remove interval from Active List.
  • Interval-Intersection: swap places of the intersecting intervals in the Active List to maintain depth order.

Between each pair of encounters (the three cases above), the final color for that particular subspan in time is computed (FIG. 8, block 47). Each such color and subspan in time is temporarily put into a resolved interval, Δ_k. Since intersection points are handled as part of the sweep, there will not be any intervals overlapping in depth in the Active List. As a consequence of that, the color of Δ_k for opaque rendering is simply the color of the nearest (in depth) interval in the Active List, again assuming that a LESS_EQUAL depth test is used. Each resolved interval's color, c_k, is then integrated against the shutter response function, w(t), and added to the final color of the pixel. This can be expressed as:

$\begin{array}{cc}{c}_{\mathrm{final}}=\sum _{k=0}^{n-1}\left({\int }_{\begin{array}{c}t\\ k\end{array}}^{\begin{array}{c}e\\ k\end{array}}w\left(t\right)c_kt\right),& \left(17\right)\end{array}$

for n disjoint intervals. If a box filter is used, the colors of all intervals are simply combined into one final color weighted by their duration in time. For the transparent resolve procedure, the only difference is that the color, c_k, of the resolved interval, Δ_k, is computed by blending the intervals in the Active List in back-to-front order based on the alpha component of each color.

When there are many small triangles with a relatively high degree of motion blur, each pixel may need to store a large number of intervals, Δ_i, in order to exactly represent the color of the pixel. We have observed up to a few hundred intervals per pixel in extreme cases. This is clearly not desirable for a rasterization-based algorithm due to the extra pressure on the memory subsystem, that is, increased memory bandwidth usage. The problem can be alleviated by using a tiling architecture, where the triangles are sorted into tiles (rectangular regions of pixels) by front-end pass, and per-pixel rendering done in a back-end pass. Tiles can be back-end processed in parallel by separate cores, since the rasterization and per-pixel work is independent at this point. However, a tiling architecture cannot solve the problem. Instead, we use lossy compression of intervals.

Merging intervals are illustrated in the top part of FIG. 7. Since the intervals 38 have similar depth functions, they are merge candidates. Assume that a pixel can only “afford” to store four intervals, and that after rendering another motion blurred triangle, the pixel actually holds five intervals (top left in FIG. 7). To be able to fit this into our frame buffer, we will need to compress this information into four intervals again. This is shown in the top right part of the figure, where the two intervals 38 with similar depth functions have been merged to a single interval. A similar type of compression has been used for deep shadow maps, but in their context, compression was only needed to be done once after the entire visibility function was known. Our goal is different since we may need to compress each pixel several times as more and more triangles are rendered to a pixel and, in addition, we compress not only depth, but may also take into account the colors, the actual depth functions, the extent of the intervals, etc. The bottom left graph illustrates pixel content after rendering many small moving triangles. The bottom right graph is a possible result using our compression algorithm. Finally, all sub-spans are blended together into a final per-sample color (FIG. 8, block 48).

We use an oracle-based approach to attack this problem. Our oracle function is denoted:

o _i,j =O(Δ_i,Δ_j),   (18)

where the oracle function, O( ) operates on two intervals, Δ_i and Δ_j, where i<j. The task of the oracle is basically to compute an estimation on how appropriate it is to merge the two input intervals. Given an oracle function, O, we compute oracle function values, o_i,j, for all i, and j∈{i+1,i+2,i+3}. For transparent scenes with a high level of depth overlap, we have increased the search range up to +10, instead of +3. In the next step, the interval pair with the lowest o_i,j is merged. Depending on the implementation, this process may continue until the number of intervals per pixel falls in a desired range, or until there are no more appropriate merges possible. Next, we describe an oracle, for two intervals, that generates a lower value the more appropriate they are to merge.

Our oracle function, O, may be described, for example, by the following formula, where i<j:

0(Δ_i,Δ_j)=h₁ max(t_j^s−t_i^e,0)+h₂|z_i−z_j|+h₃|k_i−k_j|+h₄(t_i^e−t_i^s+t_j^e−t_j^s)+h₅(|c_i,r−c_j,r|+|c_i,g−c_j,g|+|c_i,b−c_j,b|).   (19)

The first term favors merging of intervals that are located close in time (even overlapping). The second term z_i is the depth at the end of Δ_i, i.e., z_i=z_ik_i(t_i^e−t_j^s), and z_j is the depth at the beginning of the other interval, Δ_j. See Equation 16 for definitions of the interval's parameters. The third term penalizes merging of intervals with different slopes. Hence, both the second and third terms attempt to detect if the depth functions are similar, and therefore, whether they are amenable for merging. The fourth term favors merging of short (in time) intervals, while the fifth favors merging of interval with similar colors. All h_i are used-specified constants.

We will describe the merging of two intervals, Δ_i and Δ_j, into a new interval, Δ′_i. The merge is described as: Δ′_i=merge(Δ_i,Δ_j), i<j, where the new parameters are:

t′_i^s=t_i^s

t′ _i ^e=max(t_i^e,t_j^e)

z′ _i=(1−α)z_i+α(z_j−k_j(t_j^st_i^s))

k′ _i=(1−α)k_i+αk_j

c′ _i=(1−α)c_i+αc_j,   (20)

where α=(t_j^e−t_j^s)/(t_i^e−t_i^s+t_j^e−t_i^s) is used to linearly blend parameters depending on the lengths (in time) of the intervals that are merged. As can be seen, the slope, k′_i of the depth function, and the color, c′_i, are simply linear interpolations of the input intervals' parameters. The depth, z′_i, is slightly more complex because we need to blend the depth at the same instant in time. Since we want the new depth at time t_i^s, we compute the depth of Δ_j's depth function at t_i^s and use that for blending. For future work, it would be interesting to investigate other approaches to depth merging, e.g., where the area under the depth function is kept constant after compression. An example of merging two intervals is shown in FIG. 7.

The computer system 130, shown in FIG. 9, may include a hard drive 134 and a removable medium 136, coupled by a bus 104 to a chipset core logic 110. A keyboard and mouse 120, or other conventional components, may be coupled to the chipset core logic via bus 108. The core logic may couple to the graphics processor 112, via a bus 105, and the main or host processor 100 in one embodiment. The graphics processor 112 may also be coupled by a bus 106 to a frame buffer 114. The frame buffer 114 may be coupled by a bus 107 to a display screen 118. In one embodiment, a graphics processor 112 may be a multi-threaded, multi-core parallel processor using single instruction multiple data (SIMD) architecture.

In the case of a software implementation, the pertinent code may be stored in any suitable semiconductor, magnetic, or optical memory, including the main memory 132 or any available memory within the graphics processor. Thus, in one embodiment, the code to perform the sequences of FIG. 8 may be stored in a machine or computer readable medium, such as the memory 132 or the graphics processor 112, and may be executed by the processor 100 or the graphics processor 112 in one embodiment.

FIG. 8 is a flow chart. In some embodiments, the sequences depicted in this flow chart may be implemented in hardware, software, or firmware. In a software embodiment, a non-transitory computer readable medium, such as a semiconductor memory, a magnetic memory, or an optical memory may be used to store instructions and may be executed by a processor to implement the sequences shown in FIG. 8.

The graphics processing techniques described herein may be implemented in various hardware architectures. For example, graphics functionality may be integrated within a chipset. Alternatively, a discrete graphics processor may be used. As still another embodiment, the graphics functions may be implemented by a general purpose processor, including a multicore processor.

References throughout this specification to “one embodiment” or “an embodiment” mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one implementation encompassed within the present invention. Thus, appearances of the phrase “one embodiment” or “in an embodiment” are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be instituted in other suitable forms other than the particular embodiment illustrated and all such forms may be encompassed within the claims of the present application.

While the present invention has been described with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover all such modifications and variations as fall within the true spirit and scope of this present invention.

Claims

1. A method of rasterizing a motion blurred triangle comprising: analytically determining an interval of overlap for a sample point for each of n time dependent edge equations; andidentifying a time interval where a moving planar polygon with n vertices is visible over the sample point as an intersection of the n intervals for the n time dependent edges.

2. The method of claim 1 including storing said intervals in a per sample list.

3. The method of claim 2 including using temporal pixel compression.

4. The method of claim 1 including computing a color of each interval and weighting, based on time duration and visibility, of all the intervals together to get a final sample color.

5. The method of claim 1 including reducing the number of intervals per pixel by merging two similar intervals as foreseen by an oracle function.

6. The method of claim 5 including using an oracle function to estimate whether to merge two intervals.

7. The method of claim 6 including using an oracle function that favors merging intervals that are close in time.

8. The method of claim 6 including using an oracle function that disfavors merging intervals with different slopes.

9. The method of claim 6 including using an oracle function that favors merging intervals with similar depth functions.

10. The method of claim 1 including storing an interval start and end time, a depth at a point in the interval, a slope of the depth function, and a color of the interval.

11. A non-transitory computer readable medium storing instructions to enable a computer to rasterize motion blurred triangles by: analytically determining an interval of overlap for a sample point for each of three time dependent edge equations; andidentifying a time interval where a moving triangle is visible over the sample point as an intersection of the three intervals for the three time dependent edges.

12. The medium of claim 11 further storing instructions to store said intervals in a per sample list.

13. The medium of claim 12 further storing instructions to use temporal pixel compression.

14. The medium of claim 13 further storing instructions to compress using an oracle function.

15. The medium of claim 13 further storing instructions to store on an interval start and end time, a depth at a point in the interval, a slope of the depth function, and a color of the interval.

16. The medium of claim 11 further storing instructions to use clipping to create a non-time overlapping interval where intervals intersect.

17. The medium of claim 16 further storing instructions to develop a first list of possible time overlapping intervals after rendering all triangles.

18. The medium of claim 17 further storing instructions to develop a second list of all time overlapping intervals when sweeping sorted intervals.

19. The medium of claim 18 further storing instructions to sort said second list based on depth.

20. The medium of claim 19 further storing instructions to perform insertion and sort based on depth in said second list upon detecting a new interval.

21. A method of rasterizing a motion blurred triangle comprising: rendering the triangles making up an image; andafter rendering the triangles, resolving the color of each pixel or sample.

22. The method of claim 21 including computing analytically an interval of overlap for a sample point for a time dependent edge.

23. The method of claim 22 including determining an interval of overlap region of three time dependent edge equations.

24. The method of claim 23 including determining an intersection of the three intervals.

25. The method of claim 24 including storing the time interval determined as the intersection of the three time intervals.

26. The method of claim 25 including using temporal pixel compression.

27. The method of claim 26 including compressing using an oracle function.

28. The method of claim 27 including storing an interval start and end time, a depth at a point in the interval, the slope of the depth function, and a color of the interval.

29. The method of claim 21 including using clipping to create a non-time overlapping interval where intervals intersect.

30. The method of claim 29 including after rendering all triangles, developing a first list of possible time overlapping intervals.