Patent Grant
RE42847
This application is a reissue of U.S. Pat. No. 6,191,797, which issued from application Ser. No. 08/861,063 filed May 21, 1997.
The present invention relates to the creation of computer-generated images both in the form of still pictures and video imagery, and, in particular, relates to a process, apparatus, and system for creating an image made up by compositing multiple components.
Computer generated images are typically made up of many differing components or graphical elements which are rendered and composited together to create a final image. In recent times, an “opacity channel” (also known as a “matte”, an “alpha channel”, or simply “opacity”) has been commonly used. The opacity channel contains information regarding the transparent nature of each element. The opacity channel is stored alongside each instance of a colour, so that, for example, a pixel-based image with opacity stores an opacity value as part of the representation of each pixel. An element without explicit opacity channel information is typically understood to be fully opaque within some defined bounds of the element, and assumed to be completely transparent outside those bounds.
An expression tree offers a systematic means of representation for rendering objects or elements of an image. Expression trees comprise a plurality of nodes including leaf nodes, internal nodes and a root node. A leaf node, being the outer most node of an expression tree, has no descendent nodes and consists of one or more graphical elements. An internal node typically branches to left and right subtrees, wherein each subtree is itself an expression tree comprising at least one leaf node. The internal nodes of an expression tree are compositing operators, which treat the left and right subtrees as operands of the operator. The first node of the expression tree is commonly referred to as a root node. The root node of an expression tree represents the final image, and each node of the tree represents a portion of the final image.
Although a graphical element may of itself be of a certain size, it need not be entirely visible in a final image, or only a portion of the element may have an effect on the final image. For example, assume an image of a certain size is to be displayed on a display. However, if the image is positioned so that only the top left corner of the image is displayed by the display device, the remainder of the image is not displayed. The final image as displayed on the display device thus comprises the visible portion of the image, and the invisible portion in such a case need not be rendered.
Another way in which only a portion of an element may have an effect is when the portion is obscured by another element. For example, a final image to be displayed (or rendered) may comprise one or more opaque graphical elements, sonic of which obscure other graphical elements. Hence, the obscured elements have no effect on the final image.
If an element, or parts of elements, that have no effect on true final image can be identified, those elements (or parts) need not be rendered, thereby saving considerable time and possibly memory.
Problems arise with prior art methods, at least for images where overlaps occur, because these methods do not easily cope with transparent graphical objects, nor do they handle the full range of compositing operators. It is therefore desirable to at least ameliorate one or more of those problems.
In accordance with one aspect of the present invention, there is provided a method of optimising an expression tree for compositing an image, the expression tree comprising graphical elements and graphical operators, each node in the tree being either a graphical element or a graphical operator and having a region of the image represented by the node, the method comprising, for at least one node in the tree:
In accordance with another aspect of the present invention, there is provided a method of optimising an expression tree for compositing an image, the expression tree comprising a node being either a graphical element or a graphical operator and having a region of the image represented by the node, the method comprising the steps of:
In accordance with yet another aspect of the present invention there is provided a method of optimising an expression tree for compositing an image, the expression tree comprising a node being either a graphical element or a graphical operator and having a region of the image represented by the node, the method comprising the steps of:
A preferred embodiment of the present invention is hereinafter described with reference to the accompanying drawings and the Appendix, in which:
The Appendix contains pseudo-code routines suitable for computer implementation of the preferred embodiment.
In the following description of the preferred embodiment, it is assumed that an image composition expression tree, as herein described, has been determined for an image to be rendered.
Preferably, image region representations are hierarchical data structures suitable for representing a region or portion of an image and typically used in image processing. One such image region representation is known to those skilled in the art as “quadtrees”. Other forms of image region representations can serve the same purpose. For the sake of simplicity, an image region representation is hereinafter referred to as a quadtree.
Typically, the creation of a quadtree representing a region of an image requires the sub-division of the region into a plurality of cells, each cell being a portion of the region, and each cell represented by a node of the quadtree. Hence, increasing the number of subdivisions of a region of an image correspondingly increases the number of nodes of the quadtree, thereby increasing the depth of the quadtree and the resolution of the region represented by the quadtree.
Compositing operations include 13 main compositing operations; for combining two portions of a single image. The function of each of those compositing operations is set our in Table 1, where Dc is a premultiplied destination or resultant color, Do is a destination or resultant alpha (α) channel value, Ac is a premultiplied pixel color of a first portion of a first source A, Ao is an a value corresponding to the pixel having the color Ac, Bc is a premultiplied pixel color value of a portion of an image of a second source B, and Bo is the α channel value of the pixel corresponding to Bc of the source B.
Table 1 specifically shows various compositing methods for combining two different images together utilising different operators. Additional operators are also possible. The additional operators may be utilized to implement special effects.
The “wrap around” nature of the “plusW” operator means that when, for example, the addition of Ac+Bc is greater than a maximum value of a color component, the value is “wrapped around” to start again with reference to the minimum value in the color space. Alternatively, the process of “clamping” utilized by “plusC” involves clamping the addition of, for example, Ac+Bc to the maximum value of a color component when the addition is greater than this component.
In the preferred embodiment, an expression tree can contain a variety of node types including binary compositing operators, unary operators and primitives. Unary operators typically include colour transformations, image convolutions, affine transformations and image warping. Primitives typically include graphical elements like pixel-based images, spline-based paths, text, “all” (“all” is a graphical element which spans the size of the entire image being created), edge blends, boxes or the like.
Table 2 lists a set of binary compositing operators and the action to be performed when those operators are treated when simplifying an expression tree.
At a node of an expression tree represented by an operator, typically a region representation, such as a quadtree, is passed to each operand during the process of simplifying the expression tree. At the node comparing the operator, an action is to be taken as to whether a sub-tree branching off the node is to vainisih (i e , branches need to he pruned) or a quadtree corresponding to the unobscured portions of graphical elements is to be returned from this node for possible further processing at other nodes.
The following notation is used in Table 2:
In the last two columns of Table 2, typical replacement rules are specified, where “L” means replace a current node with the left sub-tree branching off the current node, “R” means replace a current node with the right sub-tree branching off the current node, and “V” means that the current node vanishes. A node that is to “vanish” implies that the region of the image represented by the node is obscured by other graphical elements. Hence, the node has no effect on the final image. If both operands vanish, the current node also vanishes.
Reverse operators can be substituted for the operators described in Table 2. The “over” operator, described as “A over B” implies graphical element “A” is over graphical element “B”. For example, the can he substituted by a reverse operator of the “over” operator, typically denoted as “rover” (reverse over), so that “B rover A” results in a composite of graphical element “A” and “B” equivalent to “A over B”.
As an illustrative example, consider the (first) operator in the first row of the “Operator” column of Table 2 (i.e., the “over” operator). At a current node of an expression tree represented by an “over” operator, a parent node passes a quadtree q0 to the current node. Following the action under the heading “Pass to left operand” (column 2 of Table 2), the quadtree q0 is passed to the left operand, which is the left sub-tree or branch at the current node.
The quadtree q0 is used to process the left operand, and a quadtree qL is returned as the obscuring area of the left operand. From “Pass to right operand” (column 3 of Table 2), the action to be take at the current node is to pass down, as the right operand a union of the parent node, quadtree q0, and the now returned, left-operand quadtree qL. The quadtree resulting from this union (q0∪qL) is used to process the right operand. A quadtree qR is returned to the current node as the obscuring area of the right operand. The current node then returns the union (qL∪qR) of the left operand qL and the right operand qR to the parent node (see “return” in column 4 of Table 2).
If the region represented by the left operand is found to be completely obscured by the quadtree q0 passed down to the left operand, the action “if left operand vanishes” of column 5 of Table 2 is to replace the current node with the right (“R”) sub-tree, or right operand. This is desirable because changing the tree by replacing the current node with its right operand does not change the rendered image, but improves the time taken to render the image. Similarly, if the region represented by the right operand is found to be completely obscured by the quadtree (q0∪qL) passed down to the right operand, the action “if right operand vanishes” of column 6 of Table 2 is to replace the current node with the left (“L”) sub-tree.
The treatment of unary operators when simplifying an expression tree depends on the type of operation:
In the preferred embodiment, an image composition expression tree (hereinafter “expression tree”) of an image to be rendered is traversed, preferably in a depth-first fashion. Each node of the expression tree receives from its parent node a region representation of one or more areas of the image. The region representation is compared to the region represented at the node to determine if the region represented by that node is obscured.
A node in which the region represented by the node is totally obscured is removed from the expression tree with an appropriate simplification of tile expression tree, as hereinafter described. In the event that the region represented by the node is only partially obscured, a clipping operator is applied to the region represented by the node to clip the region of the image represented at the node to discard the obscured portions of the image. For example, if the region represented by a node is totally obscured by one or more regions represented by other nodes of the expression tree, the node is removed from the expression tree in such a way that a graphical operation or a graphical element at the node need not be executed or rendered, whichever the case may be.
If a node is partly obscured by one or more regions represented by other nodes in the expression tree, a clipping operator is applied to the node in such a way that, when executing a compositing operator, substantially unobscured regions of the image represented at the node are in the resultant composite of the region of the node. When an image is composited and subsequently rendered from an expression tree comprising nodes clipped by a clipping operator, substantially those portions of the graphical elements that are unobscured by other graphical elements of the image are reproduced or rendered.
Applying a clipping operator to a node can, in its simplest form, result in the cropping of the graphical elements represented at the descendent nodes to substantially those portions of the graphical elements that are unobscured. However, applying a clipping operator to a node is not limited thereto. Applying a clipping operator to a node of an expression tree having a compositing operation at that node can result in a different combination of compositing operators, having an effect on the node as if the region represented is cropped to its unobscured portion.
The process of compositing an expression tree 101 shown in
Alternatively, as shown in
If a region of the image represented by a node has been determined to be unobscured or only partially obscured, the node passes the region representation that the node received from a parent node, to each of its descendant nodes in turn. The same process occurs at each descendant node with the net effect that each descendant node passes back to its parent node either an image representation of tae areas of the image obscured by the region represented at the descendant node, or an indication that the descendant node is totally obscured.
After the descendants of a node have been processed, the region representations returned from the descendants are utilized to derive a region representation of the regions of the image that are obscured by the node. This result is returned to the node's parent.
In the preferred embodiment, the traversal of the expression tree to simplify the tree is initiated at the root of the tree in a “depth-first fashion”, known to those skilled in the art. Preferably, when traversing an expression tree in a depth-first fashion, the path leading down the left branch, at any node, is given priority and this path down the tree to a descendent node is taken first. When no further left ranch paths are available at a current node, processing returns to the previous node and a path heading down a right branch of this node is taken. An expression tree is traversed in this manner until all nodes of the expression tree have been visited.
In step 906, one of the remaining nodes is selected as the current node. In step 908, the region represented by the node is compared to a region representation data structure corresponding to one or more regions represented by at least one other node. The region representation is preferably of the form of a hierarchical data structure, and still further may be a quadtree representation. In decision block 910, a check is made to determine if the region represented by the node is obscured, either totally or partially, by one of the regions. If decision block 910 returns false (no), processing continues at control step 904. Otherwise, if decision block 910 returns true (yes), processing continues at step 912. In step 912, the expression tree is modified. The modification may include removing the current node or replacing the current node with another node of the expression tree. It may further include clipping, or marking for clipping at a later time, the region represented by the current node. Processing then continues at control step 904.
In decision block 1006, a check is made to determine if the current node is a graphical element or a graphical operator. If decision block 1006 determines that the node is a graphical element, processing continues at step 1008. In step 1008, the node is removed from the expression tree and processing returns to the calling procedure in step 1014. Otherwise, if decision block 1006 determines that the node is a graphical operator, processing continues at step 1010. In step 1010, a predetermined set of node replacement rules is applied in accordance with the graphical operator.
The predetermined set of node replacement rules (not shown in
In step 1108, one of the remaining nodes is selected as the current node. In step 1110, a first region representation is received from a parent node. In step 1112, a modified first region representation passes to a first operand of the graphical operator in accordance with a first predetermined modification rule for that operator. In step 1114, a second region representation of regions obscured by a sub-tree of the first operand is determined and it is returned to the graphical operator. In step 1116, a modified second region representation passes to a second operand of the graphical operator in accordance with a second predetermined modification rule for the operator.
In step 1118, a third region representation of regions obscured by a sub-tree associated with the second operand is returned to the graphical operator. In step 1120, a final region representation is determined in accordance with a set rule for the graphical operator and is returned to a parent node of the current node. Preferably, the set rule is selected from the group consisting of:
The first predetermined modification rule preferably is to pass substantially tile first region representation as the modified first region representation if the graphical operator is an “over”, “in”, “ratop”, “plusC”, “plusW”, “Xor”, “out” (visit left operand first)” or a like operator. If the graphical operator is an “out (visit right operand first)” operation, it involves passing as the modified first region representation a union of the first region representation with the second region representation.
Further, the second predetermined modification rule is to pass substantially the first region representation as the modified second region representation if the graphical operator is an “in”, “ratop”, “out”, “plusC”, “plusW”, “Xor” or like operators. If the graphical operator is an “over” operator, it involves passing as the modified second region representation a union of the first region representation with the second region representation.
Preferably, the image representation is not created at a node, or returned to a parent node of the node, unless the image representation is subsequently utilised, or if the node is the right operand of an “over” operator and the “over” operator node does not need to return an image representation to its parent node. Likewise, the image representation is not created at a node or returned if the node is the left or the right operand of an “in”, “plusC”, “plusW” or “Xor” operator and the operator node does not need to return an image representation to its parent node. Still further, the image representation may not be created at a node or returned to the parent node if the node is the left operand of an “out” or “ratop” operator and does not need to return an image representation to its parent node. The image representation may not be created at a node or returned if the node is the right operand of a “ratop” operator, the root of the expression tree, the operand of an image warp, affine transformation or convolution operator, the operand of a colour transformation that does not preserve opaqueness, or if the transformation node does not need to return an image representation to its parent node.
Further aspects of the preferred embodiment are set forth in detail in the Appendix forming part of the description. In particular, the Appendix contains pseudocode listings for implementing the method according to the preferred embodiment. In this connection, the preferred embodiment is preferably implemented as computer software, capable of being stored on recording media, that can be carried out as a process executing on a computing device, such as a general purpose computer.
The embodiments of the invention can preferably be practiced using a conventional general-purpose computer, such as the one shown in
The computer 1202 itself consists of a central processing unit(s) (simply referred to as a processor hereinafter) 1204, a memory 1206 which can include random access memory (RAM) and read-only memory (ROM), an input/output (IO) interface 1208, a video interface 1210, and one or more storage devices generally represented by a block 1212 in
The video interface 1210 is connected to the video display 1216 aid provides video signals from the computer 1202 for display on the video display 1216. User input to operate the computer 1202 can be provided by one or more input devices. For example, a operator can use the keyboard 1218 and/or a pointing device such as the muse 1220 to provide input to the computer 1202. Exemplary computers on which the embodiment can be practiced include IBM-PC/ATs and compatibles, and Sun SparcStations.
The first node 21 is a compositing operator (ie, an “over” operator) requiring a left and right operand. The left operand is a leaf node 22 representing the sub-image 11 of
Following receipt of the empty quadtree at the first node 21, the process passes the empty quadtree to leaf node 22. At the leaf node 22, the quadtree is typically compared with the sub-image 11 to determine if the sub-image 11 is obscured. However, in this example, since the quadtree is an empty quadtree, no direct comparison is necessary to determine the result that the sub-image 11 is not (or cannot) be obscured by the empty quadtree.
Comparing a quadtree with a graphical element (eg, the sub-image 11) entails a comparison, in which regions of an image represented by the quadtree ate compared with regions of the image covered by the graphical element to determine whether one region obscures another region of the image. The comparison of a quadtree representation of a region of an image with other regions of the image includes comparing the region of the image with the other regions either by direct comparisons of their respective areas, or by comparing equivalent representations or the like.
Sub-image 11 represented at the leaf node 22 is opaque and therefore can potentially obscure other graphical objects of the image 10. A first quadtree representation of sub-image 11 is therefore constructed which includes the bounds of the sub-image 11 and is returned to the first node 21 since no further left or right branches are available at the leaf node 22 of the expression tree 10. At the first node 21, the “over” operator performs a union of the quadtree originally passed to that node, being an empty quadtree, and the quadtree representation is returned from the left node, in accordance with the rules set out in Table 2 for the treatment of binary compositing operators.
The union of an empty quadtree with the first quadtree representation of the sub-image 11 results in a quadtree equivalent (or substantially identical) to the first quadtree representative and referred to hereinafter as the first left quadtree.
The first left quadtree is forwarded to the second node 23 of the expression tree 10, and is passed following the same manner as described in relation to node 21 to the left branch of the second node to a leaf node 24 branching off the second node 23. The circle 12 is represented at the leaf node 24. Upon forwarding the first left quadtree to the leaf node 24, the process compares the first left quadtree (that is an image region represented by the first left quadtree) to the region of the image occupied by circle 12 to result, at least for this example, in a finding that the region of the circle 12 of
The second node 23 typically receives from the leaf node 24 a quadtree representative of the portion of image 10 obscured by sub-image 11 and the circle 12 (a region obtained by the union of the sub-image 11 and the circle). However, in the present example, since the circle 12 is totally obscured by the sub-image 11, a union of the quadtrees for sub-image 11 and the circle 12 does not need to be performed.
A quadtree substantially equivalent to the first left quadtree representing the sub-image 11 is returned to the second node 23, where this quadtree is passed to a right leaf node 25, branching off the second node 23. The right leaf node 25 of the expression tree represents a region of image comprising text (“hello”) 13.
The text is not obscured by the quadtree (the image region represented by the quadtree) passed down from the second node 23. Typically, a quadtree representing the region of the image which is obscured by the graphical element at the right leaf node 25 is returned to the second node 23. However, since the text does not obscure a substantial region (area) in this case, an empty quadtree is returned to the second node 23. A substantial region is preferably defined by a performance issue of the process as hereinafter described.
The second node 23 receives the empty quadtree from the right leaf node 25. Following the action (shown in Table 2) of an “over” operator at the node when the left operand is obscured, the second node 23 replaces itself with the right leaf node 25 and prunes the left “branch”, which in this example is the left leaf node 24. The quadtree (albeit the empty quadtree) returned to the second node 23 is passed back to the first node 21.
At the first node 21, neither of its descendants are pruned and the action of an “over” operator is to form a union of the quadtrees returned by it to the “over” operator left and right branches. Typically, the result of this union is passed back to the node's 21 parent node. However, this step can be optimised out of this example because the first node 21 is the top-most node of the expression tree (root node). Therefore, the result of the union is not utilised in the optimisation of the expression tree, and the simplified expression tree is illustrated in
Another example of simplifying (optimising) an expression tree is now described with reference to
The following steps S0 to S14 correspond to the action taken by a computer implemented process at each node when simplifying the expression tree 50 of
At this point, typically, a new quadtree representing the region of the image occupied by the text is created and returned to the fifth node 55. However in this embodiment, if a graphical element is too small to substantially obscure other graphical elements of the image (eg, the graphical element text “hello” 43 does not substantially obscure other graphical elements even though the bounding box of text 43 represents a substantial region), an empty quadtree is preferably returned rather than expend processing time to achieve a quadtree representation of the region of text 43. Hence, the creation of a new quadtree q3 for regions of the image occupied by text 43 is chosen as an empty quadtree. The choice to create an empty quadtree for the region represented by text 43 is an issue of performance of the process that is hereinafter described under the sub-heading “Performance issues”. While a quadtree representation for text 43 can be created, the cost in the performance speed of the process out-weighs the time it lakes to render text. Hence, the empty quadtree q3 is created and passed back to the fifth node 55.
The graphical element page “D” is therefore either tagged so as to be clipped to the boundary of the circle 44 (a clip path derived from quadtree q2) at some later stage of processing typically, before rendering, or a clipping operator is applied and the page “D” 41 is clipped so that the region described by the circle 44 is cut out of the page “D” 41. A quadtree can be created for representing the page “D” 41 so that it may be passed back to a parent node. However, in this example, the creation of such a quadtree is not needed since it can be deduced that no further graphical elements can be obscured.
The foregoing examples of quadtree representations, described with reference to
Firstly, increasing the depth (ie, the number of nodes and branches) of a quadtree increases the quadtree resolution and the ability to detect obscuration. However, beyond a predetermined resolution, the computational cost of creating and combining quadtrees increases exponentially, exceeding the savings in performance gained by attempting to eliminate from an expression tree the diminishing areas represented by the increased quadtree depth.
Secondly, it is computationally expensive to treat every opaque primitive as a potential obscurer (a graphical element likely to obscure other graphical element of an image). The smaller a primitive is the less likely it is to obscure another primitive. Hence, the creation of quadtrees is preferably limited to potential obscurers that are of a predetermined size or greater. Typically, primitives that are too costly to convert to a quadtree are not considered because they cannot guarantee a good return on the investment. Thus, a “good obscurer” preferably has the following features:
Thirdly, testing for obscuration (ie, determining whether a first graphical element obscures one or more graphical elements of an image) can be performed by representing the region covered by the first graphical element as a quadtree and testing if one or more cells of the region represented at the nodes of the quadtree obscure regions covered by the one or more graphical elements of the image. Typically, the one or more regions are also represented by quadtrees, and the cells of quadtrees are compared. However, representing an arbitrary region of an image as a quadtree representation, to a predetermined resolution, may prove very computationally expensive though entirely possible. Hence, a bounding box of a region represented at a node of an expression tree is preferably constructed. Whether the node is a graphical element or an operator, the region represented at the expression tree node is well defined.
While the bounding box at a node of an expression tree may not exactly represent the region covered by the node, the enhancement in computational performance typically out-weighs the detriment in performance by failing to detect obscurities. Testing for obscured graphical elements by comparing their respective bounding box is preferred over comparing a quadtree representation of the regions of the image occupied by the graphical elements. This may result in some obscured graphical elements, below a predetermined size, being missed and considered not obscured. However, selecting a simple test for determining whether graphical elements are obscured by other graphical elements of the image is preferable over computationally expensive tests that in most common cases do not justify the return on the investment.
The following is an example of a pseudo-code call to a routine “test” which compares the bounding box at a node with a quadtree cell (hereinafter “cell”).
This function (routine) is invoked with:
Quadtrees are created and discarded continuously. A very simple and fast scheme to manage computer memory is preferred with low memory allocation activity (eg, allocating largish blocks of memory, say, 1000 quadtree cells at a time). Cells are allocated to these blocks, treated as write-once-read-only, and not deallocated until the end of the entire expression tree traversal. This approach allows cells to be shared amongst quadtrees, and considerably reduces copying when performing quadtree set operations.
Preferably, as a performance issue, if a region representation (quadtrees) need not be created, no region representation is generated at a node. For example, a parent node may request from a descendent node a quadtree of the region which the descendent node and its descendent node may obscure. Typically, if a region representation is never to be utilized in subsequent computation, the region representation preferably does not need to be created.
The aforementioned process for optimising an expression tree is described using recursion for convenience. Implementation of the process is also possible using a non-recursive process utilising back-pointers. This is both to reduce function-call overhead, and to handle very large trees that in practise are rarely balanced.
The foregoing describes only a small number of embodiments of the present invention and modifications, obvious to those skilled in the art in view of the foregoing description, can be made thereto without departing from the scope and spirit of the present invention.
The following function tests node for obscuration against quadtree q0. It returns whether or not all visible parts of node are obscured. If need_result, then it also returns a quadtree representing what areas node obscures. It is invoked with the call:
obscure(root node of tree, false, empty quadtree)
| Number | Date | Country | Kind |
|---|---|---|---|
| PO0021 | May 1996 | AU | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5123084 | Prevost et al. | Jun 1992 | A |
| 5274718 | Leonardi et al. | Dec 1993 | A |
| 5295236 | Bjorge et al. | Mar 1994 | A |
| 5515487 | Beaudet et al. | May 1996 | A |
| 5579455 | Greene et al. | Nov 1996 | A |
| 5600763 | Greene et al. | Feb 1997 | A |
| 5724494 | Politis | Mar 1998 | A |
| 5745121 | Politis | Apr 1998 | A |
| 20020027563 | Van Doan et al. | Mar 2002 | A1 |
| 20030118250 | Tlaskal et al. | Jun 2003 | A1 |
| 20050267908 | LeTourneau | Dec 2005 | A1 |
| Number | Date | Country |
|---|---|---|
| 2336295 | Feb 1996 | AU |
| A-2336295 | Feb 1996 | AU |
| 0528631 | Feb 1993 | EP |
| 069881 | Jan 1996 | EP |
| 0694881 | Jan 1996 | EP |
| 8-115413 | May 1996 | JP |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 08861063 | May 1997 | US |
| Child | 10368583 | US |
