Patent Application
20070262988
The present invention is directed to systems, apparatus, and methods for generating images at multiple levels of detail or resolution for use in computer graphics or animation productions. As part of the image rendering process, the invention utilizes a hierarchical data structure that contains multiple levels of detail or resolution of object data, with the highest resolution data representing the object obtained by sampling a physical object or generating data from a model of an object. In some embodiments, the highest resolution data is processed to yield lower resolution data using a smoothing, averaging, or other suitable process, with the new data being stored in a separate level of the data structure. This process is repeated to produce sets of data at even lower resolutions by processing the data set obtained by processing the high resolution data, and so on. When generating an image for a scene, the level of detail or resolution required for the scene is determined, followed by accessing the level or levels of the data structure corresponding to that level of detail or resolution. The data elements or voxels contained in the accessed level are then associated with a “renderable primitive”, so that each data element or voxel has a corresponding primitive. The primitive may be a cube, sphere, disc, or other suitable element. The renderable primitive is then processed or rendered to yield the desired image. In some embodiments, the data contained in the voxel or data element may also be used as part of the rendering process.
Although the present invention will be described with reference to example embodiments, it is noted that practice of the invention is not limited to those embodiments. For example, the hierarchical data structure need not be the described voxel MIP map, brick map or an octree structure. Similarly, the number of voxels in a unit of the structure need not be the same as for the described embodiment or embodiments of the invention. Further, the renderable primitive associated with the voxel data or other data element need not be restricted to the forms or formats mentioned, but may be generalized to any suitable form. In addition, although certain elements of the inventive system may reside in a server or servers, or connected by means of a network or networks, such elements may be separate from one another, or combined, and connected as desired to implement the functions and processes of the invention.
As will be discussed in greater detail, in some embodiments of the invention, the hierarchical data structure is a brick map, which in some embodiments is an octree having a single brick of N×N×N voxels at its uppermost level, eight bricks associated with that single brick at the next lower level, eight bricks associated with each of those eight at the next lower level, and so on. In other embodiments, a brick map may be represented by a single brick at its uppermost level, four bricks associated with that single brick at the next lower level, four bricks associated with each of those four bricks at the next lower level, and so on, or any other suitable configuration. In still other embodiments, the data structure may be a voxel MIP map or similar structure.
Further, in some embodiments, the data structure may contain data sampled or otherwise generated at multiple resolutions, instead of the described process of iteratively generating multiple data sets of progressively lower resolution by operating on an initial collected or generated data set. Thus, the inventive process may automatically generate data corresponding to multiple levels of detail or resolution from an initial set of data, or utilize multiple data capture or data generation events to populate the data structure.
Next, each data element (e.g., voxel or other suitable element) in the accessed data is associated with a renderable primitive (stage 220). In the context of the present invention, a renderable primitive is a box, sphere, disc, or other shape that may be processed by the data processing operations of a rendering program. In this stage or step of the inventive process, each data element is represented by a renderable primitive, and that primitive is used as the unit of data to be processed for some or all of the data processing operations that are part of the rendering process. At the next stage (222), the renderable primitive corresponding to the accessed or interpolated data is rendered to produce the final image. As will be described, the data element may also be accessed and processed as part of the rendering process.
As mentioned, at stage or step 220, the data element, voxel, or other suitable element is represented by a renderable primitive. Each renderable primitive (e.g., cube, sphere, disc, etc.) corresponding to a voxel or data element may be said in some sense to “inherit” the information contained in the data stored in the voxel. For example, if the voxel contains color data, then the renderable primitive will be characterized by that color. Similarly, if the voxel contains position and/or surface normal data, the renderable primitive will be characterized by a position and/or orientation corresponding to that position/normal. If the voxel does not contain or specify certain data (e.g., color, position/normal data) then default values may be used for that aspect of the renderable primitive.
With regards to stage or step 222, note that although the renderable primitive is processed or otherwise operated on during the rendering process, in some situations the actual voxel or data element may also be processed as part of producing the image. For example, the visibility of an object may be determined using the renderable primitives, while certain other characteristics of the object may be determined by rendering of the voxel data, such as by operating on the voxel data using a “shader” application to determine a color or opacity of the object.
In one embodiment, user input device 140 is typically embodied as a computer mouse, a trackball, a track pad, a joystick, wireless remote, drawing tablet, voice command system, eye tracking system, and the like. User input device 140 typically allows a user to select objects, icons, text and the like that appear on the monitor 110 via a command such as a click of a button or the like.
Embodiments of computer interfaces 150 typically include an Ethernet card, a modem (telephone, satellite, cable, ISDN), (asynchronous) digital subscriber line (DSL) unit, FireWire interface, USB interface, and the like. For example, computer interfaces 150 may be coupled to a computer network, to a FireWire bus, or the like. In other embodiments, computer interfaces 150 may be physically integrated on the motherboard of computer 120, may be a software program, such as soft DSL, or the like.
In various embodiments, computer 120 typically includes familiar computer components such as a processor 160, and memory storage devices, such as a random access memory (RAM) 170, disk drives 180, and system bus 190 interconnecting the above components. In one embodiment, computer 120 includes one or more Xeon microprocessors from Intel. Further, in one embodiment, computer 120 typically includes a UNIX-based operating system.
RAM 170 and disk drive 180 are examples of tangible media configured to store data such as an animation environment, models including geometrical descriptions of objects, descriptions of illumination sources, procedural descriptions of models, frames of training data, a specification of key points, embodiments of the present invention, including executable computer code, human readable code, or the like. Other types of tangible media include floppy disks, removable hard disks, optical storage media such as CD-ROMS, DVDs and bar codes, semiconductor memories such as flash memories, read-only-memories (ROMS), battery-backed volatile memories, networked storage devices, and the like.
In some embodiments, computer system 100 may also include software that enables communications over a network such as the HTTP, TCP/IP, RTP/RTSP protocols, and the like. In alternative embodiments of the present invention, other communications software and transfer protocols may also be used, for example IPX, UDP or the like.
As noted, in some embodiments of the invention, the hierarchical data structure is a brick map, which is an octree structure. In some embodiments, each brick in the brick map is a “cube” containing 8×8×8 voxels. Thus, at the highest level (which may be referred to as level zero), the data structure has one brick. At the next highest level (which may be referred to as level 1), the data structure has eight bricks, with each brick containing 8×8×8 voxels. At the next highest level (which may be referred to as level 2), the data structure has sixty-four bricks, again with each brick containing 8×8×8 voxels, and so on.
The number of levels or depth of the octree is determined by the density of the input data (that is the highest resolution data to be stored in the data structure). In general, it is desirable to choose the finest resolution of the octree such that the voxel or data element size is roughly the same as the radius of the data points (on average one data point per non-empty voxel). At that octree resolution, the voxel representation provides a reasonable approximation of the original point data. If the finest octree resolution was chosen to be coarser, it would not provide a good approximation of the original point data. If the finest octree resolution was chosen to be finer, then the octree memory would increase without providing any significant additional accuracy in the representation.
Note that in some embodiments, the data values for a voxel are computed as a weighted average of the data values of those original data points that overlap the voxel. For rendering, typically the level of the octree will be chosen such that a voxel is roughly the size of a pixel. Each rendered voxel may contain data from just a single data point (if the finest octree level is chosen) or it may contain average data from many data points (if a coarser octree level is chosen). Thus, from the amount of data collected or generated (which may depend on the highest resolution image expected to be required), one can determine how many bricks will be used to store the data. Given this information, the total number of levels of the octree can be determined by averaging data in the bricks at each level into the next level up (and therefore at lower resolution) until reaching the top level at which there is only a single brick.
Prior to describing the process of generating and using the brick maps in greater detail, it may be instructive to illustrate the octree hierarchical data structure that is used in some embodiments of the rendering process of the present invention.
The data 430 represented by the voxels in the bricks of
In some embodiments, the inventive process operates on the data represented by the voxels in
The averaging, smoothing, or other suitable process is again applied to the data contained in the bricks and voxels of
The present invention has been described generally with reference to
In some embodiments, a brick contains 8×8×8 voxels where each voxel is associated with data values such as (but not limited to) average position, normal, color, transparency, specularity, density, texture parameters, surface parametric values (u,v) or (s,t), volume scattering coefficients, etc. Note that the invention may be implemented with a different number of voxels per brick as well. Similarly, although some embodiments of the invention utilize an octree data structure, it is not limited to that form of structure and may be used with others as well. As noted, there are two basic types of brick maps: brick maps for surface data and brick maps for volume data. Surface data points have associated normals, and to avoid mixing data from the two sides of one surface, the points can be divided into six groups depending on the dominant direction of the normal (+x, −x, +y, −y, +z, and −z). Each of the six groups may be stored in a separate octree. Volume data points do not have surface normals, so one octree is sufficient for storing that data.
One process for creating a brick map will now be described. Given a set of points with associated radii, data, and possibly normals, a brick map representation of the data will be generated for use in the inventive rendering process. This may be accomplished in three steps. First the bounding volume of the entire data set is found. Then, the data points are divided into seven groups, depending on their normals: one group for points with no normal, and one group for each of the six major directions. Data for points with normals close to 45 degrees may be assigned to 2 (or even 3) groups to facilitate faster data access.
Next, each data group is stored in a hierarchical data structure such as the brick map octree structure described previously. The octree nodes and bricks are created as the data points are inserted into the structure. Each octree node may consist of 9 pointers: eight pointers to child nodes and one pointer to a brick containing the data for the current node. For each data point p, the process computes a small volume, Vp, based on its position and radius. For each octree node, starting at the appropriate root, the data are recursively inserted into the bricks of the octree nodes that Vp overlaps. When a point's data are inserted into a brick, the process determines which brick voxels the point's volume overlaps, and adds the data values to the data in those voxels. When added, the data are multiplied by a weight (Vp∩Vv)/Vp that indicates how large a fraction of the point volume, Vp, overlaps the voxel volume, Vv. This weights the data based on the overlap between the point and voxel volumes.
When all the data have been inserted, the process of constructing the brick map proceeds to the third step. First, the data values in all voxels are divided by the weight of the voxel. The process then determines the data variation of all 2×2×2 voxel groups of the brick. If the voxel group data variation is smaller than a user-specified maximum error, then the data in those eight voxels is eliminated. If all voxel data in a brick are eliminated, that entire brick may be eliminated. If any voxel data of a brick are not eliminated by this process, the remaining voxel data of the brick are stored in memory. Empty voxels are not stored, thereby saving disk space when storing sparse brick maps. To reduce the peak memory usage during brick map construction, the process may construct and write to storage one octant of each octree at a time. Furthermore, brick voxels and voxel data may be dynamically allocated and enlarged as needed during construction. In this way, the process avoids allocating an entire set of voxels for sparse bricks.
Next, a process for determining the appropriate hierarchical level of the data structure for use in the rendering process will be described. Typically, to obtain a reasonable image quality and utilize the computational resources efficiently, it is desirable to have each pixel in the final rendered image correspond to one voxel of data. Therefore, given the desired final size of the rendered image in pixels, the number of voxels required to be processed can be determined. Given the number of voxels to be processed, the corresponding hierarchical level can be determined, since it is known how many bricks and hence how many voxels are present in each level. If the desired size of the final rendered image indicates that the number of voxels falls between two hierarchical levels, then voxels from both levels may be accessed and rendered, followed by a blending operation.
As described with reference to
For example, if the voxel contains position data, then the renderable primitive can be centered at that position. Otherwise, the voxel center may be used as the position. If the voxel contains orientation data (e.g., a surface normal), then the renderable primitive can be oriented accordingly. Otherwise, the renderable primitive can be axis-aligned, face the viewing position or rendered using another default orientation. If the voxel contains color data, then the renderable primitive can be assigned that color. Otherwise, the renderable primitive can be assigned a default color. Similarly, if the voxel contains opacity data, then the renderable primitive can be assigned that opacity. Otherwise the renderable primitive can be assigned a default opacity. Further, the renderable primitives can optionally be processed by a “shader” program that can modify the position, normal, color, and opacity of one or more renderable primitives.
In operation, bricks may be read from a data storage disk on demand and cached in memory. If the cache lookups are coherent, the cache has a high hit rate and caching is very efficient. The cache may use a least-recently-used (LRU) replacement strategy. In one implementation, the brick map cache size can be selected by the user. The default size may be set at 10 MB, corresponding to a capacity of 1574 bricks if the brick data consist of 3 floats per voxel.
As discussed, a benefit of the invention is the ability to render the brick maps directly, and if desired, to use both the renderable primitives and the voxel or data element data as part of the rendering process. This is in contrast to rendering the scanned or model generated data, as is typically done in other rendering methods. The present invention has desirable properties since the brick map is an inherent level-of-detail representation of the geometry. Since the bricks are stored in a fixed-size cache and only loaded on demand, the user can render complex scenes in a computationally efficient manner with this approach. As an example, consider a complex object defined as a very dense polygon mesh or a large collection of NURBS patches. Even if the object only covers a single pixel in the image, one would typically need to read the entire object definition into memory (and keep it there until certain it is not needed further). In contrast, if the object is represented using the inventive brick map based method, only the very coarsest brick will be read in and stored in memory. As noted, the appropriate brick map level may be determined from the screen size of the object, where a reasonable convention is to choose the brick map level such that the brick voxels are approximately the size of an image pixel.
Note that in addition to rendering directly visible brick maps, ray tracing and other operations may also be applied to them. The appropriate brick map level may be determined from the ray differential (where the ray differential describes the difference between a ray and its—real or imaginary—“neighbor” rays). The method may select the brick map level where the brick voxels are approximately the size of the ray differential cross-section. The ray-brick intersection test can be performed in several ways. A brute-force approach would be to test for intersection with each of the (up to 512) non-empty voxels in the brick (this is the most efficient approach for bricks with very few non-empty voxels). For denser bricks it is more efficient to recursively subdivide the brick into octants and test the octants for intersection before continuing the recursion (note that the recursion stops after at most three levels when the method reaches the level of individual voxels). A voxel grid-stepping approach similar to Bresenham's line-drawing algorithm is also possible to utilize.
As noted, in addition to ray tracing, other rendering operations may be applied to the data stored in the brick map. These include displacement mapping, motion blurring, shading operations, etc. In general, the brick map may be treated as a geometric primitive and operated on as one would process other geometric primitives in a rendering process.
A system, apparatus, and method of rendering images at multiple levels of detail or resolution for computer graphics or animation has been described. The invention is more computationally efficient than current approaches and provides advantages when compared to such approaches. The invention uses a hierarchical data structure (such as the described brick map or another suitable structure) to store data at multiple levels of detail or resolution, where the invention automatically generates those multiple levels of detail or resolution from an initial set of collected or generated high resolution data. As part of the rendering process, the invention determines the appropriate hierarchical level of data to access and then associates a renderable primitive with each voxel or data element in the accessed data. The process then renders the renderable primitives to produce the desired image.
Various embodiments of the invention provide several important benefits and advantages:
It should be understood that the present invention may be implemented in the form of control logic using software, hardware, or a combination of both, in a modular or integrated manner. The present invention can be implemented as a stand-alone system or as part of a larger computer system. Based on the disclosure provided herein, a person of ordinary skill in the art will know of other ways and/or methods to implement the present invention.
It is understood that the examples and embodiments described herein are for illustrative purposes only and that various modifications or changes in light thereof will be suggested to persons skilled in the art and are to be included within the spirit and purview of this application and scope of the appended claims. All publications, patents, and patent applications cited herein are hereby incorporated by reference for all purposes in their entirety.
As used herein, the use of “a”, “an” or “the” is intended to mean “at least one”, unless specifically indicated to the contrary.
This application claims the benefit of U.S. Provisional Patent Application No. 60/799,134, entitled “Method and Apparatus for Using Brick Maps as Geometry”, filed May 9, 2006, the contents of which is incorporated herein for all purposes.
| Number | Date | Country | |
|---|---|---|---|
| 60799134 | May 2006 | US |
