Patent Application
20050140668
This invention relates to computer graphics in general, and more particularly to a novel system for presenting 3D graphics using a 2D graphics rendering engine. Among other things, the present invention may be applied to computer graphics systems for presenting animated characters and the like.
In many situations, it may be desirable to present images of 3D objects on a television or computer screen. These 3D objects may comprise substantially any type of object, including real objects and imaginary objects.
One type of object which may be presented is that of animated characters.
Inasmuch as the present invention is particularly well suited to the presentation of animated characters, the following description will be delivered in the context of presenting such animated characters. However, it should be appreciated that this is solely for the sake of example and not limitation. The present invention is applicable to the presentation of substantially any type of object.
Currently, 3D graphics systems work in the following manner. First, all 3D graphical information (including colors, geometries, textures, and lighting) is conveyed as a scene graph of (x, y, z) coordinates. Secondly, before displaying the 3D information onto a computer screen, a perspective transformation is applied, which essentially means that all 3D information is changed into 2D information. Finally, rasterizing translates the resulting coordinates to screen pixels.
A scene graph is composed of nodes. All graphical information in the scene graph is traversed some number of times per second (e.g., 30 times per second). Each node contains information about the transform from the original node (T), the material used to color that node (M), the geometry of polygons around that node (G), the texture used on that node (X), and the lighting shading that node (L).
For example, where the 3D object comprises an animated character, the animated character is typically animated by a series of nodes traversed throughout the character body from the original node.
The scene graph is all the information above (including but not limited to T, M, G, X, and L) held in a data structure of (x, y, z) coordinates. This mathematical information is updated some number of times per second to reflect any movement in the animated character.
The 3D information must be converted to 2D information in order to be shown on a 2D TV or computer screen. This conversion is called a perspective transform or camera transform. This transform accounts for the perspective difference in moving from 3D to 2D. The transform occurs as if a certain angle, often called a field of view, is chosen and the entire on-screen world is seen from this angle. This allows the 2D information to be interpreted from the 3D information, taking into account perspective.
The final step in showing an image on screen is rasterizing. Rasterizing is the process of converting the final information and putting it onto a computer screen.
3D graphics systems of the type described above are preferred in many applications, inasmuch as they provide a robust graphics environment able to produce highly realistic images. Unfortunately, however, such 3D graphics systems also have some significant disadvantages associated with them, including computational requirements, large file size, etc. Some of these disadvantages can become prohibitive where data must be transferred across a network (e.g., the Internet) and images rendered on a client computer, particularly where the client computer may have a relatively slow network connection and/or limited processing power.
As a result, some 2D graphics systems have been developed which may be used even where there is a relatively slow network connection and/or limited client processing power. Macromedia's Flash system is an example of one such system which is currently in widespread use. With the Flash system, a Flash player is installed on the client computer and relatively modest data files must be sent over the network. Unfortunately, the Flash system is based on 2D graphics, which generally provides a less robust user experience than 3D graphics.
Ingeeni has created a novel system for creating a 3D look in 2D, so as to provide the visual appeal of 3D graphics with the bandwidth and processing overhead of 2D graphics.
More particularly, Ingeeni has created an interface which sits between a 3D graphics system and a 2D graphics system, whereby to permit the creation of a virtual world in 3D graphics which is then converted to 2D graphics for transmission over a network and rendering on a client computer so as to provide the appeal of 3D-looking graphics systems.
In one preferred form of the invention, the Ingeeni interface is configured for use with Macromedia's Flash player, with the Ingeeni interface being implemented as a C++ application on top of Macromedia's Flash player. This interface is hereinafter sometimes referred to as the Ingeeni FlashAdapter. The communication is implemented using Macromedia's C++ API, in particular, the setVariable( ) method.
The Ingeeni FlashAdapter interface is preferably characterized by the following features:
See
This novel approach permits both C++ and Flash to be used for what they do best:
These and other objects and features of the present invention will be more particularly disclosed or rendered obvious by the following detailed description of the preferred embodiments of the invention, which is to be considered together with the accompanying drawings wherein like numbers refer to like parts and further wherein:
As noted above, the present invention relates to an adapter which sits between a 3D graphics system and a 2D graphics system so as to provide the best advantages of each system.
As also noted above, one such 2D system currently in widespread use is the Macromedia Flash system.
Inasmuch as the present invention is particularly well suited to use with the Macromedia Flash system, and inasmuch as the Macromedia Flash system is currently in widespread use, the following description will be delivered in the context of the Macromedia Flash system. However, it should be appreciated that this is solely for the sake of example and not limitation. The present invention is also applicable to use with many other 2D graphics systems.
Ingeeni has created the FlashAdapter to sit as an interface between a 3D graphics system and the Flash player. The following is a description of how the Ingeeni FlashAdapter sets up the communication channel on both the Flash side and the C++ side.
The Macromedia Flash player has a COM interface that exposes some of the basic functionality of Flash. It generally contains simple functions, such as play( ), stop( ), rewinds, and so on. For full documentation on the Flash player interface, refer to the Macromedia website at http://www.macromedia.com/support/flash/publishexport/scriptingwithflash/.
The Macromedia Flash player interface is geared towards using the interface in JavaScript, but the set of exposed functions and their use is identical to what the Ingeeni Player sees and uses in C++.
The problem with the exposed Flash interface is that it does not expose functions to create new Flash symbols, move them around, scale them, or hide them. It does expose properties of symbol objects (Flash movies) which, in theory, would make these manipulations possible. However, in practice, Flash does not do a good job of allowing one to write into the properties that the interface exposes and have those properties have much effect on the Flash symbols themselves.
The inventors have found that the only reliable method of passing information into Flash is using the SetVariable( ) function to write into actual Flash script variables, rather than writing into the properties of Flash objects.
So, in order to talk to Flash, the Ingeeni FlashAdapter employs the following model (see also
Each base movie to be used by the Ingeeni FlashAdapter (such as BaseMovie.swf) needs to have the following two pieces:
Details of how the aforementioned two pieces need to be set up can be found in the following section entitled “Practicalities—How To Make It Work”
Any movie can be made to act as the base movie for the interaction, but it can sometimes be easier to take a template that already exists, copy it, and modify it as needed.
Each function call from the Ingeeni FlashAdapter into the FlashMovieInterface is translated into SetVariables calls that have the following pattern:
This particular example adds an empty actor of the name “myActor” at depth 2 on the stage. Suppose it is desired to not only add an actor but also position it in the same batch of commands (so that both of these commands can be executed between frame renders, for example), here is how it may be done:
This set of calls not only adds an actor of name myActor2, but also moves it to position (100, 100) and scales it to (50%, 50%).
All Flash variables beginning with “w_”, such as “w_addToBatch” and “w_execute” in the example above, are being “watched” and any value assignment to them triggers a function call. Setting “w_addToBatch” to any value, for example, causes some variable shuffling on the Flash side, such that another function can be added to the function queue. Setting “w_execute” to any value, on the other hand, causes the batch of function calls queued on the Flash side to be executed.
Flash also needs a way to communicate back to C++. The Flash player is the source of user events, such as mouse movement and key strokes, and those events need to be sent back to the Ingeeni Player for processing. Fortunately, sending information in this direction (i.e., Flash to the Ingeeni adapter) is considerably simpler than in the opposite direction (i.e., the Ingeeni adapter to Flash).
Flash has an ActionScript command called fscommand( ) that broadcasts a COM event that can be easily caught in the Ingeeni Player. This COM event has two parameters associated with it: a “command” and an “argument”, although they literally translate to two strings that can contain any data which is inserted there. For the purposes of the Ingeeni FlashAdapter, the first string is used to identify the nature of the event (e.g., mouse movement, render event, etc.) and the second string is used to transmit any data that comes along with it (e.g., coordinates, value of the key pressed, etc.). So, for example, when the user moves a mouse, the ExternalInterface class in Flash calls the following command:
This indicates that the mouse has moved to the base movie coordinates (123, 56).
Design aside, there are very specific steps that need to take place in order for the FlashMovieInterface to be able to communicate with a Flash movie. Basically, the following needs to happen:
(1) The Flash movie must include the following lines of ActionScript in its resting frame (the frame where it ends up in and stays for the rest of the interaction—presumably this is the first frame since the main timeline will generally only have a single frame):
This piece of ActionScript includes the ExternalInterface class definition, instantiates it, and lets the FlashMovieInterface in C++ know the name of the instance (through the use of the global variable above).
Alternatively, the name “EI_PlaceHolder” can be replaced with something more compact, such as “x”, but back in the ActionScript for the main timeline of the movie, exactly where the ExternalInterface class is instantiated, the global variable g_EXTERNAL_INTERFACE_MOVIE_PLACE_HOLDER also needs to be modified and set it equal to “x”, like this:
In various forms of the invention, similar steps may be used to achieve the same effect.
This is a basic overview of the Ingeeni AdapterDebugger tool, defines its use, and suggests possible extensions. This is a description of the Ingeeni AdapterDebugger, which tests the Ingeeni FlashAdapter.
Initially, this tool was designed to test some of the specific functionality of the Ingeeni FlashAdapter class and catch its bugs in simple and reproducible cases. However, it has also been found to be useful as a tool to check and “tweak” the correctness of FLing files as well. Basically, it places a UI on top of the GFXAdapter API and, specifically, hooks it up to the Ingeeni FlashAdapter. Hence, it allows the creation of actor objects, moves them around, starts and stops actions on them, and switches between all available cameras.
The way it works is that there is a separate DLL, called AdapterDebugger.dll, that contains an ActiveX control used on an HTML page, called AdapterDebugger.html. That page has a large amount of JavaScript UI that calls into the ActiveX control and that, in turn, calls into the GFXAdapter class. This could be either Ingeeni FlashAdapter or WTAdapter, depending on whether the Flash player or the WT player have been embedded in AdapterDebugger.html. So, theoretically, one could take the AdapterDebugger.html, embed the WT player object into it instead of the Flash player object, simply by switching one of the <OBJECT> tags on the page and then you have the same HTML page testing the WTAdapter instead of the Ingeeni FlashAdapter.
Finally, all files relevant to the Web page hosting the Ingeeni AdapterDebugger control, as well as the Web page itself, are also included in the Ingeeni AdapterDebugger project for easy editing. They are grouped under the “Web Page Data” project directory in the Solution Explorer.
To run the Ingeeni AdapterDebugger, first either an already existing AdapterDebugger.dll needs to be registered (this only has to be done once) using regsvr32.exe or the AdapterDebugger project must be built from within Visual Studio, which will then build and register the DLL. To do the latter, open up IngeeniComponents solution in VS, right click on the “AdapterDebugger” project in the Solution Explorer, select it as the StartUp Project, and select Build/Build Solution from the menu.
After this step, all that needs to be done is to open up AdapterDebugger.html in a Web browser.
The Ingeeni AdapterDebugger should be relatively easy to use, since all it really does is provide a UI to access the functionality of the GFXAdapter interface. Refer to the Examples.doc document that outlines some of the most common workflows.
To create an actor, a FLing file must be loaded that defines it. Click on the “Create” tab in the UI and specify the path to the actor files, the actor file name, and the instance name. When “Create Actor” is clicked, the GFXAdapter tries to load:
When an actor loads successfully, the various actor, node, camera, and action drop-downs get pre-populated with the names of the various components.
To display an actor that has been loaded, two things are needed:
To transform an actor or one of its nodes, click on the “Transforms” tab. Select an actor or its node from the lists above the tabs, select whether the transforms should happen in local or world coordinate systems, and finally select the transform which is to be performed.
Click on the “Other” tab. Select the actor or a node from the drop-down lists above the tabs that are to be hidden and click on the “Hide Object” button. To show an actor or a node that has been previously hidden, select it from the drop-downs and click on the “Show Object” button.
Click on the “Other” tab. Select the actor which is to be attached from the drop-down lists above the tabs. Even if a specific node is selected, the entire actor will be affected. On the drop-down on the bottom of the screen, select the actor and the node to act as the new parent and click on “Attach”. If an actor previously attached to another is to be detached, attach the child actor to “(world)”.
Individual characters or figures are modeled by a 3D scene graph hierarchy, with each graph node possessing six degrees of freedom (three positional, three rotational). The world position and orientation of each node can be computed at any given moment in time by concatenating the 4×4 transformation matrices of the node with those of its ancestors. The position and orientation of a node within an active viewport, taking into account the active camera position and perspective foreshortening effect, can be obtained via additional transformations and arithmetic operations. For efficiency, the 4×4 world-to-view matrix is usually combined with the local-to-world transformation.
When the Ingeeni Adapter system is used with a 2D animation-rendering engine (e.g., the Macromedia Flash system), the scene graph of each visible character is traversed during each frame update cycle. For each node, up to three Euler angles (Euler angles are commonly referred to by various names such as “roll, pitch, and yaw”) of the resultant orientation in the active view are used as indices into the set of physical animation assets representing the given <scene graph node, activity> pair. The result of the lookup is a reference to a sequence of animated frame assets, which are then rendered by an animation engine (which can be a commercial tool such as the Macromedia Flash player). The Ingeeni graphics adapter provides the rendering engine with the screen coordinates and scaled size of each animation sequence, so that the positional coordinates and display sizes of the displayed sequences are theoretically correct to within single-precision floating point error. However, since a limited number of physical animation sequences will be provided for a given <scene graph node, activity> pair, the rendered animation only visually approximates the rotational orientation implied by the underlying scene graph.
When a character rendered in the manner just described is continuously panned or otherwise rotated, visually objectionable artifacts can occur between adjacent nodes of its scene graph (for example, the head and either eye). Suppose a character's head (
The solution invented by Ingeeni involves the 2D graphics adapters (e.g., the Flash engine) maintaining essentially two local transforms for each interior node of a scene graph which is represented by a rotationally discrete set of animations; an “uncorrected” transform corresponding to the conventional 3D scene graph implementation, and a jitter-corrected transform. The jitter-corrected transform represents the sampled orientation as currently rendered by the animation engine; during a panning activity, for example, the jitter-corrected transform will remain fixed until the node's indexed 2D animation changes. Since only rotational jitter is being considered here, the jitter correction can be conveniently stored as a set of three Euler angles, or three Euler angular offsets relative to the uncorrected node transform; an equivalent 4×4 jitter-corrected transform can then be computed on the fly.
The 2D graphics adapter (e.g., the Flash engine) allows the game engine to deal exclusively with the uncorrected set of transforms, so that each scene graph node appears to have fine-grained rotational granularity, up to the limits of floating point precision. The uncorrected orientation is used for all high-level game logic such as path-finding, collision avoidance, etc. During the frame update cycle, the uncorrected 4×4 transform for each node is also used when selecting the appropriate 2D animation sequence for the node itself. To perform the rendering calculations, however, the node's local transform must be concatenated with those of its ancestors in the scene graph hierarchy to obtain a full transformation to world coordinates. It is the jitter-corrected transform which is used for each ancestor node during this calculation. Using the earlier example, the calculation of the position and orientation of each eye will concatenate the local orientation of the eye (fixed, staring straight ahead) with the jitter-corrected (discrete, sampled) head orientation.
The result is that changes to the rotational orientations of interior scene graph nodes do not cascade to affect the positions and orientations of their children, until the 2D animation sequences of the interior nodes have been updated as well. This avoids the rotational jitter artifact involving nodes on adjacent layers of a scene graph described earlier.
This patent application claims benefit of pending prior U.S. Provisional Patent Application Ser. No. 60/532,969, filed Dec. 29, 2003 by Michal Hlavac et al. for INGEENI FLASH INTERFACE (Attorney's Docket No. INGEENI-4 PROV), which patent application is hereby incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 60532969 | Dec 2003 | US |
