System and method for constraining a graphical hand from penetrating simulated graphical objects

Abstract

An apparatus, system, method, and computer program and computer program product are provided for constraining the movement of a graphical hand when the physical hand controlling the graphical hand does not have a similar physical constraint. The constraining technique may comprise use and analysis of a revolute-joint-link-spring model. In such a model, an uncompressed/unextended spring position represents the corresponding measured joint angle or link position. In addition to linear springs which follow Hook's Law, i.e., F=k*x, non-linear springs or other non-linear force functions may be employed to obtain the desired result of allowing a graphical joint or link to deviate from what the corresponding measured joint or link provides. In particular, if a graphical hand configuration corresponding to measured joint and link positions causes a portion of the hand to penetrate a simulated graphical solid object, a mathematical determination is used to compute modified joint and link positions such that the graphical hand part will no longer penetrate the graphical solid object. Such a constraint technique may include solving a spring model such that the various joint and link springs compress or extend to produce modified joint and link positions. Such a constraint technique may also be applied to constrain other graphical body parts and graphical inanimate objects, where corresponding physical controlling, i.e., measured, body parts and inanimate objects do not possess a similar constraint or impediment.

Description

FIELD OF THE INVENTION

[0002] This invention relates generally to systems, apparatuses, and methods for simulation and training using interactive computer graphics.

BACKGROUND OF THE INVENTION

[0003] Description of the Relevant Art

[0004] In the past few years, computers have seen a significant increase in 3D graphical capabilities. Not surprisingly, interfaces capable of interacting with these complex 3D environments have started to move from laboratories to the enterprise and the consumer market as companies have begun to produce them in volume. While interfacing hardware has made significant advances, the accompanying software has lagged behind. This is not surprising considering the complexity involved in physically modeling computer-generated environments. Nonetheless, companies producing some of the less-complex interface hardware have introduced Application Program Interfaces (APIs) and Software Development Kits (SDKs) that facilitate the task of integrating their products into third-party applications.

[0005] Some of the hardware interfaces include force-feedback joysticks and steering wheels. Two companies which have introduced API's and SDK's for their joystick/steering wheel peripherals include Immersion Corporation and Cybernet Systems. Immersion markets its I-FORCE™ API, a programming interface that lets developers simulate various force-feedback effects such as: impacts, recoils, vibrations, springs, etc. It has been integrated into Microsoft's DirectX specification [4]. Similarly, Cybernet has introduced its CyberImpact™ SDK, with comparable features to Immersion's I-FORCE.

[0006] Another company that produces a haptic SDK is SensAble Technologies. SensAble markets a 3-degree-of-freedom (3DOF) force-feedback stylus called the PHANToM™ and their accompanying SDK is named Ghost™ [5]. Ghost is an object-oriented toolkit that represents the haptic environment as a hierarchical collection of geometric objects (spheres, cubes, cylinders, etc.) and spatial effects (springs, dampers, vibrations, etc.) The Ghost toolkit includes a spring model for presenting a single graphical point, as controlled by the PHANToM, from penetrating a graphical object. Noma and Miyasato propose a technique similar to Ghost, but also limit their technique to a single, non-articulated point [7].

[0007] Predictably, the first SDK's and API's to appear have been associated with relatively simple devices such as joysticks, steering wheels and 3D styli. In these cases, either the degrees of freedom of movement of the device are limited, or a single point is interacting with the environment. Not surprisingly, comparable software for whole-hand input interfaces has lagged behind because of the inherent complexity of the whole-hand interaction paradigm, with its multiple constraints and multiple, articulated degrees of freedom. Although whole-hand input devices present a greater challenge to integrate, they provide an extremely appealing advantage for interaction. What better way to manipulate complex virtual objects than to just “reach in and grab them?”

[0008] When trying to manipulate 3D objects, interfaces such as 2DOF mice, joysticks and styli quickly display their inherent limitations. However, users wearing instrumented gloves, or using some alternative hand-measurement system, if properly implemented, can manipulate digital objects just as they would in the real world. Up until a few years ago, these whole-hand software development efforts have resided in university and government laboratories. Examples include the work of Burdea et al. at Rutgers University using the Rutgers Master II [2], and the work of Coiffet et al., at the Laboratoire de Robotique de Paris, using the LRP Hand Master [1]. More recently, Luecke et al. have developed software to use an exoskeleton haptic device to apply virtual forces [3] and Thompson et al. [6] have used the Sarcos Dextrous Arm Master to perform direct parametric tracing on maneuverable NURBS models.

[0009] Commercially, two companies have introduced whole-hand software libraries. Virtual Technologies, Inc. has offered its VirtualHand® Software Library, while 5DT has offered an API for its DataGlove. Both let the user display a graphical hand on the screen that mimics the glove wearer's movement, but they do not provide manipulation capabilities; such capabilities are presently left to the user to develop. Inventive features of the subject invention include enhanced manipulation capabilities, the ability to prevent the virtual hand from penetrating grasped objects and virtual walls, and enabling dynamic interaction between the virtual hand and various controls.

REFERENCES

[0010] 1. P. Coiffet, M. Bouzit and G. Burdea, “The LRP Dextrous Hand Master,” VR Systems Fall 1993 Conference, Sig Advanced Applications, New York City, October, 1993.

[0011] 2. D. H. Gomez, G. Burdea and N. Langrana, “Integration of the Rutgers Master II in a Virtual Reality Simulation,” 1995 IEEE Virtual Reality Annual International Symposium, pp. 198-202, San Francisco, Calif., 1995.

[0012] 3. G. R. Luecke, Y. H. Chai, J. A. Winkler and J. C. Edwards, “An Exoskeleton Manipulator for Application of Electro-Magnetic Virtual forces,” Proceedings of the 1996 ASME Dynamics Systems and Control Division, pp. 489-494, Atlanta, Ga., Nov. 17-22, 1996.

[0013] 4. L. Rosenberg, “A Force Feedback Programming Primer—For PC Gaming Peripherals Supporting I-Force 2.0 and Direct X 5.0,” San Jose, Calif., 1997.

[0014] 5. SensAble Technologies, “GHOST Software Developer's Toolkit—Programmer's Guide Version1.1,” Cambridge, Mass. 1996.

[0015] 6. T. V. Thompson II, D. D. Nelson, E. Cohen and J. Hollerbach, “Maneuverable Nurbs Models within a Haptic Virtual Environment,” Proceedings of the 1997 ASME Dynamics Systems and Control Division, pp. 37-44, Dallas, Tex., Nov. 16-21, 1997.

[0016] 7. H. Noma and T. Miyasato, “Cooperative Object Manipulation in Virtual Space Using Virtual Physics,” Proceedings of the 1997 ASME Dynamic Systems and Control Division, pp. 101-106, Dallas, Tex., Nov. 16-21, 1997.

[0017] The following cited patents provide additional description that may assist in understanding the invention. Each of these patents is hereby incorporated by reference in its entirety.

[0018] 1. J. Kramer et al., “Strain-sensing goniometers, systems and recognition algorithms,” U.S. Pat. No. 5,280,265, Jan. 18, 1994.

[0019] 2. J. Kramer, “Force feedback and texture simulating interface device,” U.S. Pat. No. 5,184,319, Feb. 2, 1993.

[0020] 3. J. Kramer, “Determination of Thumb Position Using Measurements of Abduction and Rotation,” U.S. Pat. No. 5,482,056, Jan. 9, 1996.

[0021] 4. J. Kramer, “Determination of kinematically Constrained Multi-Articulated Structures,” U.S. Pat. No. 5,676,157, Oct. 14, 1997.

SUMMARY OF THE INVENTION

[0022] The subject invention includes a structure and method for determining the configuration of a graphical hand and arm based on the measurement of a controlling physical hand and arm, specifically where the graphical hand and arm are constrained in movement upon coming into contact with an immoveable graphical object or other simulated graphical impediment, but where the physical hand and arm are not similarly restrained in their movement.

[0023] In one aspect, the invention provides a system for moving a simulated multi-articulated structure in relation to the movement of an analogous physical multi-articulated structure, where the simulated structure moves in a simulated environment having a simulated impediment to free motion, where the physical structure moves in an environment lacking an analogous physical impediment, and where in one embodiment the system includes a device for measuring the configuration of the physical structure and the spatial placement of the physical structure relative to an inertial reference frame and providing digitized signals associated with the configuration and spatial placement; and a data processor, including data related to the spatial placement of the simulated impediment, for receiving the digitized signals and modifying the signals to generate a set of modified signals specifying the configuration and spatial placement of the simulated structure, whereby the free motion of the simulated structure is impeded when encountering the simulated impediment.

[0024] In another aspect, the invention provides a method for moving a simulated multi-articulated structure in a graphical environment, having an impediment to free motion, in relation to the movement of an analogous physical multi-articulated structure in a physical environment lacking the impediment, where in one embodiment the inventive method includes the steps of generating digitized signals representing the configuration and spatial placement of the physical structure and transferring the digitized signals to a data processor; and in the data processor, recording the spatial placement of the impediment, generating modified signals from the digitized signals specifying the configuration and spatial placement of the graphical structure, and impeding the free motion of the simulated structure when the simulated structure encounters the impediment.

[0025] Device, computer program, and computer program product implementing embodiments of the inventive system and method are also provided.

BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG. 1 shows a overview of an embodiment of the VirtualHand toolkit.

[0027] FIG. 2 shows a demonstration (computer screen capture) using the VirtualHand Toolkit to generate a graphical hand which is constrained to not penetrate a control panel and associated buttons.

[0028] FIG. 3 shows an embodiment of where the graphical hand is following constraints as it tries to track the ghost hand.

[0029] FIG. 4 is a diagrammatic view of an embodiment of a joint-link model of a human hand and forearm.

[0030] FIG. 5 is a diagrammatic side view of an embodiment of a spring model for the graphical hand, including a joint-link model of the human hand and forearm, with associated joint springs and tracker springs.

[0031] FIG. 6 is a diagrammatic view of an embodiment of: (A) ghost hand and (B) graphical hand with springs compressed due to constraint of the compliant wall.

[0032] FIG. 7 is a diagrammatic view of an embodiment of a spring-mass-dashpot model for the hand for use in dynamic situations.

[0033] FIGS. 8A-8F are diagrammatic views of various embodiments of a spring models for the graphical hand.

[0034] FIG. 9 is a diagrammatic side view of an embodiment of a spring model for a portion of a graphical human figure, with associated joint springs and tracker springs.

[0035] FIG. 10 is a diagrammatic side view of an embodiment of a spring model for a portion of a generic mechanical structure, with associated joint springs and tracker springs.

[0036] FIG. 11 is a diagrammatic view of two objects, A and B, with various critical elements identified.

[0037] FIG. 12 is a diagrammatic view of objects A, with various critical elements identified.

[0038] FIG. 13 is a diagrammatic view of two intersecting objects, A and B, with various critical elements identified.

[0039] FIG. 14 is a diagrammatic view of the palm of the hand.

[0040] FIG. 15 is a diagrammatic view of a facet with surrounding voronoi planes.

[0041] FIG. 16 is a diagrammatic side view of a finger, with links and joint angles defined.

[0042] FIGS. 17A and 17B are diagrammatic side views of a finger: (A) before calculating joint adjustments to back the finger out from the wall, and (B) after performing such calculations.

[0043] FIG. 18 is a diagrammatic side view of a physical hand and instrumented glove continuing to move after a graphical hand, being controlled by the glove, is caused to stop at a simulated table by the inventive constraining method.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

[0044] Overview of Aspects of the Invention

[0045] Graphical object manipulation in a computer-simulated environment holds much promise for increasing the productivity of users in a variety of occupations. Many of today's computer interfaces, such as mice and joysticks, are ill-suited for such a task and a better interaction paradigm is required. One such approach is the use of a graphical hand which mimics the movement of the user's hand as measure via an instrumented glove or other measurement device. While the concept of whole-hand interaction is intuitive and natural, its implementation is complex.

[0046] The field of virtual reality typically uses an instrumented glove or other hand-measurement apparatus to measure the movements of a user's physical hand and produce a graphical rendition of the hand (i.e., graphical hand) on a computer screen. Commonly, the goal of rendering the graphical hand is to allow contact and interaction between the graphical hand and a graphical object to be controlled or manipulated. The most common drawback to such a graphical simulation occurs when the physical hand does not possess any force feedback, or lacks adequate force feedback, to stop the physical hand and fingers from moving once the graphical hand has contacted a graphical impediment, such as an immovable graphical object. In such cases, the graphical hand associated with the physical hand will also continue to move, and thus penetrate the graphical object, greatly reducing the desired perception of the simulation that one is using their hand to interact with a real, solid object. There are two possible resolutions. One is to provide sufficient force feedback to stop the physical hand very precisely once the graphical hand impinges on the immoveable graphical object, such that the graphical hand may continue to correspond to the movements of the physical hand. That is, since the physical hand can be stopped by the force feedback device, likewise, the graphical hand would not be commanded to penetrate the object. An alternative solution, which finds use whenever there is non-existing, or insufficient, force feedback to the physical hand, is where the graphical hand ceases to exactly follow the physical hand. For example, a separate computer calculation might change the configuration, orientation or spatial placement, of the graphical hand such that it doesn't follow the measured physical hand, and such that the graphical hand obeys laws of physics during interaction with graphical objects. In particular, a graphical finger might bend backward such that it doesn't penetrate a graphical wall.

[0047] In typical previous attempts at simulating manual interaction with virtual objects, the simulation user or trainee wears an instrumented glove, i.e., a glove with sensors in it which measure the subtle movements of the fingers and hand. The industry standard instrumented glove on the market today is known as the CyberGlove®, manufactured by Virtual Technologies, Inc., the Assignees of this patent application. Hand-movement data is transmitted from the instrumented glove to the computer, which in turn draws a graphical image of the hand, i.e., a virtual hand, which replicates the movements of the physical hand. This virtual hand can then be made to interact with graphical objects, such as simulated controls. Based on the results of the interactions, the trainee learns how to operate the necessary controls, and also learns the effect the controls have on the high-level simulation. The problem with such simulations thus far has been the lack of intelligent software that realistically models the interactions between the virtual hand and an object to be manipulated. For instance, an all-to-typical scenario is that the virtual hand passes through the simulated control-panel surface as well as the controls that the virtual hand is intended to interact with. This unfortunately reduces the effectiveness of the simulation, since the trainee must change their manner of interaction to tailor to the limitations of the simulation. Thus “bad habits” might be learned which enable the trainee to efficiently operate the simulation, but due to their disparity from reality, do not lead to skills which transfer to the ultimate real task.

[0048] For example, in the physical world, when one turns on a household light via a standard wall-mounted toggle switch, one might extend their hand until it stops at the switch strike plate, and then slide their hand along the plate until contact is made with the switch. The sliding motion then continues until the switch has been toggled. In contrast, in present “virtual worlds,” the trainee extends their physical hand, causing the virtual hand to also extend. However, when the virtual hand contacts the virtual strike plate, it passes right through! Thus, the trainee must visually concentrate to prevent their virtual hand from penetrating the strike plate, and must focus intently on visually contacting the toggle lever with their virtual hand. In other words, since the virtual hand's movement is not constrained by the wall surface, the trainee must concentrate unrealistically hard to locate the toggle switch in the “depth dimension,” which is the dimension most difficult to judge visually.

[0049] The innovation of this inventive structure and method includes a hand-centric simulation method that adds physical constraints to virtual hand/object interaction and manipulation, and determines whether the virtual hand should not be allowed to pass through the graphical object, such as a control panel, or whether the virtual hand instead is capable of modifying the position or shape of the object, e.g., toggling a virtual switch control.

[0050] The subject invention provides an apparatus and method for constraining the movement of a graphical hand when the physical hand controlling the graphical hand does not have a similar physical constraint. The constraining technique may comprise use and analysis of a revolute-joint-link-spring model. In such a model, an uncompressed/unextended spring position represents the corresponding measured joint angle or link position. In addition to linear springs which follow Hook's Law, i.e., F=k*x, the springs may follow any desirable non-linear function to obtain the desired result of allowing a graphical joint or link to deviate from what the corresponding measured joint or link provides. If a graphical hand configuration corresponding to the measured joint and link positions would cause a portion of the hand to penetrate a simulated graphical solid object, a mathematical determination is used to solve the spring model such that the various joint and link springs compress or extend to allow the graphical joints and links to change their measured positions such that the graphical hand part will no longer penetrate the graphical solid object. Such a constraint technique may also be applied to constrain other graphical body parts and graphical inanimate objects, where corresponding physical controlling, i.e., measured, body parts and inanimate objects do not possess a similar constraint or impediment.

[0051] One embodiment of the subject invention comprises a host computer or data processor comprising a CPU (or other processor or micro-processor) and memory (such as RAM) coupled to the CPU. The memory stores data and procedures, including the inventive procedures, and cooperates with the CPU during execution. The procedures may be implemented with software executing on the host computer or data processor. The data processor may have one or more graphical objects or impediments loaded into its memory, its database, its scene graph, and/or the like. The data processor may have data related to the spatial placement of such graphical objects or impediments loaded into its memory, database, scene graph, and the like. Typically the data is stored, at least temporarily, in a data structure defined in the memory of the computer. The data processor may receive digitized signals (or other data) relating to the spatial placement and/or configuration of a controlling hand, body-part or compliant or articulated inanimate object. In the case where a hand is being measured, a CyberGlove instrumented glove may be used, or some other appropriate measurement device. The data processor may execute an algorithm residing in its memory for modifying the received digitized signals to generate a set of modified signals specifying a modified configuration and/or spatial placement of a corresponding simulated hand, body part or inanimate object, whereby the free motion of the simulated hand, body part or inanimate object is impeded or otherwise restricted when encountering the simulated impediment.

[0052] The simulated hand, body part or inanimate object may be displayed graphically on a computer monitor, although it need not be. The simulated hand may be a physical robotic gripper.

[0053] In another embodiment of the invention, the method for constraining the graphical or simulated hand, body part or other articulated structure, may be incorporated into software or firmware or other recording medium. In a further embodiment, the software is provided as a computer program product tangibly stored or recorded in a medium, such as a computer disk, CDROM, or other formats as are known in the art.

[0054] Additional Technology Description

[0055] FIG. 1 shows a diagrammatic view of the relationship between a Host Simulation Application and Virtual Technologies' VirtualHand Toolkit (VHT), a portion of which is the subject of this invention. FIG. 1 also shows the relationship between the VHT and a Hand Manager module capable of constraining the virtual hand from penetrating graphical objects. In particular, we describe a Hand-Interaction Scene Graph (or more generally, the Interaction Scene Graph), which functions within a Hand-Interaction Simulation. This hand-interaction scene graph interacts with a simulation's Master (a.k.a. World or Graphical) Scene Graph.

[0056] Before proceeding with the discussion of the details of the hand-constraint capabilities of the hand manager, we will provide a brief background description of the remainder of the supporting software architecture as provided in FIG. 1.

[0057] VirtualHand Toolkit

[0058] The VirtualHand Toolkit, which makes up a large portion of the supporting software, comprises the following three main components:

[0059] 1. Hand-Interaction Simulation (H.I.S.)

[0060] 2. I/O Daemon

[0061] 3. Calibration Panel

[0062] Hand-Interaction Simulation (HIS)

[0063] A central module of the VirtualHand Toolkit architecture is the Hand-Interaction Simulator (HIS) which controls all hand interaction. It may be a separate process running on the host computer or, alternatively, if the Virtual Technologies CyberGrasp haptic-feedback system is used with the CyberGlove, the HIS may alternatively run on the CyberGrasp controller. The HIS comprises four major components:

[0064] 1. Hand-Interaction Scene Graph (or more generally, just Interaction Scene Graph)

[0065] 2. Collision Manager

[0066] 3. State Manager

[0067] 4. Hand Manager

[0068] A principle component of the HIS is a Hand-Interaction Scene Graph (HISG), which contains a node for each of the interactive objects of interest in the “hand scene.” The hand scene is defined as all the objects that fall within a certain radius of the hand; this includes all the objects that form the hand itself. To assist with collision detection, each node may have a bounding sphere associated with it. Thus, one simple way to determine if an object falls within the realm of the graphical hand is to check for a collision (binary) with a large bounding sphere associated with the base node in the HISG. The bounding sphere of the base node of the hand can be used to determine the hand scene, i.e., the hand's region of influence. There are exceptions to this rule: If a linked object enters the hand scene, then all the other objects it is linked to, whether inside the bounding sphere or not, automatically are included in the hand scene. The same holds true for unlinked objects that are in contact with an object in the hand scene, as long as they remain in contact.

[0069] The HIS may also contain a State Manager which is responsible for updating the state of each of the nodes that are in the scene graph. During each cycle through the HIS, the state manager traverses the HISG tree, assigning new states (e.g., positions, velocities, etc.) to the various dynamic nodes. If the HIS is not a process running on the host computer, typically, the state manager updates the states of the HISG running in the HIS only, and not the HISG in the host application. In such a case, typically only the HISG in the HIS will have the ability to synchronize with the corresponding HISG in the host application.

[0070] A Collision Manager navigates the HISG and determines which dynamic nodes are colliding with one another. Static nodes are typically ignored by the collision manager. Support may include sphere/plane, sphere/box and sphere/sphere collisions, as well as other object collisions. The hand itself is part of the haptic scene graph and consists of multiple dynamic nodes. In one embodiment, we use multiple dynamic sphere nodes (i.e., contact spheres) in the fingers and hand to check for collision with other objects, while other embodiments may use multiple dynamic bounding cylinders on the phalanges of the fingers, and a bounding box on the palm, as an alternatives or in addition to the spheres. The collision manager has at least two objectives: to determine which two nodes (objects) are colliding, and to determine how deeply inside one another they are (i.e., the depth of penetration). The depth of penetration may be used to compute force feedback commands based on a stiffness model.

[0071] It should be noted that the nodes in the hand-interaction scene graph do not necessarily correspond exactly to the nodes in a graphics (e.g., the master) scene graph. For example, a complex shape could be rendered “as is” graphically, but from a haptic point of view, it would be represented by a bounding box.

[0072] In another embodiment, there is another Data Neutral Scene Graph (DNSG) for coordinating and linking data between the master (i.e., graphical) scene graph and the HISG. The DNSG contains a node for each interactive element or other element of interest existing in either the graphical scene graph or the HISG. Elements of interest typically include elements which might change with time. When a DNSG is employed, the DNSG is typically the scene graph which is updated with each pass through the simulation loop. (That is, in effect, it becomes the “master” scene graph.) After the DNSG is updated, it passes the update information along to the corresponding node of the graphical scene graph and/or the HISG for updating. When a DNSG is used, the HISG typically sends force commands, or local force patch approximations, to an I/O Daemon (discussed below) running on a force controller.

[0073] As discussed, the Hand Manager is responsible for dealing with all the specifics of hand interaction. The hand manager is also responsible for selecting the grasping mode and providing all manipulation algorithms. The hand manager is the process that sets the environment constraints, i.e., ensures that the graphical fingers don't penetrate a virtual object and that the graphical hand doesn't go through a graphical wall, etc. This is achieved by using two separate hand models: the ghost hand and the graphical hand. The ghost hand corresponds exactly to what is being measured by an instrumented glove and spatial tracker. The graphical hand is constrained by the objects, or other graphical impediment, in the virtual environment. When no objects or impediments are encountered, the two hands overlap completely and have the same configuration and location. When constraints or impediments are encountered, the correspondence ceases and the two hands may be modeled as linked via a network of virtual springs and dashpots. This ensures that when the user's real hand passes through a virtual wall and re-enters elsewhere, the graphical hand will be waiting at the appropriate location such that the two overlap again.

[0074] This is illustrated in FIG. 3, where for simplicity, the hand configuration is shown to remain constant. In step 1, the graphical hand is approaching a wall. In step 2, the user's hand passes through the wall, but the graphical hand stops at the constraint. In step 3, the user's hand moves sideways while the graphical hand tracks the wall. In step 4, the user's hand backs away from the constraint and both hand models overlap.

[0075] I/O Daemon

[0076] The I/O daemon runs as a background process and is responsible for communicating with the hardware devices. For example, the daemon may acquire data from a Polhemus or Ascension electromagnetic tracker, and may also specify force and vibration setpoints to the CyberGrasp or CyberTouch (i.e., another Virtual Technologies haptic-feedback product) devices, respectively. The HIS communicates with the daemon via info packets that make the physical location of the daemon transparent to the user. It may run on a separate machine, such as the CyberGrasp controller, on the host itself, or any other computer on the network. The sensor stream is the data coming from the I/O daemon. The stream may include data from the CyberGlove and various 6D position trackers. The feedback stream may include force or vibration setpoints that are sent to the I/O daemon intended for a haptic system, such as Virtual Technologies' CyberGrasp or CyberTouch, respectively. Open-loop pre-defined functions such as Jolt, Vibration, Texture, etc., may also be included in the feedback stream.

[0077] Host Application

[0078] The VirtualHand Toolkit (VHT) may be used with a high-level software simulation package. Common commercial package examples include Sense8's World Toolkit™, Division's dView™, etc. In such cases, these applications already have defined a graphical master (i.e., world) scene graph, and data must be extracted from the master scene graph to construct a hand-interaction scene graph. (Note also that a DNSG as previously described might preferably be constructed to efficiently communicate with the graphical scene graph and the HISG.) Hence, a custom plug-in to the host application is required to permit the master scene graph to communicate with a host copy of the HISG. Such a HISG plug-in is shown in the host application and represents a host-application copy of the HISG. The master scene graph and the host application HISG copy may synchronize each other using the DNSG. They may also synchronize with each other using either of the following two communication methods: poll mode or call-back mode. In poll mode, the master scene graph requests the new data from the host application HISG and updates the master states itself. In call-back mode, the master scene graph passes a function pointer to the host HISG copy which then directly updates the object states in the master scene graph. When the HIS is running on a separate computer, such as a force controller, then a “mediator” process running in the HIS lets the HISGs running in the host application and in the HIS update one another asynchronously.

[0079] Most commonly, when using the toolkit, the user will modify the Application software such that a hand-interaction scene graph may be constructed from, and/or communicate with, the master scene graph (or alternatively, the DNSG). Typically, applications using the VirtualHand Toolkit fall within three scenarios:

[0080] Scenario #1

[0081] The VHT is being used with a third party software package such as the world-building toolkits provided by Sense8 Corp., Division Inc. and Deneb Robotics Inc. In this case, the applications already have a master scene graph and data must be extracted from it to construct a hand-interaction scene graph (and/or a DNSG) residing on the host. If the HIS is running on a separate computer, such as a force controller, then this hand-interaction scene graph may be copied over to the HIS and the two are updated asynchronously. This is done because the simulation running on the host will typically run at a much slower rate than the simulation running on the dedicated CyberGrasp controller. A high servo rate is desirable for high-fidelity feedback.

[0082] Scenario #2

[0083] In this scenario, an application that is “interaction scene graph aware” is being written from the ground up using OpenGL, OpenInventor, etc. In this case, the interaction scene graph may be directly incorporated into the master scene graph.

[0084] Scenario #3

[0085] In some instances, the users might want to bypass the interaction scene graph altogether and write their own hand interaction simulation. In these instances, they can communicate directly with the I/O daemon to read data from the devices and control them directly.

[0086] Embodiment

[0087] An embodiment which illustrates the functionality of the VirtualHand Toolkit consists of a simulation of a virtual control panel (FIG. 2). The user may interact with the control panel and buttons using a CyberGlove instrumented glove, and optionally with a CyberGrasp force-feedback device. The control panel contains grounded objects and surfaces, in addition to push buttons and toggle switches. When the CyberGrasp force-feedback device is used, the user can also feel the resistance of the buttons being pressed.

[0088] FIG. 4 is a link-revolute-joint model of the human hand (400). The forearm link (401) is connected to the wrist yaw joint (402) which is connected to the wrist pitch joint (403). This joint is connected to the metacarpus link (404) which extends to connect to the thumb roll joint (405). This joint is connected to the thumb abduction joint (406) which is connected by the thumb metacarpal link (407) to the thumb metacarpal roll joint (408). This joint is connected to the thumb MP joint (409) which is connected by the thumb proximal phalanx (410) to the thumb IP joint (411), which is connected to the thumb distal phalanx (412). The hand metacarpus link (404) also connect to the joints at the base of each of the fingers. It connects to the index abduction joint (413), which connects to the index MP joint (414). The index MP joint is connected by the index proximal phalanx link (415) to the index. PI joint (416) which is connected by the index medial phalanx link (417) to the index DI joint (418) which ultimately connects to the index distal phalanx link (419). The structure is similar for each of the middle, ring and small fingers. Such a link-revolute joint hand model is described in Kramer's U.S. Pat. No. 5,482,056, “Determination of Thumb Position Using Measurements of Abduction and Rotation.” Link-revolute joint models for the entire human body are described in Kramer's U.S. Pat. No. 5,676,157, “Determination of Kinematically Constrained Multi-Articulated Structures.”

[0089] FIG. 5 provides one embodiment for constraining the virtual hand at the surface of a virtual object. The concept is as follows. Associated with the measured physical hand of the user are two computer-simulated virtual hands, one is a graphical hand which is rendered, and the other is a “ghost” hand, which is not displayed, but given the measurements of the hand reproduces as precisely as possible the movements of the physical hand. The graphical hand (500) is modeled as having springs at each of the joints (e.g., one or more of the joints in FIG. 5), in addition to a spring at the fingertip and one at the wrist. The unstretched, i.e., zero position of each joint spring (e.g., Δx10 and Δθ20, etc.) is defined by the corresponding joint angle of the ghost hand. The unstretched position of the wrist spring is defined by the position of the ghost wrist position, and the unstretched position of the fingertip spring is defined by the distance from a point about ¼ into the fingertip out to the fingertip surface when it isn't in contact with an object.

[0090] In FIG. 5, the position and orientation of the hand is measured to correspond to when the forearm attachment point (501) and the tracker position point (502) coincide with coordinate frames aligned, i.e., the corresponding orientation springs (503) and (504) are unstretched. Note that in general, the forearm attachment point and the tracker position point represent 3 degrees of freedom of orientation capability each, and thus have 3 rotary springs each. When the hand is at the tracker measurement, the tracker translation spring (505) is also unstretched and has zero length. To illustrate the tracker translation spring with spring constant k1 (a.k.a. Young's Modulus), the graphical hand is shown displaced from the measure position by x1. Note that the tracker position point may be selected to be at a variety of locations, such as the forearm link (as is shown here), the metacarpus link, or even a point distal to the physical hand. The placement of the tracker position point will affect how simulated impediments will affect the movement of the hand.

[0091] Continuing with FIG. 5, the forearm link is connected by the wrist joint (506), with rotary spring (507) to the metacarpus link (508). Actually, the wrist joint as shown typically comprises two orthogonal joints, a wrist flexion (pitch) and wrist abduction (yaw). For clarity, the double joint is shown as a single wrist flexion joint. The rotary spring associated with this joint has spring constant k2, and is unstretched at an angle of θ20. The metacarpus link is connected by the metacarpophalangeal joint (509), with corresponding rotary spring (510), to the proximal phalanx (511). As with the wrist, the metacarpophalangeal joint actually comprises two orthogonal joints, one for flexure and one for abduction. For clarity, this double joint is also shown as a single point. The rotary spring corresponding to the flexure of this joint has spring constant k3, and is unstretched at an angle of θ30. The proximal phalanx is connected to the PI joint (512), which has a rotary spring (513) of spring constant k4 and is unstretched at an angle of 040. The PI joint is also connected to the medial phalanx (514). The medial phalanx is connected by the DI joint (515) to the distal phalanx (516). The DI joint has a rotary spring (517) of spring constant k5 and is unstretched at an angle of 050. Finally, the distal phalanx has a spring (518) to the fingertip (519), which represents the compliance of the fingertip. This spring has spring constant of k6, and is unstretched when at a length of x60.

[0092] When the graphical hand contacts an object, the hand flexes such that the solid links articulate at the revolute joints in relation to the stiffnesses of the associated joint springs. FIG. 6 shows the embodiment where the graphical hand contacts a compliant surface. FIG. 6A shows the ghost hand (600), which represents the true configuration of the physical hand, positioned relative to the object surface (601), while FIG. 6B shows the graphical hand (602) ultimately drawn after it “flexes.” Note that in this example, the surface (603) is also shown to translate to the left (604) due to a modeled spring stiffness of the surface, where, in this example, the stiffness is on the order of stiffness of the hand springs. The fingertip of the hand (602) is shown to compress by an amount Δx6 (605), in addition to the other rotary joints angles also being modified. The tracker position is also shown to translate by an amount Ax, (606).

[0093] The graphical hand, which may employ virtual springs at one or more of its joints such that it may bend naturally, is typically the hand used to actually compute simulated interaction forces for the embodiment where the CyberGrasp system, or some other haptic-feedback display, is used. The hand manager may include a generalized hand-constraint procedure that solves the hand-spring model given a list of constraining impediment collisions from the collision manager. It should be obvious to one skilled in the art of mathematics and dynamic simulation how to solve the multi-articulated spring model given a set of constraints.

[0094] It has been shown that when the visual display contradicts one's proprioception about where their hand is, the visual display dominates. Thus, it is typically more disconcerting for an individual to see their hand penetrating an object they know they should be unable to penetrate, than it is for their graphical hand to stop at the surface of the object, even though the individual's physical hand actually moves slightly past.

[0095] A separate “constraint model” may be associated with each colliding digital object, built up from a variety of shape primitives for which the hand manager knows how to interpret constraints. In this way an “object filter” is created which automatically fits such shape primitives to an arbitrary digital object to generate the constraint model. Such a constraint model may also be employed as a simplification of the true digital object for collision-detection and depth-of-penetration calculations for force-feedback.

[0096] With respect to a further embodiment of the invention, it is noted that the hand-spring model provided thus far in FIGS. 5 and 6 is purely a static model, i.e., it does not account for dynamic properties of the hand or object, such as mass and damping. Thus, the response to impact between the hand and an object with mass can only be evaluated in the steady state; reactions to impacts must be ignored. FIG. 7 provides an extension of the hand-spring model to a hand-spring-mass-dashpot model for use in dynamic calculations.

[0097] In the embodiment illustrated in FIG. 7, object m7 (700) is traveling with velocity vo (701) at point po (702) with normal n (703). Point po is coupled to the lumped-mass model mass of m7 via spring (704) with spring constant k7 and dashpot (705) with damping b7. The finger (706) now has masses m3 (707), m4 (708) and m5 (709) associated with the links (710), (711) and (712) as shown, rotary springs (713) and (714) with spring constants k4, k5, and rotary dashpots (715) and (716) with damping factors b4, b5 associated with the joints as shown, and spring (717) and dashpot (718) k6 and b6 between the end of the finger solid link (712) and the contact point pF (719) at the fingertip which is traveling with velocity vF (720). With such a model, upon impact between the object and the finger, the fingertip will compress and the finger will recoil based upon the dynamics involved. Note that when the velocities (and higher order derivatives of position) are known for the fingertip and object, at a particular time step, the hand-interaction scene graph can be updated to include objects that are estimated to enter the sphere of interest and impact within the next few update cycles.

[0098] The dynamic hand-spring model of FIG. 7 provides a model for determining hand constrains and manipulation of free-floating objects which might be moving at the time of contact with the virtual hand. In such cases, one might want to have a virtual finger recoil upon impact with the object, before the finger returns to a steady-state position. An example of such a free-floating object is a baseball, coffee cup or soda can. For free-floating object manipulation, various physics-based manipulation algorithms may be employed without departing from the scope and intent of the invention. To determine a stable grasp of the free-floating object using the virtual hand requires solving a system comprising multiple objects which yield an over-constrained contact problem: The fingers and the palm area of the virtual hand are essentially six objects which interact with the object to be manipulated.

[0099] FIG. 8 show various example in which a simulated graphical hand can be constrained to flex, translate and rotate relative to the measured ghost hand based on a curved surface impediment. These and other combinations of flexure, translation and rotation are obtained by adjusting the relative stiffnesses of modeled springs corresponding to the various articulated degrees of freedom of the finger joints, wrist, tracker, etc. FIG. 8A shows the top view (800) of a simulated graphical hand controlled by a physical hand. The fingernails (801, 802, 803, 804 and 805), respectively, of the thumb (806), the index (807), the middle (808), the ring (809) and the small finger (810) are shown to provide perspective when the hand is rotated to an end view (811).

[0100] In FIG. 8A, the ghost hand separates from the simulated graphical hand (811) at the surface of the impediment (814) and moves to a first position (812). The ghost hand then translates to the side to a second position (825). The ghost hand has fingernails (815, 816, 817, 818 and 819) of the thumb (820), the index (821), the middle (822), the ring (823) and the small finger (824). Note the subtle changes in orientation of the fingernails to help understand how the measured joint angles, orientation and spatial placement of the hand are modified in each figure to produce the resulting simulated graphical hand.

[0101] FIG. 8B shows how the simulated hand moves to a new position (826) which, due to the curvature of the impediment surface, and the attractive force of the tracker translation spring shown here generically (813), is slightly closer to the ghost hand in its second position than the simulated hand was to the ghost hand in its first position. One technique to solve the model for the final configuration and spatial placement of the hand is, based on the choice of spring constants and joint-link structure, to determine what configuration and spatial placement corresponds to the lowest total spring energy state. In this example, the simulated hand is only allowed to translate along the same direction vector as originally caused the ghost hand to split off from the simulated hand.

[0102] FIG. 8C shows an embodiment similar to 8B; however, the fingers of the simulated hand are permitted to flex so that the simulated hand is closer to the ghost hand, again this position may be determined to result in a lower total spring energy state. Note that in this embodiment, the simulated hand is not allowed to modify its orientation. The dashed line (826) denotes the orientation of the simulated palm (i.e., metacarpus) from which the fingers flex to reach the surface.

[0103] FIG. 8D shows an embodiment where the simulated hand is allowed to translate along an arbitrary vector to find the shortest tracker-spring distance (i.e., disparity) to the ghost hand, thus minimizing the total spring energy. Note that again in this case, the simulated hand is not allowed to modify its orientation.

[0104] FIG. 8E shows an embodiment which combines the finger-flexing capability of 8C with the arbitrary translation direction of 8D, in addition to allowing the entire simulated hand to modify its orientation. The dashed line (827) denotes the modification of orientation angle (828) of the simulated palm from which the fingers flex further to reach the surface. The amount that the orientation is modified, relative to the amount that the spatial placement of the hand translates from the measure tracker spatial placement, relative to the amount that each joint on each finger flexes beyond the corresponding measured joint flexure, is determined by the relative spring constants, with the goal of minimizing the total spring energy.

[0105] Finally, FIG. 8F shows an embodiment where the fingers of the simulated hand are not allowed to flex, but the palm of the simulated hand is allowed to modify its orientation angle (829) in order to minimize the total spring energy.

[0106] FIG. 9 shows an embodiment where various limbs of a simulated body, typically controlled by measurements of a corresponding physical body, are constrained to not penetrate a simulated wall impediment. In FIG. 9A, both the hand (901) and foot links (902) of a simulated body (900) are penetrating a wall (903). The tracker element (904) is shown fastened to a position on the body. In FIG. 9B, the various joint angles of the simulated limbs are shown modified. Additionally, the translational position of the entire torso is shown modified by the spring (905). The resulting modification of measured values provides the result that no part of the simulated body penetrates the impediment.

[0107] FIG. 10 is an embodiment comprising a simulated inanimate object (1000), where the spatial placement of the object is measured, for instance by a sensor (1001), and the configuration of the simulated object is either measured or otherwise known. In FIG. 10A, the measurement value of the physical object causes the corresponding ghost object to penetrate a simulated wall (1002). In FIG. 10B, the measurement of the spatial placement has been modified, due to extension of a simulated spring (1004), and additionally, the object is simulated to flex, such that the simulated object no longer penetrates the wall. The flexure may be at an intended joint, or may be due to flexure of a compliant member (1003). The simulated object is shown here with rotary springs (1005 and 1006) about flexure joints (1007 and 1008).

[0108] To one skilled in the art there may be several techniques for how to formulate and solve for the configuration and spatial placement of an articulated system with allowable compliance. One way to solve such a system is to mathematically model the articulated structure with springs as shown above and then solve for the steady-state angles and positions of the links. In one embodiment, a closed-form solution exists. In another embodiment, an iterative approach might be faster to calculate or provide a more stable mathematical calculation or be more quickly solved on a data processor.

[0109] Although the joint-link-spring model provides a convenient and intuitive way to visualize how to modify measured values to produce a simulated hand (for instance) configuration and spatial placement such that it does not significantly penetrate another simulated object, the spring model is not necessary to use to develop a technique for accomplishing the same goal. Thus, in broad terms, this invention discloses a general idea and method for modifying, based on simulated impediments, measured control values to constrain a simulated hand, body part or other articulated inanimate object.

[0110] The spring model also gives an intuitive way to visualize what should happen when the simulated impediment moves or is removed. In general, the simulated hand will return to a position consistent with unmodified measured values. In general, the spring model provides a vector of attraction, so it is easier to visualize what the algorithm will do in a situation. However, the spring model may be replaced by an abstract, non-linear force vector, with properties that define how the simulated hand or object will deviate from, and then return to coincidence with, the measured hand or object in a given situation where a simulated impediment exists.

[0111] One benefit of this invention is an improved perception of interaction with solid objects by a user who is (a) not using a force-feedback device, or (b) is using a force-feedback device with inadequate stiffness.

[0112] One exemplary technique is provided below, where the technique for stopping a graphical hand from penetrating a graphical object is accomplished by calculating an offset with is added to the measured value, such that the graphical hand will not penetrate a graphical impediment. That is, the offset which is determined and added to the measured value, accounts for spring compliance.

[0113] We often refer to a ghost-hand algorithm as a method for modifying a measured value to ensure that a simulated hand (or object) does not significantly penetrate another object. The simulated hand (or object) based on the unmodified measured values is referred to as the ghost hand.

[0114] Nomenclature and Definitions

[0115] In the following sections, we use upper case roman letters to denote matrices, A, B, etc. We use lower case roman letters to denote vectors, scalars and functions. For geometric objects A and B, the local coordinate frames relative to a global origin, O, are defined by oAT, oBT, respectively, or more simply just AT and BT when it is clear that the coordinate frames are relative to origin, O. AT is a homogeneous transformation matrix which transforms vectors defined in the frame of object A to become vectors defined in the global frame, O. Frame transformations are equivalent to matrix multiplication if the transformations are represented as homogeneous 4×4 matrices and vectors are homogenous. Note that such matrices are always invertible.

[0116] Collision Manager

[0117] A fundamental component of our ghost-hand algorithm is the ability of the Collision Manager to track collision witness points on both the hand and the environment. We refer to Collision Tracking as the function which determines a coherent (i.e., continuous in time) reference surface (e.g., a continuous set of tangent planes) that separates the hand from the environment at all times. Our collision tracking algorithm relies on accurate collision detection and proximity queries.

[0118] For the case where the regions in question of objects A and B have large radii of curvature, collision tracking operates as follows: Consider two objects A (1100) and B (1101). To track object B with respect to object A, we first determine the closest points on the surface of both objects (see FIG. 11). These two points are known as witness points and will be referred to as opA (1102) and opB (1103), respectively, i.e., represented in the global inertial reference frame, O (1104). When the reference frame is understood to be the inertial reference frame, the notation will be simplified to pA and pB, respectively. The same two vectors, but represented in the local frame of each object will be denoted ApA(1105) and BpB(1106), respectively. This initial step only works if the two objects are initially disjoint. The point (1200) on the surface of object B (1201) is used to construct the surface tangent plane t(pB) (1202) (see FIG. 12). This tangent plane has a normal (1203) whose projection onto the line (1107) joining pB with pA is positive if object A and object B are disjoint. We denote to(x) (1204) and tn(x) (1205) as the offset and normal, respectively, of the tangent plane at point x, represented in the global frame.

[0119] MTD (minimum translational distance) is defined as the length of the vector that joins pB to pA if pA is exterior to object B. If pA is inside object B, the MTD is defined as the negative of the length (magnitude) of this vector.

[0120] At the beginning of each new time step, after the object positions have been updated, the collision query algorithm is run again, and new values for pA and pB are calculated. The tangent plane is updated while the MTD is greater than 0. When the point pA lies inside object B, the dot product of the line joining pB to pA with the tangent plane normal will become negative. This test is used to determine when to transition the tracking algorithm from “tangent-plane-update mode” to “witness-point-projection mode.” Another possible test is to check the sign of the MTD as returned by the collision-query algorithm.

[0121] We define a collision event as the transition from positive to negative MTD. We denote by pA0 and pB0 the witness points on objects A and B immediately before a collision event. Thus the tangent plane at a collision event is t(pB0). We cache (save) the local frame witness point ApA0=ApA at the time of the collision event (we also refer to pA0=AT ApA0)

[0122] Once a collision event has occurred, the tangent plane is no longer updated as discussed above. Instead, at the beginning of each new frame, we check the current MTD as returned by the collision detection algorithm.

[0123] If the MTD is positive, we return to the above algorithm since object A and object B are again disjoint. In addition, we use the component of pA-pB along tn(pB) as the offset vector oAn (where the n denotes “normal”). This offset vector is the ultimate output value for this collision-tracking algorithm.

[0124] If the MTD is less than or equal to zero, we calculate p′A=AT ApA0, where AT is the current frame transformation for object A, i.e., at the present time step. Since object A moves, in general AT changes also. Thus, AT will not be the same frame as at the beginning of a collision, and so the calculated vector p′A (1300) also will not correspond to pA0. The projection of p′Aonto t(pB0) (1301), Proj(p′A, t(pB0)) (1302), gives the tracked witness point for object A (1303). The projected point is the closest point on the plane, tangent to B (1306) at pB0(1304), to the current position of the contact witness point p′A. We define the witness offset as the difference Proj(p′A, t(pB0))−p′A. This offset is always a vector normal to the tangent plane, and is denoted oAn (1305). See FIG. 13.

[0125] The output of the collision tracking algorithm is an offset vector lying in the direction of the normal to the surface tangent plane of object B and, when objects A and B are disjoint, gives the displacement vector necessary for object A to just touch object B. We denote this collision-tracking algorithm by the function c operating on two objects, A and B, as oAn=c(A, B).

[0126] Ghost-Hand Algorithm—Using Only Tracker Offset

[0127] Given the collision-tracking algorithm c(A,B), we construct the tracker-offset-based ghost-hand algorithm for an articulated human-hand geometry with hand-like kinematics. We assume our hand model is constructed from a set of primitives that a collision algorithm (such as c(A,B)) can use to find witness points. Examples of this include: (1) convex polytope geometries along with a GJK (Gilbert, Johnson and Keethri)-based collision algorithm, and (2) convex NURBS geometries along with the corresponding enhanced GJK collision algorithm. Note that other geometry-algorithm pairs are possible.

[0128] An example of a convex polytope geometry is shown in FIG. 14. The hand (1400) consists of 16 convex polyhedra that represent the palm (1 polyhedron) (1401) and each finger (3 polyhedra per finger, i.e., the proximal (1402), medial (1403) and distal phalanges (1404)).

[0129] For each component of the hand geometry model, we record the offset vector. For a 16-component hand model, this results in an array of 16 offset values per frame. For scenes containing more than one non-hand object, only the closest of each pair is used for collision tracking. A collision pair is any collection of two objects, exactly one of which is a hand geometry component. The array of offsets will be denoted as OHn—j, with j=1. . . N (where N is the number of hand-geometry components).

[0130] Once the array of offsets has been obtained, we determine the differential tracker offset value for the hand. This is computed as follows:

[0131] Let J be the subset of the N finger-geometry components with MTD <=0. Then for each k in 1$j^{\prime }=\underset{jinJ}{\arg \max }\&LeftDoubleBracketingBar;o_{\mathrm{Hn\_j}\left(k\right)}\&RightDoubleBracketingBar;\mathrm{and}\mathrm{set}\Delta o_{H\left(k\right)}=o_{{\mathrm{Hn\_j}}^{\prime }\left(k\right)}.$

[0132] This result for each vector component, x, y, z, is used to update the overall tracker offset vector

O H ═O H +ΔO H .

[0133] In situations where the hand position contains a small noise component, this update is only performed if the magnitude of ΔoH is greater than some epsilon.

[0134] At each time step of the algorithm, we keep a counter ne of the number of hand components whose MTD is greater than some small positive constant, e. While ne is non-zero, the differential offset update procedure proceeds as above. While ne is zero, the update procedure is changed to

O H =p*O H ,

[0135] where p is a non-negative scalar value less than 1. This gives exponential convergence of the tracker offset to zero. In addition, if ||oH||<ec (for some small positive ec), we set oH=0 Setting p=0 gives one step convergence of the hand offset back to correspond to the measured tracker values. Additionally, after performing a convergence step, the tracker offset may be recalculated before displaying the hand, since the convergence procedure might yield a tracker offset which places the hand back into intersecting contact with the object.

[0136] The convergence technique described for the embodiment above is time based. That is, once the impediment has been removed, the simulated hand will move toward the ghost hand at a rate that is functionally related to time. In an alternate embodiment, the simulated hand moves toward the ghost hand by a distance amount that is functionally related to the distance amount that the ghost hand moves. Thus, when the physical hand stops moving, the simulated hand will stop converging. Likewise, when the simulated hand is disparate from the ghost hand, any movement of the physical hand will result in some movement of the simulated hand —a desirable feature that is perceptually satisfying. In yet another embodiment, the convergence technique causes the simulated hand to converge with the ghost hand at a rate that is both a function of time and movement of the physical hand.

[0137] Modification for Force Feedback

[0138] The above algorithm can be modified to allow for realistic force feedback per finger. We assume that force feedback is accomplished by generating a tangent plane for each finger that is used in an impedance style control loop on a force controller. We also assume that the tracker offset calculation above being done on a host computer does not directly affect the forces in any way being calculated by the force controller.

[0139] We use the above tracker-offset algorithm to determine an array of offsets for each fringer. These offsets are computed in exactly the manner discussed for the hand, using differential updates. That is:

[0140] Let i be the finger index, and let Ji be the set of finger-geometry components of finger i with MTD <=0. Then for each i, and each k in {x,y,z}, we find j′ such that 2$j^{\prime }=\underset{jin{J}_i}{\arg \max }\&LeftDoubleBracketingBar;o_{\mathrm{Hn\_j}\left(k\right)}\&RightDoubleBracketingBar;\mathrm{and}\mathrm{set}\Delta o_{\mathrm{H\_i}\left(k\right)}=o_{{\mathrm{Hn\_j}}^{\prime }\left(k\right)}.$

[0141] where i=1 . . . 5 is the finger index. The tangent plane offset value, determining the force for each finger, is calculated as

O H — i =O H — i +ΔO H — i .

[0142] The convergence of the finger offsets back to zero force may use the same convergence procedure at the hand offset convergence.

[0143] Before the force feedback tangent planes are sent to the feedback controller, the component of the finger offset in the normal direction to the tangent plane at the contact point is added to the plane offset so the controller sets the force as if the finger had actually penetrated the object by the offset oH—i.

[0144] Ghost-Hand Algorithm —Using Both Joint-Angle Offset and Tracker Offset

[0145] The algorithm discussed above results in a single vector that is used to adjust the tracker offset value for the whole hand. In this section, we discuss the extensions required to constrain fingers individually as well as the whole hand.

[0146] The joint-angle-offset algorithm discussed in this section consists of two components: (1) surface tracking and (2) an inverse-kinematic (IK) solver for each finger's joint angles.

[0147] Surface Tracking

[0148] The tracking algorithm discussed above works well for contact situations in which the contacting finger does not move significantly around inside the object during a collision. When the contacting finger moves, the tracked witness point p′A moves along the plane which is tangent to the surface at the initial contact point, pB0. This plane corresponds to the surface of a polygonal geometry for a small range around the contact point. For larger ranges, or for curved geometries, the tangent plane may become invalid in the sense that p′A is no longer close enough to the surface of object B to be a valid assumption.

[0149] For each facet of a polygonal geometry, it is possible to know the Voronoi regions that correspond to it. Using the Voronoi region yields a simple algorithm for checking if a point can be projected onto a finite planar polygonal patch, f (see FIG. 15). One way to check whether the point (1504) can be projected onto f (1500) is to find dot products onto all of the planes (1501, 1502 and 1503) perpendicular to the face that also contain an edge of the face.

[0150] Surface tracking for two objects A and B proceeds exactly as collision tracking described in the section above when the two objects are disjoint. Once a collision event occurs, the pre-contact tangent plane t(pB0) is preserved exactly as before. However, now we also find the tangent planes for all the neighbors, fj, of the contact facet f(pb0) on B, where neighbors include all facets, fj, that have a common edge or vertex with the contact facet, f. Note that each facet defines a tangent plane.

[0151] During a collision, in addition to the offset vector between the contact witness point and the contact facet, offsets for all the neighboring facets are also computed. If the witness point cannot be projected onto a facet, the offset for that facet is set to infinity. In the case where multiple objects are in contact, the offsets are the sorted to find the smallest, and this is the offset that is returned by the algorithm.

[0152] Inverse-Kinematics (IK) Solver

[0153] In this section we extend the tracker ghost-hand algorithm to cover a finger-based contact situation. If we apply the surface tracking algorithm discussed above to each finger geometry, we get a single offset vector that would place the finger on the object surface. However, each finger is a kinematically constrained linkage whose degrees of freedom are rotational joints (see FIG. 16).

[0154] The fingertip position (1600) can be given in the abduction plane (i.e., the plane defined by at least two of the three phalanges) by the following two forward-kinematic equations:

p x =d p *cos(θ1)+dm*cos(θ1+θ2)+dd*cos(θ1+θ2+θ3)

p y =d p *sin(θ1)+dm*sin(θ1+θ2)+dd*sin(θ1o+θ2+θ3),

[0155] where dp, dm and dd are the lengths of the proximal (1601), medial (1602) and distal (1603) phalanges, respectively. The angles θ1 (1604), θ2 (1605) and θ3 (1606) are measured joint angles.

[0156] The surface tracking algorithm gives us a desired change in position (i.e., offset, o)

[0157] (1700) for each fingertip (see FIGS. 17A and 17B). To convert this into corresponding change in joint angles, we compute the Jacobian, J, of the above equation. To determine the individual joint angle offsets from the desired change in fingertip position, p, we solve the following equation for Θ,

J Θ=o,

[0158] where Θ is the vector (Δθ1, Δθ2, Δθ3)T of angle modifications for the three angles (1701, 1702 and 1703) and O (1700) is the desired surface offset vector. To bias the various joint angles in the solution, much in the same way that we would by using different spring stiffnesses in the spring model, we may set Θ=(w1.Δθ1, w2.Δθ2, w3.ΔΘ3)T, where w1, w2 and w3 are weighting (biasing) matrices. The matrix J is singular, and to invert we may use the Moore-Penrose pseudo inverse. Another possible solution technique is the minimum-energy least-squares approach (which might have some stability problems). The inverse may also be calculated using singular-value decomposition (SVD).

[0159] Once new joint-angle offsets have been computed, they are added to the current joint angle values. If the new joint angle value is outside of the joint limits, i.e., the range of allowable joint-angle values, it is simply clamped there. Once this is done for all three joints, it is possible that (due to joint limits), the finger is still penetrating the object.

[0160] After all fingers have been constrained in this way, there will in general be 5 offset vectors that indicate how far the IK solver was from solving the offset problem. These offsets are used as inputs to the tracker offset algorithm discussed in the previous section. Once the tracker is offset, a new “desired offset” vector for each finger may be calculated, and a new Θ vector may correspondingly be determined for each finger.

[0161] Even in the case where all 5 fingers can be offset within their joint limits, it may be desirable to distribute some of the desired finger offset toward repositioning the entire hand. One way to accomplish this is by adding all finger x-axis offsets to yield an overall x-axis offset for the hand. Y-axis and z-axis offsets for the hand are determined similarly. Weighting factors, wx, wy and wz, typically <1, may also be used to choose an overall hand offset which is proportional to the summed offsets.

[0162] FIG. 18 shows an embodiment where a physical hand (1800) is wearing an instrumented glove (1801), such as a Virtual Technologies, Inc. CyberGlove. The glove is connected via a wire or wireless link (1807) to an instrumentation unit (1802) which provides digitized values to the computer system (1803), which comprises at least a CPU and memory. The computer displays a graphical image (1804) of the measured physical hand on the computer monitor (1805). As the physical hand moves downward, the graphical hand collides with the simulated graphical table (1806) on the screen. Using the inventive step provided by the subject invention, the computer calculates the modifications for the hand-measurement data necessary to prevent the simulated hand from penetrating the simulated table.

[0163] The inventive structure and method provides an unprecedented level for computer-simulated virtual hand interaction with graphical objects. The hand-constraint technology provides significant commercial potential wherever humans use their hands to interact with digital objects. A generic hand-based simulator is the panacea of virtual reality and has unfathomable commercial potential, limited only by the imagination. Some of the commercial products and services being contemplated and some corresponding market niches include:

[0164] 1. 3D computer operating systems

[0165] 2. Computer games

[0166] 3. Training of factory employees in assembly line operation.

[0167] 4. Simulation of aircraft cockpits and airport control towers.

[0168] 5. Virtual prototyping of car interiors and ergonomic analysis of proposed designs.

[0169] 6. Design and usability testing of a multitude of control panels.

[0170] 7. Training and performance testing of NASA, Navy, Air Force, Army and other personnel.

[0171] 8. Rapid prototyping.

[0172] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference.

[0173] The foregoing descriptions of specific embodiments of the present invention have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed, and obviously many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, to thereby enable others skilled in the art to best use the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.

Claims

1-8. (Canceled)

9. A system for moving a simulated-multi-articulated structure in relation to the movement of an analogous physical-multi-articulated structure, where said simulated-multi-articulated structure moves in a simulated environment comprising a simulated impediment to free motion, where said physical-multi-articulated structure moves in an environment lacking an analogous physical impediment, said system comprising: a device for measuring the configuration of said physical-multi-articulated structure and the spatial placement of said physical-multi-articulated structure relative to an inertial reference frame and providing digitized signals associated with the configuration and spatial placement; and a data processor, comprising data related to the spatial placement of said simulated impediment and constraints of said simulated impediment and said simulated-multi-articulated structure, for receiving said digitized signals and modifying said digitized signals using said data to generate a set of modified signals specifying the configuration and spatial placement of said simulated-multi-articulated structure, such that when said simulated-multi-articulated structure encounters said simulated impediment, the configuration and spatial placement of said simulated-multi-articulated structure is in part determined by the constraints causing said simulated-multi-articulated structure to flex.

10. A system for moving a simulated hand in relation to the movement of a physical hand, where said simulated hand moves in a simulated environment comprising a simulated impediment to free motion, where said physical hand moves in an environment lacking an analogous physical impediment, said system comprising: a device for measuring the configuration of said physical hand and the spatial placement of said physical hand relative to an inertial reference frame and providing digitized signals associated with the configuration and spatial placement; and a data processor, comprising data related to the spatial placement of said simulated impediment and constraints of said simulated impediment and said simulated hand, for receiving said digitized signals and modifying said digitized signals using said data to generate a set of modified signals specifying the configuration and spatial placement of said simulated hand, such that when said simulated hand encounters said simulated impediment, the configuration and spatial placement of said simulated hand is in part determined by the constraints causing said simulated hand to flex.

11. A system according to claim 10, wherein said device comprises goniometers for measuring the angles of the joints of a physical hand, a tracking device for measuring the spatial placement of said physical hand relative to an inertial reference frame and means for mounting said device on said physical hand, said device providing digitized measured signals associated with the angles and the spatial placement; and said data processor producing modified signals from said digitized measured signals and said constraints using a simulated spring attached between first and second simulated hands, where the angles and placement, of said first simulated hand uses said digitized measured signals and the angles and placement of said second simulated hand uses said modified signals and said first and second simulated hands are superimposed in the absence of said second simulated hand encountering said simulated impediment; such that when said second simulated hand encounters said impediment, said second simulated hand flexes and is displaced from said first simulated hand and realigns with said first simulated hand when said impediment is removed.

12. A system for moving a simulated graphical hand in relation to the movement of a physical hand, where said simulated graphical hand moves in a simulated environment comprising a simulated graphical impediment to free motion, where said physical hand moves in an environment lacking an analogous physical impediment, said system comprising: a device comprising goniometers for measuring the angles of the joints of said physical hand, a tracking device for measuring the spatial placement of said physical hand relative to an inertial reference frame and means for mounting said device on said physical hand, said device providing digitized measured signals associated with the angles and the spatial placement; and a data processor, comprising data related to the spatial placement of said simulated impediment and constraints of said simulated impediment and said simulated graphical hand, for producing modified signals from said digitized measured signals using said data and a simulated spring mass dashpot attached between first and second simulated hands, where the angles and placement of said first simulated hand uses said digitized measured signals and the angles and placement of said second simulated hand uses said modified signals and said first and second simulated hands are superimposed in the absence of said second simulated hand encountering said simulated graphical impediment, such that when said second simulated hand encounters said simulated graphical impediment, said second simulated hand is flexed and displaced from said first simulated hand and realigns with said first simulated hand when said simulated graphical impediment is removed at a rate regulated by said spring mass dashpot, said second simulated hand being depicted graphically as said simulated graphical hand.

13. A system for moving a simulated graphical hand in relation to the movement of a physical hand, where said simulated graphical hand moves in a simulated environment comprising a simulated graphical impediment to free motion, where said physical, hand moves in an environment lacking analogous physical impediments, said system comprising: a device comprising goniometers for measuring the angles of the joints of said physical hand, a tracking device for measuring the spatial placement of said physical hand relative to an inertial reference frame and means for mounting said device on said physical hand, said device providing digitized measured signals associated with the angles and the spatial placement; and a data processor, comprising data related to the spatial placement of said simulated graphical impediment and constraints of said simulated graphical impediment and said simulated graphical hand, for producing modified signals from said digitized measured signals using said data and a simulated spring attached between first and second simulated hands, where the angles and placement of said first simulated hand ins said digitized measured signals and the angles and placement of said second simulated hand uses said modified signals and said first and second simulated hands are superimposed in the absence of said second simulated hand encountering said simulated impediment; such that when said second simulated hand encounters said simulated graphical impediment, said second simulated hand is flexed and displaced from said first simulated hand and when said simulated graphical impediment is removed, becomes superimposed with said first simulated hand at a rate in proportion to the movement of said first simulated hand, said second simulated hand being depicted graphically as said simulated graphical hand.

14. A method for moving a simulated-multi-articulated structure in a graphical environment, comprising an impediment to free motion, in relation to the movement of an analogous physical-multi-articulated structure in a physical environment lacking said impediment, said method comprising steps of: generating digitized signals representing the configuration and spatial placement of said physical-multi-articulated structure; transferring said digitized signals to a data processor; recording, as data, in said data processor, the spatial placement of said impediment and constraints of said impediment and said simulated-multi-articulated structure; generating modified signals from said-digitized signals and said data specifying the configuration and spatial placement of said simulated-multi-articulated structure; and impeding the free motion of said simulated-multi-articulated structure when said simulated-multi-articulated structure encounters said impediment causing said simulated-multi-articulated structure to flex.

15. A computer program product for use in conjunction with a computer system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer, program mechanism, comprising: a program module that moves a simulated-multi-articulated structure in, a graphical environment having an impediment to free motion in relation to the movement of an analogous physical-multi-articulated structure in a physical environment lacking said impediment, said program module including instructions for: generating first data representing the configuration and spatial placement of said physical-multi-articulated structure; recording as second data the spatial placement of said impediment and constraints of said impediment and said simulated-multi-articulated structure; generating third data from said first data and said second data specifying the configuration and spatial placement of said simulated-multi-articulated structure; and impeding the free motion of said simulated-multi-articulated structure when said simulated-multi-articulated structure encounters said impediment causing said simulated-multi-articulated structure to flex.

16. A system for moving a simulated structure in relation to the movement of an analogous physical structure, where said simulated structure moves in a simulated environment comprising a simulated impediment to free motion, where said physical structure moves in an environment lacking an analogous physical impediment, said system comprising: a device measuring the configuration of said physical structure and the spatial placement of said physical structure relative to a reference frame and providing information associated with the configuration and spatial placement; and a processor, comprising values related to the constraints of said simulated structure and said simulated impediment, receiving said information and modifying said information using said values to generate a set of modified signal's specifying the configuration and spatial placement of said simulated structure, such that the free motion of the simulated structure is impeded when encountering said simulated impediment causing said simulated structure to flex.

17. A device, comprising: a multi-articulated structure; and a sensor coupled to the multi-articulated structure, the sensor configured to determine a configuration of the multi-articulated structure and a spatial placement of the multi-articulated structure relative to an inertial reference frame, said sensor further configured to transmit a signal associated with the configuration and the spatial placement of the multi-articulated structure, the signal being configured to generate a set of modified signals specifying data values associated with a configuration and a spatial placement of a simulated multi-articulated structure displayed in a graphical environment, the set of modified signals configured to deform the simulated multi-articulated structure when a simulated interaction occurs between the simulated multi-articulated structure and a simulated object.

18. The device of claim 17, wherein said sensor includes a goniometer configured to determine at least one angle of a joint of the multi-articulated structure.

19. The device of claim 17, wherein said sensor includes a tracking device configured to determine the spatial placement of the multi-articulated structure relative to the inertial reference frame.

20. The device of claim 17, the set of modified signals being a first set of modified signals, the simulated multi-articulated structure being a first simulated multi-articulated structure, wherein the signal associated with the configuration and the spatial placement of the multi-articulated structure is further configured to generate a second set of modified signals specifying data values associated with a configuration and a spatial placement of a second simulated multi-articulated structure.

21. The device of claim 20, the first simulated multi-articulated structure being superimposed on the second simulated multi-articulated structure, wherein when the simulated interaction occurs between the simulated multi-articulated structure and the simulated object, the first simulated multi-articulated structure is separated from the second simulated multi-articulated structure.

22. A method, comprising: determining a configuration and a position of a multi-articulated structure; updating data values associated with a simulated-multi-articulated structure based on the configuration, the position, and a simulated interaction of the simulated-multi-articulated structure with a simulated graphical object; and simulating a flex of the simulated multi-articulated structure without a corresponding flex in the multi-articulated structure.