Grasp Planning Of Unknown Object For Digital Human Model

Abstract

An embodiment receives models of an object and an environment and an indication of position of a digital human model (DHM). An oriented bounding box (with a plurality of faces) surrounding the model of the object is determined and, for each of the plurality of faces, a candidate grasp location, a candidate grasp orientation, and a candidate grasp type is determined. From amongst the plurality of faces, one or more graspable faces is determined based on: the candidate grasp locations, the candidate grasp orientations, the environment model, and dimensions of each face. Then, an optimal graspable face is identified based on a hierarchy and the position of the DHM. An inverse kinematic solver determines position and orientation, i.e., grasp, of an end effector of the DHM grasping the object based on the candidate grasp location, candidate grasp orientation, and candidate grasp type of the optimal graspable face.

Description

BACKGROUND

A number of existing product and simulation systems are offered on the market for the design and simulation of objects, e.g., humans, parts, and assemblies of parts, amongst other examples. Such systems typically employ computer aided design (CAD) and/or computer aided engineering (CAE) programs. These systems allow a user to construct, manipulate, and simulate complex three-dimensional models of objects or assemblies of objects. These CAD and CAE systems, thus, provide a representation of modeled objects using edges, lines, faces, polygons, or closed volumes. Lines, edges, faces, polygons, and closed volumes may be represented in various manners, e.g., non-uniform rational basis-splines (NURBS).

CAD systems manage parts or assemblies of parts of modeled objects, which are mainly specifications of geometry. In particular, CAD files contain specifications, from which geometry is generated. From geometry, a representation is generated. Specifications, geometries, and representations may be stored in a single CAD file or multiple CAD files. CAD systems include graphic tools for representing the modeled objects to designers; these tools are dedicated to the display of complex objects. For example, an assembly may contain thousands of parts. A CAD system can be used to manage models of objects, which are stored in electronic files.

CAD and CAE systems use of a variety of CAD and CAE models to represent objects. These models may be programmed in such a way that the models have the properties (e.g., physical, material, or other physics based) of the underlying real-world object or objects that the models represent. CAD/CAE models may be used to perform simulations of the real-word objects that the models represent.

SUMMARY

Simulating a human interacting with an object is a common simulation task implemented and performed by CAD and CAE systems. Performing these simulations requires setting grasping parameters. These parameters include the locations where the human model grasps the object model and the finger positioning on that object (i.e., the grasp itself). For instance, instantiating and positioning a digital human model (DHM) in a scene to simulate a manufacturing task typically requires specifying how to grasp the object(s) being manufactured, e.g., assembled.

While grasp is a popular topic in the field of digital human modeling, no solution exists which can automatically determine grasping for objects, e.g., unknown objects, while accounting for posture of the DHM performing the grasping.

An embodiment provides a grasp planner for unknown objects grasped by a DHM. Such a grasp planner takes into account final DHM posture when choosing the preferred grasp. This is particularly useful to achieve plausible DHM posture. Embodiments may be implemented in existing ergonomics frameworks, such as the Smart Posturing Engine (SPE™) framework available from Dassault Systemes, which automatically places and postures a DHM in a 3D environment, and focuses on grasping objects in virtual manufacturing contexts. Moreover, embodiments can also be implemented in existing ergonomics applications such as Dassault Systèmes'/DELMIA's “Ergonomic Workplace Design” application that helps manufacturing engineers design safe and efficient workplaces.

Another embodiment is directed to a computer-implemented method of determining position and orientation of an end effector of a DHM for grasping an object. Such an embodiment begins by receiving (i) a computer-based model of an object, (ii) a computer-based model of an environment, and (iii) an indication of position of a DHM in the environment. Next, an oriented bounding box surrounding the received model of the object is determined, where the oriented bounding box includes a plurality of faces. For each of the plurality of faces, a candidate grasp location, a candidate grasp orientation, and a candidate grasp type are determined and, then, from amongst the plurality of faces, one or more graspable faces is determined based on: (a) the determined candidate grasp location of each face, (b) the determined candidate grasp orientation of each face, (c) the received model of the environment, and (d) dimensions of each face. From amongst the determined one or more graspable faces, an optimal graspable face is identified based on a predetermined grasping hierarchy and the received indication of position of the DHM in the environment. An inverse kinematic solver is then utilized to determine position and orientation of an end effector of the DHM grasping the object based on the determined candidate grasp location, the determined candidate grasp orientation, and the determined candidate grasp type of the determined optimal graspable face.

According to an embodiment, determining the oriented bounding box comprises determining a minimum bounding box surrounding the received model of the object and determining a principal axis of inertia of the object based on the received model of the object. Such an embodiment orients the determined minimum bounding box based on the determined principal axis of inertia and sets the oriented minimum bounding box as the oriented bounding box surrounding the received model of the object. Yet another embodiment determines a candidate grasp orientation for a given face of the plurality of faces by setting the candidate grasp orientation for the given face based on the determined principal axis of inertia of the object.

An embodiment determines a candidate grasp location for a given face of the plurality of faces by, first, calculating a geometrical center of the object based on the received model of the object. Such an embodiment then projects from the calculated geometrical center of the object to the given face and sets location of an intersection of the projection and the given face as the candidate grasp location for the given face.

Another embodiment determines a candidate grasp type for a given face of the plurality of faces by calculating length of a first edge and a second edge of the given face, wherein the first edge and the second edge are perpendicular to each other. Such an embodiment also calculates length of a face edge normal to the first edge and the second edge. In turn, the candidate grasp type for the given face is determined based on: (i) the calculated length of the first edge, (ii) the calculated length of the second edge, and (iii) the calculated length of the face edge normal to the first edge and the second edge.

According to an embodiment, each determined candidate grasp type is one of: a pinch type, a medium-wrap type, and a precision sphere type.

As noted above, an embodiment determines one or more graspable faces based on: (a) the determined candidate grasp location of each face, (b) the determined candidate grasp orientation of each face, (c) the received model of the environment, and (d) the dimensions of each face. According to an embodiment, such an embodiment identifies a given face as a graspable face if (i) the end effector of the DHM, at the determined candidate grasp location in the determined candidate grasp orientation, does not collide with an element in the model of the environment and (ii) dimensions of the given face do not exceed a threshold.

In yet another embodiment, the DHM includes a left end effector and a right end effector. Such an embodiment may further include receiving an indication of the end effector, from amongst the left end effector and the right end effector, of the DHM grasping the object. This indication may be used to select the predetermined grasping hierarchy.

Embodiments may also configure the inverse kinematic solver. For instance, one such embodiment configures the inverse kinematic solver to have an unconstrained rotation degree of freedom along an axis normal to the determined optimal graspable face.

Another embodiment applies a respective label to each face of the plurality of faces. In one such embodiment, the respective label of each face is a function of position of the DHM in relation to the face. In such an embodiment, the predetermined grasp hierarchy may indicate a preferred order of graspable faces as a function of each respective label.

Embodiments can simulate physical interaction between the DHM and the object using the determined position and orientation of the end effector. Such functionality can be used to design, amongst other examples, real-world manufacturing lines, and modify/improve real-world environments to improve, for instance, ergonomics.

Yet another embodiment is directed to a system that includes a processor and a memory with computer code instructions stored thereon. In such an embodiment, the processor and the memory, with the computer code instructions, are configured to cause the system to implement any embodiments or combination of embodiments described herein.

Another embodiment is directed to a cloud computing implementation for determining position and orientation of an end effector of a DHM for grasping an object. Such an embodiment is directed to a computer program product executed by a server in communication across a network with one or more clients. The computer program product comprises program instructions which, when executed by a processor, causes the processor to implement any embodiments or combination of embodiments described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing will be apparent from the following more particular description of example embodiments, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments.

FIG. 1 is a flowchart of a method for determining position and orientation of an end effector of a DHM for grasping an object according to an embodiment.

FIG. 2 illustrates a computer-based model of an environment that may be utilized in embodiments.

FIG. 3A is an example computer-based model of an object that may be used in embodiments.

FIG. 3B is an exploded view of the object of FIG. 3A.

FIGS. 4A-B illustrate inputs that may be employed by embodiments.

FIG. 5 is a flowchart of a method for determining grasp according to an embodiment.

FIG. 6 illustrates steps of a method for determining a bounding box that may be implemented in embodiments.

FIG. 7 depicts example candidate grasping locations that may be determined by embodiments.

FIG. 8 depicts example candidate grasping orientations that may be determined by embodiments.

FIG. 9 is a table showing grasp types that may be determined by embodiments.

FIG. 10 depicts example end effector configurations that may be employed in embodiments.

FIG. 11 illustrates functionality of characterizing faces of a bounding box according to an embodiment.

FIG. 12 depicts a bounding box labeling technique that may be implemented in embodiments.

FIG. 13 depicts steps of a method for identifying graspable faces according to an embodiment.

FIG. 14 illustrates steps of an inverse kinematic solver determining a grasp according to an embodiment.

FIGS. 15A-D depict grasping results determined using embodiments.

FIG. 16 is a simplified diagram of a computer system for determining position and orientation of an end effector of a DHM for grasping an object according to an embodiment.

FIG. 17 is a simplified diagram of a computer network environment in which embodiments of the present invention may be implemented.

DETAILED DESCRIPTION

A description of example embodiments follows.

Digital Human Models (DHMs) offer the unique possibility to simulate worker tasks in a three-dimensional (3D) environment. This is particularly useful in the manufacturing world because such simulations allow users to, amongst other examples, detect ergonomic problems before production lines are built and detect and correct ergonomic problems in existing production lines. This does not replace traditional ergonomics, but can help detect problems in the virtual stage of the design phase to avoid costly changes on the production line in the real world.

Today, different DHMs are available in commercial products: DELMIA Ergonomics (Dassault Systemes), Jack™ (Badler 1999), and Santos® Pro (VSR 2004). Zhou (2009) explained that the biggest challenge in DHM applications is the low efficiency of the manikin positioning in 3D, due to the time-consuming processes of manual posture creation and moving each joint separately. Jack (Cort 2019) and IMMA (Hanson 2014) proposed methods to automatically posture a manikin in a 3D environment. However, the posture prediction process in these existing methods is not fully automatic because the manikin must be placed close to the object by the user before resolving the posture. However, this is a step forward to reduce the time the manikin posture creation phase takes.

Dassault Systèmes released an application called “Ergonomic Workplace Design” (EWD) that helps manufacturing engineers design safe and efficient workplaces in 3D. The Smart Posture Engine (SPE™) technology was developed to reach that particular goal. The SPE is a framework that performs an autonomous posturing of a DHM based on minimal user inputs (Lemieux 2017), (Lemieux 2016), (Zeighami 2019).

Embodiments, which can be implemented as part of the SPE™ focus on the grasp planning portion of automatic posture generation. Bohg (2013) divided the grasp problem into three categories based on whether the object to grasp is: (1) known, (2) familiar, or (3) unknown. Known objects are previously encountered objects for which grasps have been previously generated. Familiar objects are new objects that can be grasped in a similar way to a known object. Unknown objects are objects for which there is no prior grasp experience.

As explained by (Zhou 2009), grasp planners typically try to find the best hand location on the object without considering the final DHM posture. Such methods often produce results with unrealistic final postures when reaching for the object.

A grasping algorithm was described in Bourret 2019 to automatically grasp tools that were considered known objects. The objective of this tool grasping algorithm was to have a better DHM posture when grasping the tools by allowing range of motion to the hand on the object. A method has also been proposed to automatically find grasping cues on familiar tools, so as to allow the grasp planner to grasp familiar objects automatically (Macloud 2019) (Macloud 2021).

Embodiments introduce a complementary grasp planner for grasping, e.g., with a single hand, unknown objects, which may be referred to herein as “parts”. Like methods used for known and familiar objects, embodiments provide a grasp planner that accounts for different aspects of the DHM final posture when choosing the proper way to grasp the unknown object. Amongst other applications, embodiments determine a visually plausible grasp on unknown objects in a manufacturing context.

FIG. 1 is a flowchart of a computer-implemented method 100 for determining position and orientation of an end effector of a DHM for grasping an object according to an embodiment.

The method 100 starts at step 101 by receiving (i) a computer-based model of an object, (ii) a computer-based model of an environment, and (iii) an indication of position of a DHM in the environment. Next, at step 102, an oriented bounding box surrounding the received model of the object is determined. In such an embodiment, the determined oriented bounding box includes a plurality of faces. In turn, at step 103, for each of the plurality of faces, a candidate grasp location, a candidate grasp orientation, and a candidate grasp type are determined. Then, at step 104, from amongst the plurality of faces, one or more graspable faces is determined based on: (a) the determined candidate grasp location of each face, (b) the determined candidate grasp orientation of each face, (c) the received model of the environment, and (d) dimensions of each face. From amongst the determined one or more graspable faces, an optimal graspable face is identified at step 105 based on a predetermined grasping hierarchy and the received indication of position of the DHM in the environment. An inverse kinematic solver is then utilized at step 106 to determine position and orientation of an end effector of the DHM grasping the object based on the determined candidate grasp location, the determined candidate grasp orientation, and the determined candidate grasp type of the determined optimal graspable face.

The method 100 is computer-implemented and, as such, the models and indication received at step 101 may be received from any memory or other such data source that is communicatively coupled or capable of being communicatively coupled to the processor(s) implementing the method 100. In embodiments, the model received at step 101 may be any computer-based models known in the art. For instance, according to an embodiment, the model of the object and the model of the environment are each CAD models. Moreover, the indication of position received at step 101 indicates location of the DHM in the three-dimensional space of the environment as represented by the model of the environment. FIGS. 4A-B, as described hereinbelow, illustrate example input data that may be received at step 101 of the method 100. Further, the models and position indication received at step 101 may be based on real-world measurements of an object and environment. In such an embodiment, the method 100 may be used to evaluate the real-world interaction between a human and the object in the real-world environment.

According to an embodiment of the method 100, determining the oriented bounding box at step 102 comprises determining a minimum bounding box surrounding the received model of the object and determining a principal axis of inertia of the object based on the received model of the object. Such an embodiment, at step 102, orients the determined minimum bounding box based on the determined principal axis of inertia and sets the oriented minimum bounding box as the oriented bounding box surrounding the received model of the object. In an embodiment of the method 100, the oriented bounding box is determined at step 102 using the functionality described hereinbelow in relation to FIG. 6. For instance, such an embodiment may determine each principal axis of inertia of the object and orient the bounding box based upon each principal axis of inertia.

Step 103 of the method 100 determines a candidate grasp location, a candidate grasp orientation, and a candidate grasp type for each face of the bounding box determined at step 102.

In an embodiment, a candidate grasp orientation for a given face of the plurality of faces is determined at step 103 by setting the candidate grasp orientation for the given face based on a determined principal axis of inertia of the object. Another embodiment of the method 100 implements the functionality described hereinbelow in relation to FIG. 8, at step 103, to determine the candidate grasp orientation of each face.

An example implementation of the method 100 determines a candidate grasp location for a given face of the plurality of faces at step 103 by, first, calculating a geometrical center of the object based on the model of the object received at step 101. Such an embodiment projects from the calculated geometrical center of the object to the given face and sets location of an intersection of the projection and the given face as the candidate grasp location for the given face. Such functionality may be implemented for each face of the plurality of faces of the bounding box. In an example embodiment, candidate grasp locations are determined at step 103 utilizing the functionality described hereinbelow in relation to FIG. 7.

Embodiments of the method 100 may identify, at step 103, one of a plurality of different grasp types for each face. FIG. 9, described hereinbelow, illustrates example candidate grasp types that may be determined at step 103. According to an embodiment, each candidate grasp type determined at step 103 is one of: a pinch type, a medium-wrap type, and a precision sphere type. Moreover, it is noted that embodiments are not limited to the foregoing grasp-types and embodiments, at step 103, may determine candidate grasps of any type known in the art.

Another embodiment of the method 100 determines a candidate grasp type for a given face of the plurality of faces at step 103 by calculating length of a first edge and a second edge of the given face and calculating length of a face edge normal to the first edge and the second edge. In such an embodiment, the first edge and the second edge are perpendicular to each other. In turn, the candidate grasp type for the given face is determined at step 103 based on: (i) the calculated length of the first edge, (ii) the calculated length of the second edge, and (iii) the calculated length of the face edge normal to the first edge and the second edge. An example of such functionality is described hereinbelow in relation to FIG. 11.

At step 104, the method 100 determines one or more graspable faces based on: (a) the determined candidate grasp location of each face, (b) the determined candidate grasp orientation of each face, (c) the received model of the environment, and (d) the dimensions of each face. According to an embodiment of the method 100, the determining at step 104 identifies a given face as a graspable face if (i) the end effector of the DHM, at the determined candidate grasp location in the determined candidate grasp orientation, does not collide with an element in the model of the environment and (ii) dimensions of the given face do not exceed a threshold. An embodiment of the method 100 implements the functionality described hereinbelow in relation to FIG. 13 at step 104 to determine one or more graspable faces.

At step 105, the method 100 determines an optimal graspable face based on a predetermined grasping hierarchy and the received indication of position of the DHM in the environment. Table 1, described herein below, is an example hierarchy that may be used in embodiments. According to an embodiment, the indicated position of the DHM dictates the hierarchy that is utilized at step 105 to determine the optimal graspable face.

In yet another embodiment of the method 100, the DHM includes a left end effector and a right end effector. Such an embodiment may further include receiving, e.g., at step 101, an indication of the end effector, from amongst the left end effector and the right end effector, of the DHM grasping the object. Such an embodiment may select the predetermined grasping hierarchy used at step 105 based on the received indication of the end effector. In other words, such an embodiment uses a different hierarchy depending on the end effector (right or left) performing the grasping.

Another embodiment of the method 100 applies a respective label to each face of the plurality of faces. In such an embodiment, each label is a function of position of the DHM in relation to the face. In such an embodiment, the predetermined grasp hierarchy utilized at step 105 indicates a preferred order of graspable faces as a function of the labels. This hierarchy can be used to select the optimal face as a function of each respective label. An example of such functionality is described hereinbelow in relation to FIG. 12.

Embodiments of the method 100 may configure the inverse kinematic solver used at step 106. For instance, one such embodiment configures the inverse kinematic solver to have an unconstrained rotation degree of freedom along an axis normal to the determined optimal graspable face. FIG. 14 illustrates functionality of an inverse kinematic solver that may be implemented at step 106 to determine the position and orientation of the end effector of the DHM grasping the object.

Yet another example embodiment of the method 100 simulates physical interaction between the DHM and the object using the determined position and orientation of the end effector. Results of such a simulation may, amongst other examples, be used to improve ergonomics for a human in a real-world environment. For instance, if the method 100 is implemented during the design stage of a manufacturing line, results of the simulation may be used to improve ergonomics in the design and ultimately the real-world manufacturing line that is built. Similarly, the method 100 can be used to evaluate an existing real-world manufacturing line. In such an embodiment, the models received at step 101 are based on measurements of the real-world manufacturing line and a simulation performed using the determined grasp from step 106 indicates behavior of the human in the real-world environment. The determined behavior may, for instance, indicate that there is an ergonomics issue with the manufacturing line and a shelf should be lowered so that the human can more easily grasp the object. In this way, embodiments can be used to improve real-world environments.

Virtual Environment Example

Amongst other examples, embodiments provide methodologies to determine grasps of unknown objects in manufacturing contexts. One such example context is the production line environment 220 illustrated in FIG. 2. According to an embodiment, the object for which the grasp is determined is a part (e.g., one of the parts 332 shown in the exploded view 331 of FIG. 3B) that composes a product (e.g., the product 330 of FIG. 3A) assembled on a production line (e.g., the production line environment 220). Applications of grasp planning methodologies applied to the production line environment 220 of FIG. 2 where the product 330 of FIG. 3A (or one of the parts 332 of FIG. 3B) is grasped are described throughout this document to explain and illustrate embodiments.

Inputs And Outputs

The inputs of an embodiment are: a 3D model of an object to grasp, a 3D model of an environment, and an indication (e.g., 3D coordinates) of initial position of a DHM in the 3D environment. FIG. 4A illustrates an example model 440 of an object to grasp and FIG. 4B illustrates an example model 441 of an environment, e.g., a production line. FIG. 4B also depicts an initial position 442 of the DHM in the environment model 441. In an embodiment, this initial position 442 is automatically determined using functionality described in U.S. Patent Publication No. 2023/0021942 A1. In other embodiments, the position 442 is determined using one of a variety of different options, including: functionality provided by the 3DExperience platform, a user selected existing method for setting DHM positions, or setting of the position manually. Embodiments may also receive, as input, an indication of which DHM end effector (e.g., right hand or left hand) is used to grasp the object.

The outputs of embodiments may include an indication grasp type to use and a grasp target, e.g., position and orientation of an end effector. This grasp type and grasp target can be used in a DHM posture solving method, such as an inverse kinematic method, which is an element of the SPE framework, to determine position and orientation of the upper limb end effector (i.e., the hand). According to an embodiment, the end effector can reach the target on the object with the DHM using an inverse kinematic solver.

Example Method

FIG. 5 is a flowchart for determining grasp according to an embodiment. The method 550 begins at step 551 by determining a bounding box of the object to be grasped and determining candidate grasp target locations and orientations. At step 552, grasp types are determined for each of the candidate grasp target locations and, at step 553, graspable faces of the bounding box are identified. The graspable faces determined at step 553 are then ranked at step 554 to determine an optimal graspable face. This optimal graspable face is used at step 555 to execute the grasp and determine the position and orientation of the end effector.

Bounding Box And Target Calculation

Embodiments, e.g., at step 551 of the method 550, approximate the object to be grasped using the object's minimum oriented bounding box. FIG. 6 illustrates a method 660 for determining an object's minimum oriented bounding box according to an embodiment. The method 660 begins with a model 440 of the object to be grasped. Next, the principal axes of inertia 662a-c of the object are identified. In an embodiment, the principal axes 662a-c are determined using methods known to those of skill in the art, such as functionality available in the 3DExperience platform. For instance, in an embodiment, the principal axes of inertia 662a-c are axes orthogonal to a bounding box of the model 440 or are based on eigenvectors of the model 440. In turn, the bounding box 663 is oriented along the principal axes of inertia 662a-c. In this way, the method 660 determines the minimum oriented bounding box 663 to approximate the object 440. It is noted that in an embodiment, the bounding box 663 is determined using procedures known to those of skill in the art, such as functionality provided by the 3DExperience platform.

Embodiments use the determined bounding box 663 and associate, e.g., in computer memory, a potential grasp target with each face of the bounding box 663. FIG. 7 illustrates an example where the candidate grasping locations 774a-f are determined for the object 440 using the bounding box 663. In an embodiment, the geometrical center 775 of the object 440 is calculated. According to an embodiment, the geometrical center 775 is determined using procedures known to those of skill in the art, such as functionality provided by the 3DExperience platform. Further, in an embodiment, the geometrical center 775 is a centroid of external 3D coordinates of vertices of the bounding box 663. To continue, the geometrical center 775 is then projected on each of the faces of the bounding box 663. The intersection of the projections from the geometrical center 775 with the faces are the candidate target grasp locations 774a-f Such an embodiment follows the heuristic that humans prefer to grasp an object close to the object's center of mass, likely to reduce effort on joints (Bekey 1993). However, because, in such an embodiment, the distribution of the mass of the object 440 is not known embodiments use the geometrical center 775.

Embodiments also determine candidate grasp orientations for each face of the bounding box, i.e., for each candidate grasping location. FIG. 8 illustrates example orientations 886a-f determined for the object 440 using the bounding box 663. According to an embodiment, the orientations 886a-f, i.e., hand orientation, for each target location 774a-f, is defined by reusing the orientation of the minimum bounding box 663 determined based on the principal axis of inertia 662. In an embodiment, the z axes of the orientations 886a-f are each normal to their respective bounding box face and the x and y axes of the orientations 886a-f can be determined arbitrarily or in accordance with any desired, e.g., user-desired, procedure.

Grasp Type Determination

Feix 2015 described a taxonomy of different grasps that a human can perform. In Feix's work, a statistical analysis was performed of the different grasp characteristics based on measuring the object (size, weight), and grasp frequency for each grasp type. An embodiment leverages this statistical analysis and uses three of the most frequently used grasp types. FIG. 9 is a table 990 showing grasp types 991 and images 992 thereof, that are utilized in an embodiment. In such an embodiment the grasp types 991 include a pinch grasp 991a, medium wrap grasp 991b, and precision sphere grasp 991c. By using three of the most frequently used grasps 991a-c such an embodiment provides ample coverage of objects grasped in a manufacturing context.

According to an embodiment, for each grasp type, e.g., 991a-c, an open and closed hand configuration is created, e.g., manually by a user so as to correspond to certain grasp types, and used during hand closure on the object. FIG. 10 illustrates an example open configuration 1010 and closed configuration 1011 for the medium wrap grasp type 991b. These open and closed hand configurations can be re-used during different implementations of embodiments.

From amongst the various grasp types, e.g., 991a-c, embodiments select which grasp type to use for each face of the bounding box, e.g., each target grasp location 774a-f. An example embodiment uses dimensions of the bounding box faces to determine the grasp type for each face, e.g., each candidate grasp target location 774a-f and orientation 886a-f FIG. 11 illustrates functionality for determining the grasp type for the candidate location 774f and candidate orientation 886f, which are on the face 1100. Further, it is noted that while FIG. 11 illustrates functionality for the face 1100, embodiments determine a candidate grasp type for each face of the bounding box 663. For the face 1100 (which is the face of candidate location 774f and orientation 886f) two dimensions 1101 and 1102 are determined, i.e., the length and width of the face are determined. Further, the dimension 1103 of an edge normal to the edges of the dimensions 1101 and 1102 is determined. In summary, the dimensions 1101 and 1102 are the dimensions of the face 1100 and the dimension 1103 is the dimension of an edge normal to the face 1100.

These dimensions 1101, 1102, and 1103 are then used in the following logic to select the grasp type to use:

• • If Dimension 1101<60 mm And Dimension 1102<35 mm
    • Grasp Type=Pinch
  • Else If Dimension 1101≤90 mm And Dimension 1102≤90 mm And Dimension 1103≤50 mm
    • Grasp Type=Precision sphere
  • Else
    • Grasp Type=Medium Wrap

Based upon the above logic, a small object is grasped with a pinch grasp 991a and a bigger object that has a small 1103 dimension, e.g., a flat object, is grasped using a precision sphere grasp 991c (using the tip of the fingers). Otherwise, a medium wrap grasp 991b is used. The values in the above logic are based upon a Feix 2014 article and have been refined based on results of testing performed on different manufacturing parts. Further, it is noted that embodiments are not limited to using the above logic and specific dimensions therein and embodiments can consider different grasp types and use different tolerances, i.e., dimensions for selecting grasp types.

Face Labeling

An embodiment labels faces of the bounding box. According to an embodiment, the labeling enables (i) ranking of the grasps and (ii) using heuristics to determine an optimal grasping location. FIG. 12 illustrates an example of labeling where each face of the bounding box 663 is labeled depending on its position relative to the manikin 1220 initial position. The six faces labels used in FIG. 12 are: front, back, left, right, top, and bottom. Further, it is noted that while FIG. 12 illustrates labelling based upon position of the manikin, embodiments are not limited to such a method and embodiments can use any labelling technique that facilitates ranking/choosing target locations.

Graspable Faces

Embodiments determine which faces of the bounding box are graspable. In one such embodiment checks are performed to identify graspable faces.

One such embodiment, first, evaluates accessibility of each face. FIG. 13 depicts steps of a method 1330 for identifying graspable faces according to an embodiment. The method 1330 begins at step 1331 with the model of the object 440 and a model of the environment 1335. Next, at step 1332, an isolated hand 1336a-e is positioned at the targets location, e.g., 774a-f, in an open position, e.g., 1010. Further, the method 1330 can receive an indication of the hand being used to grasp the object and eliminate the face of the bounding box opposite the grasping hand. Such a face is eliminated because it can yield unrealistic grasps and postures. For instance, in the illustrated method 1330, the right hand is grasping the object 440 and the face 1337 is not considered. At step 1333, the method 1330 checks each face to determine if the face is accessible to the hand 1336a-e. If a collision between the isolated hand 1336a-e and the environment (shown by the model 1335) around the object (shown by the model 440) is detected, then the face is considered not accessible and is ignored when choosing the final grasp. In the example of FIG. 13, the final step 1334 illustrates that the bottom face is not accessible to the hand 1336d. While the bottom face is determined to not be accessible, the top, back, right, and front face are determined to be accessible to the hands 1336a, 1336b, 1336c, and 1336e, respectively. As such, going forward, the faces (and their corresponding candidate locations and orientations) accessible to the hands 1336a, 1336b, 1336c, and 1336e are candidates for determining an optimal graspable face.

After checking accessibility, the second check when identifying graspable faces is based on face dimensions. For each of the accessible faces, dimensions, e.g., 1101 and 1102 shown in the FIG. 11, are checked and if a dimension is greater than 100 mm then the face is considered too big to be grasped. This limit value follows a (Feix 2014) observation regarding dimension limits that a human can grasp. Returning to the faces accessible to the hands 1336a, 1336b, 1336c, and 1336e, each of said faces has dimensions under 100 mm and, thus, remain candidate graspable faces. Thus, in this example, the top, back, right, and front faces are candidate graspable faces.

Grasp Ranking

After identifying the graspable faces, embodiments rank the faces to determine an optimal graspable face. Table 1 below illustrates bounding box face rankings according to an embodiment.

TABLE 1
Bounding Box Face Rankings
	Grasp Side	Top	Right/Left	Bottom	Front	Back
	Rank	1	2	3	4	5

Table 1 shows that when the top side is graspable, it is considered the optimal grasping face. If the top side is not graspable, i.e., it is inaccessible or too big, the next face in the ranking that is graspable (right/left, bottom, front, back) is considered the optimal graspable face. If no face is graspable, the top face is chosen. In an embodiment using Table 1, the second rank is right/left, and the side selected is based on which end effector is involved in the grasping. Specifically, if the left end effector, e.g., hand, is used to grasp the object, then, the second rank is the left side and if the right hand is used to grasp the object, then, the second rank is the right side. In an embodiment, when grasping an object with the right hand, the left side is considered to not be graspable because it would result in the DHM having to be in an unrealistic posture. A similar logic is also applied when grasping with the left hand, i.e., the right face is considered ungraspable.

Grasp Execution

After determining an optimal face to grasp, embodiments determine the grasp, i.e., position and orientation, of an end effector. An embodiment determines the grasp using the determined candidate grasp location, the determined candidate grasp orientation, and the determined candidate grasp type of the determined optimal graspable face using an inverse kinematic solver.

FIG. 14 illustrates steps implemented by the inverse kinematic solver to determine the grasp according to an embodiment. Such an embodiment provides the target grasp location associated with the selected face to the inverse kinematic solver. The solver then matches the upper limb end effector frame 1440 with the target frame 1441. In such an embodiment, the target frame is the candidate location of the optimal face and the candidate orientation of the optimal face. In the embodiment depicted in FIG. 14 a more probable posture is determined by allowing a rotation degree of freedom along each direction of rotation 1442, 1443, and 1444 of the end effector. In an embodiment, the rotation about the hand palm plane is kept free (direction 1444, i.e., normal to the optimal grasping face) while the other rotations (1442 and 1443) are limited to some extent based on empirical tests (e.g. ±10 to ±30°). This gives the inverse kinematic solver more room to find a visually plausible posture while avoiding constraining the wrist too much. Once the target is reached, the hand closes on the object. The hand starts in its open configuration for the determined candidate grasp type of the optimal graspable face (shown by the visualization 1445) and each finger is moved toward its closed configuration for the determined candidate grasp type for the determined candidate grasp type of the optimal graspable face (shown by the visualization 1446). When a collision is detected between the finger and the object to grasp, the closure ends for that finger. The closure continues until all fingers are in collision or until all fingers reach their closed configuration. The position and orientation of the end effector at this stage, all fingers in collision or at the closed configuration, is the grasp.

Example Results

FIGS. 15A-D illustrate grasps on a gearbox assembly line determined by embodiments. The grasps were determined using embodiments, e.g., the method 100 of FIG. 1, the method 550 of FIG. 5, for the task of assembling the parts that compose a gearbox. FIGS. 15A-D show the overall DHM positioning 1550a-d in the environments while executing the determined grasps 1551a-d on the bearing cover 1552a, housing 1552b, screw 1552c, and flange 1552d. The examples shown in FIGS. 15A-D provide a good representation of grasps that can be determined by embodiments with different grasp types and locations. FIGS. 15A-D also show that the overall manikin postures 1550a-d are plausible. This is because the degrees of freedom allowed to the upper limb end effector by the inverse kinematic solver provide the solver with sufficient room to find plausible body postures.

Embodiments work well when grasping objects that are well represented by their oriented bounding box. More complex and bigger parts may be further segmented into multiple smaller subparts (Miller 2003) and, in turn, embodiments may be implemented on more specific locations on the object, i.e., the smaller subparts.

Computer Support

Embodiments can be implemented in the Smart Posture Engine (SPE) framework inside Dassault Systèmes application “Ergonomic Workplace Design”. With the Ergo4All (Bourret 2021) technology, the SPE enables assessment and minimization of ergonomic risks involved in simulated workplaces.

Moreover, embodiments may be implemented in any computer architectures known to those of skill in the art. For instance, FIG. 16 is a simplified block diagram of a computer-based system 1000 that may be used to determine grasps, i.e., position and orientation of an end effector of a digital human model for grasping an object, according to any variety of the embodiments of the present invention described herein. The system 1600 comprises a bus 1603. The bus 1603 serves as an interconnect between the various components of the system 1600. Connected to the bus 1603 is an input/output device interface 1606 for connecting various input and output devices such as a keyboard, mouse, display, speakers, etc. to the system 1600. A central processing unit (CPU) 1602 is connected to the bus 1603 and provides for the execution of computer instructions. Memory 1605 provides volatile storage for data used for carrying out computer instructions. In particular, memory 1605 and storage 1604 hold computer instructions and data (databases, tables, etc.) for carrying out the methods described herein, e.g., 100, 550, 660, 1330 of FIGS. 1, 5, 6, and 13, respectively. Storage 1604 provides non-volatile storage for software instructions, such as an operating system (not shown). The system 1600 also comprises a network interface 1601 for connecting to any variety of networks known in the art, including wide area networks (WANs) and local area networks (LANs).

It should be understood that the example embodiments described herein may be implemented in many different ways. In some instances, the various methods and machines described herein may each be implemented by a physical, virtual, or hybrid general purpose computer, such as the computer system 1600, or a computer network environment such as the computer environment 1710, described herein below in relation to FIG. 17. The computer system 1600 may be transformed into the machines that execute the methods (e.g., 100, 550, 660, and 1330) and techniques described herein, for example, by loading software instructions into either memory 1605 or non-volatile storage 1604 for execution by the CPU 1602. One of ordinary skill in the art should further understand that the system 1600 and its various components may be configured to carry out any embodiments or combination of embodiments of the present invention described herein. Further, the system 1600 may implement the various embodiments described herein utilizing any combination of hardware, software, and firmware modules operatively coupled, internally, or externally, to the system 1600.

FIG. 17 illustrates a computer network environment 1710 in which an embodiment of the present invention may be implemented. In the computer network environment 1710, the server 1711 is linked through the communications network 1712 to the clients 1713a-n. The environment 1710 may be used to allow the clients 1713a-n, alone or in combination with the server 1711, to execute any of the embodiments described herein. For non-limiting example, computer network environment 1710 provides cloud computing embodiments, software as a service (SAAS) embodiments, and the like.

Embodiments or aspects thereof may be implemented in the form of hardware, firmware, or software. If implemented in software, the software may be stored on any non-transient computer readable medium that is configured to enable a processor to load the software or subsets of instructions thereof. The processor then executes the instructions and is configured to operate or cause an apparatus to operate in a manner as described herein.

Further, firmware, software, routines, or instructions may be described herein as performing certain actions and/or functions of the data processors. However, it should be appreciated that such descriptions contained herein are merely for convenience and that such actions in fact result from computing devices, processors, controllers, or other devices executing the firmware, software, routines, instructions, etc.

It should be understood that the flow diagrams, block diagrams, and network diagrams may include more or fewer elements, be arranged differently, or be represented differently. But it further should be understood that certain implementations may dictate the block and network diagrams and the number of block and network diagrams illustrating the execution of the embodiments be implemented in a particular way.

Accordingly, further embodiments may also be implemented in a variety of computer architectures, physical, virtual, cloud computers, and/or some combination thereof, and thus, the data processors described herein are intended for purposes of illustration only and not as a limitation of the embodiments.

The teachings of all patents, published applications and references cited herein are incorporated by reference in their entirety.

While example embodiments have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the embodiments encompassed by the appended claims.

REFERENCES

• V.S.R. Group. Technical report for project virtual soldier research. Center for Computer-Aided Design, the University of IOWA. 2004
• Badler, Norman I., Palmer, Martha S., Bindiganavale, Rama. Animation control for real-time virtual humans. Communications of the ACM, 1999, vol. 42, no 8, p. 64-73.
• Zhou, Wei, Armstrong, Thomas J., Reed, Matthew P., et al. Simulating complex automotive assembly tasks using the HUMOSIM framework. SAE Technical Paper, 2009.
• Cort J A, Devries D. Accuracy of Postures Predicted Using a Digital Human Model During Four Manual Exertion Tasks, and Implications for Ergonomic Assessments. IISE Transactions on Occupational Ergonomics and Human Factors. 7(1):43-58 (2019).
• Hanson L, Hogberg D, Carlson J S, Bohlin R, Brolin E, Delfs N, et al. IMMA—Intelligently moving manikins in automotive applications. Third International Summit on Human Simulation (ISHS2014) (2014).
• Lemieux, P.-O., Barré, A., Hagemeister, N., Aissaoui, R.: Degrees of freedom coupling adapted to the upper limb of a digital human model. Int. J. Hum. Factors Model. Simul. 5(4), 314-337 (2017)
• Lemieux, P., Cauffiez, M., Barré, A., Hagemeister, N., Aissaoui, R.: A visual acuity constraint for digital human modeling. In: 4th Conference proceedings (2016)
• Zeighami, A., Lemieux, P., Charland, J., Hagemeister, N., Aissaoui, A.: Stepping behavior for stability control of a digital human model. ISB/ASB (2019)
• Bohg, J.; Morales, A.; Asfour, T.; Kragic, D.; Data-driven grasp synthesis—a survey, IEEE Transactions on Robotics, 30(2), 2013, 289-309. https://doi.org/10.1109/TRO.2013.2289018
• Zhou, Wei, ARMSTRONG, Thomas J., REED, Matthew P., et al. Simulating complex automotive assembly tasks using the HUMOSIM framework. SAE Technical Paper, 2009.
• Bourret, Q., Lemieux, P., Hagemeister, N., Aissaoui, R.: Flexible hand posture for tools grasping. DHM (2019)
• Macloud, Alexandre, Zeighami, Ali, Aissaoui, Rachid, et al. Extracting grasping cues from pistol-shaped tools for digital human models. Computer-Aided Design and Applications, 2021, vol. 18, no 6, p. 1167-1185.
• Macloud, A.; Zeighami, A.; Aissaoui, R.; Rivest, L.; Extracting grasping cues from one-handed tools geometry for digital human models, International Conference on Human Systems Integration (HSI2019), Biarritz, France, 2019.
• Bekey, George A., Liu, Huan, Tomovic, Rajko, et al. Knowledge-based control of grasping in robot hands using heuristics from human motor skills. IEEE Transactions on Robotics and Automation, 1993, vol. 9, no 6, p. 709-722.
• Feix, Thomas, Romero, Javier, Schmiedmayer, Heinz-Bodo, et al. The grasp taxonomy of human grasp types. IEEE Transactions on human-machine systems, 2015, vol. 46, no 1, p. 66-77.
• Feix, Thomas, Bullock, Ian M., et Dollar, Aaron M. Analysis of human grasping behavior: Object characteristics and grasp type. IEEE transactions on haptics, 2014, vol. 7, no 3, p. 311-323.
• Miller, A. T., Knoop, S., Christensen, H. I. and Allen, P. K., Automatic grasp planning using shape primitives. in Robotics and Automation, 2003. Proceedings. ICRA'03. IEEE International Conference on, (2003), IEEE, 1824-1829.
• Bourret, Quentin, et al. “Ergo4A11: An Ergonomic Guidance Tool for Non-ergonomist.” Congress of the International Ergonomics Association. Springer, Cham, 2021.

Claims

1. A computer-implemented method of determining position and orientation of an end effector of a digital human model (DHM) for grasping an object, the method comprising: receiving (i) a computer-based model of an object, (ii) a computer-based model of an environment, and (iii) an indication of position of a DHM in the environment;determining an oriented bounding box surrounding the received model of the object, wherein the oriented bounding box includes a plurality of faces;for each of the plurality of faces, determining: a candidate grasp location, a candidate grasp orientation, and a candidate grasp type;from amongst the plurality of faces, determining one or more graspable faces based on: (a) the determined candidate grasp location of each face, (b) the determined candidate grasp orientation of each face, (c) the received model of the environment, and (d) dimensions of each face;from amongst the determined one or more graspable faces, determining an optimal graspable face based on a predetermined grasping hierarchy and the received indication of position of the DHM in the environment; andusing an inverse kinematic solver to determine position and orientation of an end effector of the DHM grasping the object based on the determined candidate grasp location, the determined candidate grasp orientation, and the determined candidate grasp type of the determined optimal graspable face.

2. The method of claim 1 wherein determining the oriented bounding box comprises: determining a minimum bounding box surrounding the received model of the object;determining a principal axis of inertia of the object based on the received model of the object;orienting the determined minimum bounding box based on the determined principal axis of inertia; andsetting the oriented minimum bounding box as the oriented bounding box surrounding the received model of the object.

3. The method of claim 2 wherein determining a candidate grasp orientation for a given face of the plurality of faces comprises: setting the candidate grasp orientation for the given face based on the determined principal axis of inertia of the object.

4. The method of claim 1 wherein determining a candidate grasp location for a given face of the plurality of faces comprises: based on the received model of the object, calculating a geometrical center of the object;projecting from the calculated geometrical center of the object to the given face; andsetting location of an intersection of the projection and the given face as the candidate grasp location for the given face.

5. The method of claim 1 wherein determining a candidate grasp type for a given face of the plurality of faces comprises: calculating length of a first edge and a second edge of the given face, wherein the first edge and the second edge are perpendicular to each other;calculating length of a face edge normal to the first edge and the second edge; anddetermining the candidate grasp type for the given face based on: (i) the calculated length of the first edge, (ii) the calculated length of the second edge, and (iii) the calculated length of the face edge normal to the first edge and the second edge.

6. The method of claim 1 wherein each determined candidate grasp type is one of: a pinch type, a medium-wrap type, and a precision sphere type.

7. The method of claim 1 wherein, determining one or more graspable faces based on: (a) the determined candidate grasp location of each face, (b) the determined candidate grasp orientation of each face, (c) the received model of the environment, and (d) the dimensions of each face comprises: identifying a given face as a graspable face if (i) the end effector of the DHM, at the determined candidate grasp location in the determined candidate grasp orientation, does not collide with an element in the model of the environment and (ii) dimensions of the given face do not exceed a threshold.

8. The method of claim 1 wherein the DHM includes a left end effector and a right end effector and the method further comprises: receiving an indication of the end effector, from amongst the left end effector and the right end effector, of the DHM grasping the object.

9. The method of claim 8 further comprising: selecting the predetermined grasping hierarchy based on the received indication of the end effector.

10. The method of claim 1 further comprising: configuring the inverse kinematic solver to have an unconstrained rotation degree of freedom along an axis normal to the determined optimal graspable face.

11. The method of claim 1 further comprising: applying a respective label to each face of the plurality of faces, wherein for each face the respective label is a function of position of the DHM in relation to the face.

12. The method of claim 11 wherein the predetermined grasping hierarchy indicates a preferred order of graspable faces as a function of each respective label.

13. The method claim 1 further comprising: simulating physical interaction between the DHM and the object using the determined position and orientation of the end effector.

14. A system for determining position and orientation of an end effector of a digital human model (DHM) for grasping an object, the system comprising: a processor; anda memory with computer code instructions stored thereon, the processor and the memory, with the computer code instructions, being configured to cause the system to: receive (i) a computer-based model of an object, (ii) a computer-based model of an environment, and (iii) an indication of position of a DHM in the environment;determine an oriented bounding box surrounding the received model of the object, wherein the oriented bounding box includes a plurality of faces;for each of the plurality of faces, determine: a candidate grasp location, a candidate grasp orientation, and a candidate grasp type;from amongst the plurality of faces, determine one or more graspable faces based on: (a) the determined candidate grasp location of each face, (b) the determined candidate grasp orientation of each face, (c) the received model of the environment, and (d) dimensions of each face;from amongst the determined one or more graspable faces, determine an optimal graspable face based on a predetermined grasping hierarchy and the received indication of position of the DHM in the environment; anduse an inverse kinematic solver to determine position and orientation of an end effector of the DHM grasping the object based on the determined candidate grasp location, the determined candidate grasp orientation, and the determined candidate grasp type of the determined optimal graspable face.

15. The system of claim 14 wherein: in determining the oriented bounding box, the processor and the memory, with the computer code instructions, are further configured to cause the system to: determine a minimum bounding box surrounding the received model of the object;determine a principal axis of inertia of the object based on the received model of the object;orient the determined minimum bounding box based on the determined principal axis of inertia; andset the oriented minimum bounding box as the oriented bounding box surrounding the received model of the object; andin determining a candidate grasp orientation for a given face of the plurality of faces, the processor and the memory, with the computer code instructions, are further configured to cause the system to: set the candidate grasp orientation for the given face based on the determined principal axis of inertia of the object.

16. The system of claim 14 wherein, in determining a candidate grasp location for a given face of the plurality of faces, the processor and the memory, with the computer code instructions, are further configured to cause the system to: based on the received model of the object, calculate a geometrical center of the object;project from the calculated geometrical center of the object to the given face; andset location of an intersection of the projection and the given face as the candidate grasp location for the given face.

17. The system of claim 14 wherein, in determining a candidate grasp type for a given face of the plurality of faces, the processor and the memory, with the computer code instructions, are further configured to cause the system to: calculate length of a first edge and a second edge of the given face, wherein the first edge and the second edge are perpendicular to each other;calculate length of a face edge normal to the first edge and the second edge; anddetermine the candidate grasp type for the given face based on: (i) the calculated length of the first edge, (ii) the calculated length of the second edge, and (iii) the calculated length of the face edge normal to the first edge and the second edge.

18. The system of claim 14 wherein, in determining one or more graspable faces based on: (a) the determined candidate grasp location of each face, (b) the determined candidate grasp orientation of each face, (c) the received model of the environment, and (d) the dimensions of each face, the processor and the memory, with computer code instructions, are further configured to cause the system to: identify a given face as a graspable face if (i) the end effector of the DHM, at the determined candidate grasp location in the determined candidate grasp orientation, does not collide with an element in the model of the environment and (ii) dimensions of the given face do not exceed a threshold.

19. The system of claim 14 wherein the processor and the memory, with the computer code instructions, are further configured to cause the system to: configure the inverse kinematic solver to have an unconstrained rotation degree of freedom along an axis normal to the determined optimal graspable face.

20. A non-transitory computer program product for determining position and orientation of an end effector of a digital human model (DHM) for grasping an object, the computer program product executed by a server in communication across a network with one or more client and comprising: a computer readable medium, the computer readable medium comprising program instructions which, when executed by a processor, causes the processor to: receive (i) a computer-based model of an object, (ii) a computer-based model of an environment, and (iii) an indication of position of a DHM in the environment;determine an oriented bounding box surrounding the received model of the object, wherein the oriented bounding box includes a plurality of faces;for each of the plurality of faces, determine: a candidate grasp location, a candidate grasp orientation, and a candidate grasp type;from amongst the plurality of faces, determine one or more graspable faces based on: (a) the determined candidate grasp location of each face, (b) the determined candidate grasp orientation of each face, (c) the received model of the environment, and (d) dimensions of each face;from amongst the determined one or more graspable faces, determine an optimal graspable face based on a predetermined grasping hierarchy and the received indication of position of the DHM in the environment; anduse an inverse kinematic solver to determine position and orientation of an end effector of the DHM grasping the object based on the determined candidate grasp location, the determined candidate grasp orientation, and the determined candidate grasp type of the determined optimal graspable face.