1. Field of the Invention
This invention relates, generally, to the arts of robotics and neuroscience. More particularly, it relates to a system for studying human grasping techniques by simultaneously identifying the contact position and orientation of the fingertip and measuring the force and torque exerted by the fingertip during various predetermined grasps.
2. Description of the Prior Art
Understanding of local mechanical interactions between human fingertips and an object is crucial to understanding human dexterity. With various kinds of force sensors, the fingertip force at the contact points on an object may be measured to study human dexterous grasping and manipulation. Many robot grasp planning and control algorithms have been developed with inspirations from human grasping studies. However, models that simply include contact force and contact location on the object have limited utility.
Studying human grasping techniques has significant implications for robotics because a human hand exhibits some of the most complex human motor skills. Many studies have proposed various taxonomies of human grasping. A basic distinction is between a power grasp, where the goal is to stabilize an object to resist external forces, and a precision grasp, which is an essential skill for manipulating an object, maintaining the grasp during various tasks, and re-grasping.
A preponderance of research has focused on precision grasps and on instrumentation to measure the fingertip forces and position on the object during contact. A rich experimental paradigm for the study of human grasping was disclosed by Johansson and Westling in Roles of Glabrous Skin Receptors and Sensorimotor Memory in Automatic Control of Precision Grip when Lifting Rougher or More Slippery Objects. This approach to studying human grasping involves parallel force-sensing pads for opposition grasps between the index finger and the thumb. This approach has been generalized to add more touch pads to include the middle, ring, and little fingers in addition to the index finger-thumb pair. In general, instrumented objects are typically created that incorporate miniature 6-axis force/torque sensors at predefined grasp points.
Besides the complex hand postures and forces during grasping, the great variance in what parts of the fingertip contact an object makes the grasping studies very complex. Many studies have been carried out to characterize the mechanical profile in the fingertip responding to a force. For example, Serina et al. and Pawluk and Howe have examined the dynamic force response of the fingertip to indenters of different sizes and shapes.
Many research teams have experimented with using various kinds of force sensors to measure the fingertip force at the contact points on an object. Many encouraging results have been achieved. However, more and more researchers have realized the limitation of the models that only include contact force and contact position on the object. Many researchers have sought for a way to measure the contact position on the fingertip to better model and understand more sophisticated grasping techniques such as rolling and re-grasping.
Commercial tactile sensors and flexible tactile sensors have been successfully embedded on many robotic fingers to provide rich spatial grasping information for grasping control. However, due to their stiffness, those tactile sensors cannot deform along with the human finger tissue to preserve the elastic profile, which prevents them from being used in many human grasping studies. Most studies attach tactile sensors on the force sensors rather than on the fingertip, thus obtaining local contact spatial profiles around the contact point, but not being capable of identifying the contact position on the fingertip.
Thus there is a need in neuroscience and robotics for a sensor capable of measuring the fingertip force and torque, the contact location, and the orientation of the fingertip.
However, in view of the art considered as a whole at the time the present invention was made, it was not obvious to those of ordinary skill in the art how the needed structure could be provided.
The long-standing but heretofore unfulfilled need for studying human grasping techniques through a system capable of measuring the force and torque exerted by a human fingertip during a grasp and simultaneously determining that fingertip's contact position and orientation is now met by a new, useful, and non-obvious invention.
The inventive system utilizes a six-axis force/torque sensor to measure the force and torque exerted by a fingertip during various grasps. The invention also utilizes a fingertip sensor rigidly mounted onto the force/torque sensor to simultaneously determine the contact position and orientation of the fingertip. The system is placed on a physical object, grasping of which is being studied, at a point where a fingertip makes contact with that object during a grasp. To obtain the most comprehensive data about the grasp, five such inventive systems can be placed at all five contact points corresponding to the five fingertips of a human hand. The positioning of the sensor system can be adjusted to correspond to the fingertip contact points for the particular grasp being studied. The invention can be used to study a wide variety of grasping techniques for a plethora of different objects.
For each human subject whose grasping techniques are being studied, a fingerprint map of each fingertip must be constructed. The invention contemplates construction of a two-dimensional fingerprint map from a sequence of overlapping fingerprint images. Such sequence is obtained by rolling a subject's fingertip on a fingerprint scanner. If the fingerprint scanner has a sufficiently large window, the images are assumed to be pre-aligned and no registration or realignment is required. However, if the fingerprint sensor does not have a scanning window of a sufficient size, the fingerprint map is constructed through utilization of one of many techniques known in the art for aligning the fingerprint images and then combining the multiple partially overlapping images into a single fingerprint image. Due to unique nature of every fingerprint, the fingerprint map construction procedure outlined above must be performed for each fingertip of every human subject participating in the study.
In addition to measuring force and torque, the invention allows researchers to determine the contact position and orientation of the fingertip during a grasp. The contact position of the fingertip relative to the force sensor is obtained by extracting the fingerprint area from the background of the fingertip image and computing the centroid of the fingerprint area.
The contact position and orientation relative to the fingertip are obtained by registering the fingerprint image onto a preconstructed fingerprint map. A number of suitable registration approaches are known in the art. However, for the purpose of reducing computation time, the invention contemplates using a registration approach based on the following parameters: coordinates of the core points on the fingerprint image and the fingerprint map and the rotation angle, which is defined as the angular difference between the local ridge orientation at the core point on the fingerprint image and the local ridge orientation at the core point on the fingerprint map. Once these parameters are obtained, the center point of the fingerprint image is registered onto the fingerprint map, and the contact position and orientation relative to the fingertip are fully characterized.
In grasping and hand manipulation studies, the novel device is not limited to measuring only the contact force, which is a limitation present in most prior art devices, but is also capable of determining the contact spatial features relative to the fingertip and the force sensor without sacrificing the precision. This constitutes a major advantage over prior art and enables roboticists and neuroscientists to characterize human hands locomotion in substantially enhanced detail.
In summary, the inventive system is capable of measuring all three components of force and all three components of torque exerted by a human fingertip during a grasp and simultaneously determining the fingertip's contact position relative to the force sensor, the contact position relative to the fingertip, and the orientation of the fingertip relative to the force sensor. The invention enables researchers to collect detailed and comprehensive data about human grasps, thus enabling those skilled in the art of robotics to study human grasping techniques and develop algorithms closely mimicking those techniques for implementation in robotics.
The invention accordingly comprises the features of construction, combination of elements, and arrangement of parts that will be exemplified in the disclosure set forth hereinafter and the scope of the invention will be indicated in the claims.
For a fuller understanding of the nature and objects of the invention, reference should be made to the following detailed disclosure, taken in connection with the accompanying drawings, in which:
Weight and thickness of a typical fingerprint sensor are relatively small compared to those of a force sensor, and therefore, do not substantially affect the accuracy of the force and torque readings. Due to their thinness and low weight, solid-state fingerprint sensors, such as the one depicted in
Compared to a sampling rate of a typical force sensor, most conventional fingerprint sensors have very low sampling rates. However, since the contact position of a fingertip does not change rapidly during a grasp, the relative low sampling rate of a fingerprint sensor does not constitute a substantial limitation for typical grasping studies. Nevertheless, fingerprint sensors capable of higher sampling rates are preferred as they will provide more comprehensive and accurate data, especially when studying complex grasps.
Inventive system uses a computer to collect the data from both force sensor 10 and fingerprint sensor 20. A data acquisition card reads the force data from force sensor 10 and fingerprint sensor 20 at a specified sampling rate. Fingerprint images are processed with an image enhancement algorithm to reduce noise and increase robustness by improving the clarity of ridges and valleys; enhancement of a fingerprint image is depicted in
To accurately represent the location of the contact area on fingertip 30, a fingerprint map 40 (
A preferred method for constructing fingerprint map 40 requires a fingerprint sensor with a sufficiently large scanning window. Fingerprint map 40 is constructed from a sequence of fingerprints obtained by rolling fingertip 30 on a fingerprint sensor. As fingertip 30 is being rolled, the sensor acquires a sequence of fingerprint images. These images are then combined into a single fingerprint map.
An alternative method for constructing a fingerprint map utilizes a fingerprint sensor with a small scanning window. According to this method, portions of the fingerprint are captured one at a time. The acquired images are then aligned to properly fit with each other. A number of well-known techniques exist for combining multiple partially overlapping fingerprint images into a single fingerprint image. One such technique involves the fingerprint foregrounds being segmented from the background and then stacked together. The pixel values at overlapping areas are computed by averaging the pixel values of the overlapping foregrounds.
u=a sin φ cos θ (1)
v=b sin φ sin θ (2)
w=c cos φ (3)
Variables a, b, and c are fixed, and the pair (θ, φ) can sufficiently represent any point on fingertip 30 with −π/2<θ<π/2 and 0<φ<π/2, where φ is the colatitude, or zenith, and θ is the longitude, or azimuth. This coordinate system is depicted in
u=r sin φ cos θ (4)
v=r sin φ sin θ (5)
w=c cos φ. (6)
To identify the contact position on fingerprint map 40, the following coordinate system is defined on fingerprint map 40: the origin (0,0) is located at the core point, the x-axis is along the horizontal axis of the fingerprint, and the y-axis is along its vertical axis. This coordinate system makes it possible to represent any point on fingertip 30 with a set of numerical coordinates corresponding to a location on fingerprint map 40. This coordinate system is depicted in
Fingerprint sensor 20 is aligned with the W-axis in fingertip coordinate system depicted in
x=2πrφ (7)
The rolling behavior on the pitch direction relates the colatitude angle φ1 to the y-axis in fingerprint map 40 with the elliptic integral of the second kind, as expressed in the following equation, where y is the length of the elliptical arc with φ degree:
If a contact position (x, y) is located on fingerprint map 40, its (θ, φ) in fingertip coordinate system can be computed with Equations 7 and 8. The parameters c and r are calibrated with the following initial values: θ=0, θ=π/2, φ=0, and φ=π/2, and φ may be retrieved from a φ−y table pre-computed with numerical integration.
Turning now to contact characterization, it is understood that there are number of variables when fingertip 30 contacts force sensor 10. Most prior art research studies have analyzed only force and torque, failing to consider the contact position and orientation of fingertip 30 relative to force sensor 10 or relative to fingertip 30. Through utilization of fingerprint sensor 20, the novel sensor system can measure three additional contact characteristics: the contact position relative to force sensor 10, the contact position on fingertip 30, and the relative orientation between force sensor 10 and fingertip 30.
Considering the contact position relative to force sensor 10, since fingerprint sensor 20 is rigidly attached and aligned to force sensor 10, the contact area on fingerprint sensor 20 can be used to compute the location at which the contact force is applied. Unlike prior art studies, this approach does not require assumptions that the force is always applied at the center of force sensor 10 or that the contact area covers the entire surface of force sensor 10. For many human grasping studies, it is crucial to know the exact position at which fingertip 30 comes into contact with force sensor 10.
To determine the contact position on fingertip 30 and orientation of fingertip 30 relative to force sensor 10, fingerprint image 50 is registered onto fingerprint map 40. A number of registration approaches are known in the prior art. The contact position and orientation on the fingerprint can be obtained from the registration result. Although many registration approaches exist, most of them involve feature extraction and correlation matching and, therefore, require a long computation time.
The novel sensor system uses the fingerprint features, specifically, core points 60 and 70 (
The local ridge orientation at a pixel is defined as the rotation angle α at a point crossing through a small neighborhood centered at that pixel. Since fingerprint ridges are not directed, the rotation angle α is defined in the range of 0≦α≦π. Extraction of local ridge orientations requires computation of gradients in fingerprint image 50. The orientation around core points 60 and 70 in both fingerprint map 40 and fingerprint image 50 can be obtained. Their angle difference is computed and defined as the rotation angle α.
With translation and rotation between the new fingerprint and fingerprint map being known, a point p′=(x′, y′) on fingerprint image 50 can be transformed to a point p=(x, y) in fingerprint map 40 by using the following equations:
(p−p′)=R(α)(p−p′)+(p−p′core) (9)
pcore and p′core are core points 60 and 70 respectively; and α is the relative rotation angle between fingerprint map 40 and fingerprint image 50. The center point of fingerprint image 50 is registered to fingerprint map 40 with Equation 9. The contact properties are fully characterized with Equations 7 and 8.
A contact force between fingertip 30 and an object can be characterized in both the human subject's coordinate system and the world coordinate system. The contact force (Fx, Fy, Fz) and torque (Tx, Ty, Tz) are not assumed to be always applied at the center of force sensor 10. Instead, fingerprint sensor 20 measures the real contact position (Px, Py, Pz) relative to the center of force sensor 10, and it is a variable during a grasp. Contact position (φ,θ) and orientation α relative to fingertip 30 can also be determined. The force (Fx, Fy, Fz) and torque (Tx, Ty, Tz) can be expressed in fingertip coordinate system.
Fingerprint sensor 20 can also be used to design an artificial robotic finger that is illustrated in
The same image registration approach as above is used to estimate the contact location and orientation on rubber/silicon 100. Since rubber/silicon 100 is fixed above fingerprint sensor 20, the contact location on fingerprint sensor 20 is also obtained. Due to the elasticity of rubber/silicon 100, the same contact location on fingerprint sensor 20 does not necessarily give the same contact location on the rubber/silicon 100. The difference between those two measurements represents the shear movement of rubber/silicon 100, which represents the shear force applied on the artificial finger through rubber/silicon 100. The orientation measurement represents the normal torque on artificial finger surface 90 through rubber/silicon 100. The area of the contact can represent the normal force level.
The relation between the area of the contact obtained through fingerprint sensor 20 and the normal force level on artificial finger surface 90, the relation between the fingerprint orientation measurement through fingerprint sensor 20 and the normal torque on artificial finger surface 90, and the relation between the difference of the two contact location measurements and the shear force on artificial fingertip surface 90 can all be calibrated with experiment and computation from the design.
A force and torque sensor can be mounted under fingerprint sensor 20 (
It will thus be seen that the objects set forth above, and those made apparent from the foregoing disclosure, are efficiently attained and since certain changes may be made in the above construction without departing from the scope of the invention, it is intended that all matters contained in the foregoing disclosure or shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.
It is also to be understood that the following claims are intended to cover all of the generic and specific features of the invention herein described, and all statements of the scope of the invention that, as a matter of language, might be said to fall therebetween.
This application claims priority to U.S. Provisional Application No. 61/420,007 entitled “Fingertip Force, Location, and Orientation Sensor”, filed Dec. 6, 2010 by the same inventor, the contents of which are hereby incorporated by reference into this disclosure.
Number | Name | Date | Kind |
---|---|---|---|
4156230 | Riganati et al. | May 1979 | A |
4696046 | Schiller | Sep 1987 | A |
4783167 | Schiller et al. | Nov 1988 | A |
5613014 | Eshera et al. | Mar 1997 | A |
5742279 | Yamamoto et al. | Apr 1998 | A |
5799098 | Ort et al. | Aug 1998 | A |
5892838 | Brady | Apr 1999 | A |
6031942 | Nakayama | Feb 2000 | A |
6064753 | Bolle et al. | May 2000 | A |
6282304 | Novikov et al. | Aug 2001 | B1 |
6400836 | Senior | Jun 2002 | B2 |
6435794 | Springer | Aug 2002 | B1 |
6707934 | Takeda et al. | Mar 2004 | B1 |
6819782 | Imagawa et al. | Nov 2004 | B1 |
6970234 | Nagasaka et al. | Nov 2005 | B2 |
7627150 | Abiko et al. | Dec 2009 | B2 |
7671845 | Keely | Mar 2010 | B2 |
7885437 | Hara | Feb 2011 | B2 |
7899216 | Watanabe et al. | Mar 2011 | B2 |
8145916 | Boshra et al. | Mar 2012 | B2 |
20020031244 | Rozenberg et al. | Mar 2002 | A1 |
20030149661 | Mitchell et al. | Aug 2003 | A1 |
20030169910 | Reisman et al. | Sep 2003 | A1 |
20060078176 | Abiko et al. | Apr 2006 | A1 |
20060239514 | Watanabe et al. | Oct 2006 | A1 |
20060284858 | Rekimoto | Dec 2006 | A1 |
20080117409 | Nagasaka et al. | May 2008 | A1 |
20080298648 | Lo et al. | Dec 2008 | A1 |
20090278798 | Kim et al. | Nov 2009 | A1 |
Entry |
---|
Johansson, R.S.; Westling, G. Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects. Experimental Brain Research. 1984, vol. 56, No. 3, pp. 550-564. |
Number | Date | Country | |
---|---|---|---|
61420007 | Dec 2010 | US |