Patent Application
20090118864
In the early days of using robots for automated manufacturing, robot tasks were programmed by manually teaching the robot where to go. While manufacturing tasks remained of the relatively simple pick-and-place type, this method of robot programming was adequate because the number of robot poses required was small. However, as the complexity of automated systems increased, so did the need for a higher-level type of programming. The concept of offline programming arose, which basically means that instead of manually recording joint angles for each desired position, a high level task description may be specified, and then automatically translated into a set of joint angles in order to accomplish the desired task. In order to go from task space to joint space, a mathematical (i.e., kinematic) model for the robot was used.
An embodiment of the present invention may comprise a method for vision-based calibration of a tool-frame for a tool attached to a robot using a camera comprising: providing the robot, the robot having a wrist that is moveable, the robot having a control system that moves the robot and the wrist into different poses, the tool attached to the robot being at different orientations for the different poses, the robot control system defining a wrist-frame for the wrist of the robot such that the robot control system knows a position and an orientation of the wrist for the different poses via a kinematic model of the robot; providing the camera, the camera being mounted external of the robot, the camera capturing an image of the tool; designating a point on the tool in the image of the tool as an image tool center point of the tool, the image tool center point being a point on the tool that is desired to be an origin of the tool-frame for the kinematic model of the robot; moving the robot into a plurality of wrist poses, each wrist pose of the plurality of wrist poses being constrained such that the image tool center point of the tool is located within a specified geometric constraint in the image captured by the camera; calculating a tool-frame tool center point relative to the wrist-frame of the wrist of the robot for the tool as a function of the specified geometric constraint and also as a function of the position and the orientation of the wrist of the robot for each wrist pose of the plurality of wrist poses; defining the tool-frame of the tool relative to the wrist-frame for the kinematic model of the robot as the tool-frame tool center point; and, operating the robot to perform desired tasks with the tool using the kinematic model of the robot with the defined tool-frame.
An embodiment of the present invention may further comprise a vision-based robot calibration system for calibrating a tool-frame for a tool attached to a robot using a camera comprising: the robot, the robot having a wrist that is moveable, the robot having a control system that moves the robot and the wrist into different poses, the tool attached to the robot being at different orientations for the different poses, the robot control system defining a wrist-frame for the wrist of the robot such that the robot control system knows a position and an orientation of the wrist for the different poses via a kinematic model of the robot; the camera, the camera being mounted external of the robot, the camera capturing an image of the tool; a wrist pose sub-system that designates a point on the tool in the image of the tool as an image tool center point of the tool and moves the robot into a plurality of wrist poses, the image tool center point being a point on the tool that is desired to be an origin of the tool-frame for the kinematic model of the robot, each wrist pose of the plurality of wrist poses being constrained such that the image tool center point of the tool is located within a specified geometric constraint in the image captured by the camera; a tool center point calculation sub-system that calculates a tool-frame tool center point relative to the wrist-frame of the wrist of the robot for the tool as a function of the specified geometric constraint and also as a function of the position and the orientation of the wrist of the robot for each wrist pose of the plurality of wrist poses; a robot kinematic incorporation subsystem that defines the tool-frame of the tool relative to the wrist-frame for the kinematic model of the robot as the tool-frame tool center point.
An embodiment of the present invention may further comprise a vision-based robot calibration system for calibrating a tool-frame for a tool attached to a robot using a camera comprising: means for providing the robot, the robot having a wrist that is moveable, the robot having a control system that moves the robot and the wrist into different poses, the robot control system defining a wrist-frame for the wrist of the robot such that the robot control system knows a position and an orientation of the wrist for the different poses via a kinematic model of the robot; means for providing the camera, the camera being mounted external of the robot, the camera capturing an image of the tool; means for designating a point on the tool in the image of the tool as an image tool center point of the tool; means for moving the robot into a plurality of wrist poses, each wrist pose of the plurality of wrist poses being constrained such that the image tool center point of the tool is located within a specified geometric constraint in the image captured by the camera; means for calculating a tool-frame tool center point relative to the wrist-frame of the wrist of the robot for the tool as a function of the specified geometric constraint and also as a function of the position and the orientation of the wrist of the robot for each wrist pose of the plurality of wrist poses; means for defining the tool-frame of the tool relative to the wrist-frame for the kinematic model of the robot as the tool-frame tool center point; and, means for operating the robot to perform desired tasks with the tool using the kinematic model of the robot with the defined tool-frame.
An embodiment of the present invention may further comprise a computerized method for calculating a tool-frame tool center point relative to a wrist-frame of a robot for a tool attached at a wrist of the robot using a camera comprising: providing a computer system for running computer software, the computer system having at least one computer readable storage medium for storing data and computer software; mounting the camera external of the robot; operating the camera to capture an image of the tool; defining a point on a geometry of the tool as a tool center point of the tool; defining a constraint region on the image captured by the camera; moving the robot into a plurality of wrist poses, each wrist pose of the plurality of wrist poses having a known position and orientation within a kinematic model of the robot; each wrist pose of the plurality of wrist poses having a different position and orientation from other wrist poses of the plurality of wrist poses; analyzing the image captured by the camera with the computer software to locate the tool center point of the tool in the image for each wrist pose of the plurality of wrist poses; correcting the position and orientation of each wrist pose of the plurality of wrist poses using the camera such that the tool center point of the tool located in the image captured by the camera is constrained within the constraint region defined for the image; calculating a tool-frame tool center point relative to the wrist-frame of the robot with the computer software as a function of the position and orientation of each wrist pose of the plurality of wrist poses as corrected to constrain the tool center point in the image to the constraint region on the image; updating the kinematic model of the robot with the computer software to incorporate the tool-frame tool center point relative to the wrist-frame of the robot as an origin of the tool-frame of the tool within the kinematic model of the robot; and, operating the robot using the kinematic model as updated to incorporate the tool-frame tool center point to perform desired tasks with the tool.
An embodiment of the present invention may further comprise a computerized calibration system for calculating a tool-frame tool center point relative to a wrist-frame of a robot for a tool attached at a wrist of the robot using an externally mounted camera comprising: a computer system that runs computer software, the computer system having at least one computer readable storage medium for storing data and computer software; operating the camera to capture an image of the tool; a constraint definition sub-system that defines a point on a geometry of the tool as a tool center point of the tool and defines a constraint region on the image captured by the camera; a wrist pose sub-system that moves the robot into a plurality of wrist poses, each wrist pose of the plurality of wrist poses having a known position and orientation within a kinematic model of the robot; each wrist pose of the plurality of wrist poses having a different position and orientation from other wrist poses of the plurality of wrist poses; an image analysis sub-system that analyzes the image captured by the camera with the computer software to locate the tool center point of the tool in the image for each wrist pose of the plurality of wrist poses; a wrist pose correction sub-system that corrects the position and orientation of each wrist pose of the plurality of wrist poses using the camera such that the tool center point of the tool located in the image captured by the camera is constrained within the constraint region defined for the image; a tool-frame tool center point calculation sub-system that calculates a tool-frame tool center point relative to the wrist-frame of the robot with the computer software as a function of the position and orientation of each wrist pose of the plurality of wrist poses as corrected to constrain the tool center point in the image to the constraint region on the image; and, a kinematic model update sub-system that updates the kinematic model of the robot with the computer software to incorporate the tool-frame tool center point relative to the wrist-frame of the robot as an origin of the tool-frame of the tool within the kinematic model of the robot.
An embodiment of the present invention may further comprise a robot calibration system that finds a tool-frame tool center point relative to a wrist-frame of a tool attached to a robot using an externally mounted camera comprising a computer system programmed to: analyze an image captured by the externally mounted camera to locate a point on the tool in the image designated as an image tool center point of the tool for each wrist pose of a plurality of wrist poses of the robot, each wrist pose of the plurality of wrist poses being constrained such that the image tool center point is constrained within a geometric constraint region on the image, each wrist pose of the plurality of wrist poses having a known position and orientation within a kinematic model of the robot, each wrist pose of the plurality of wrist poses having a different position and orientation within the kinematic model of the robot from other wrist poses of the plurality of wrist poses; calculate the tool-frame tool center point relative to the wrist-frame of the robot as a function of the position and orientation of each wrist pose of the plurality of wrist poses; update the kinematic model of the robot to incorporate the tool-frame tool center point relative to the wrist-frame of the robot as an origin of the tool-frame of the tool within the kinematic model of the robot; and, deliver the updated kinematic model of the robot to the robot such that the robot operates using the updated kinematic model to perform desired tasks with the tool attached to the robot.
In the drawings,
There are several important coordinate frames 114-120 that are usually defined for a robotic system 102. The world-frame 114 is typically defined somewhere in space, and does not necessarily correspond to any physical feature of the robot 102 or of the work cell. The base-frame 116 of the robot 102 is typically centered at the base 104 of the robot 102, with the z-axis of the base-frame 116 pointing along the first joint 106 axis. The wrist-frame 118 of the robot is typically centered at the last link (usually link 6) (aka. wrist 108). The relationship between the base-frame 116 and the wrist-frame 118 is typically determined through the kinematic model of the robot 102, which is usually handled inside the robot 102 controller software. The tool-frame 120 is typically specified with respect to the wrist-frame 116, and is usually defined with the origin 122 at the tip of the tool 112 and the z-axis along the tool 112 direction. The tool 112 direction may be somewhat arbitrary, and depends to a great extent on the type of tool 112 and the task at hand. The tool-frame 120 is typically a coordinate transformation between the wrist-frame 118 and the tool 112, and is sometimes called the tool offset. The three-dimensional (3-D) position of the origin 122 of the tool-frame 120 relative to the wrist-frame 118 is typically also called the tool center point (TCP) 122. Tool 112 calibration generally means computing both the position (TCP) 122 and orientation of the tool-frame 120.
A distinction is typically made between accuracy and repeatability in robot systems. Accuracy is the ability of the robot 102 to place its end effector (e.g., the tool 112) at a pre-determined point in space, regardless of whether that point has been reached before or not. Repeatability is the ability of the robot 102 to return to a previous pose. Usually a robot's 102 repeatability will be better than the robot's 102 accuracy. That is, the robot 102 can return to the same point every time, but that point may not be exactly the point that was specified in task space. Thus, it is likely better to use relative motions of the robot 102 for calibration instead of relying on absolute positioning accuracy.
As offline programming of industrial robotic systems has become more prevalent, the need for accurate calibration techniques for components in an industrial robot cell has increased. One of the factors required for successful offline programming is an accurate calibration of the robot's 102 tool-frame 120. Excessive errors in the calibration of the tool-frame 120 will result in tool positioning errors that may render the system useless. Methods for calibrating the tool-frame 120 typically are manual, time consuming, and often require a skilled operator. Various embodiments of the present invention are directed to a simple, fast, vision-based method and system for calibrating the tool-frame 120 of a robot 102, such as an industrial robot 102.
Usually in the case of kinematic calibration, which typically deals directly with identifying and compensating for errors in the robot's 102 kinematic model, the tool-frame 120 is either assumed to be known or is included as part of the full calibration procedure. A large number of tools 112, including welding and cutting tools, may not be capable of providing any information about the tool's 112 own position or orientation. In contrast, various embodiments offer a method of calibrating the tool-frame 120 quickly and accurately without including the kinematic parameters.
The tool-frame 120 calibration algorithm of the various embodiments offers several advantages. First, a vision-based method is very fast while still delivering excellent accuracy. Second, minimal calibration and setup is required. Third, the various embodiments are non-invasive (i.e., require no contact with the tool 112) and do not use special hardware other than a camera, enclosure, and associated image acquisition hardware. While vision-based methods are not appropriate for every situation, using them to calibrate the tool-frame 120 of an industrial robot offers a fast and accurate way of linking the offline programming environment to the real world.
In practice, the mathematical kinematic model of the robot 102 will invariably be different than the real manipulator 102. The differences cause unexpected behaviors and positioning errors. To help alleviate the unexpected behaviors and positioning errors, a variety of calibration techniques may be employed to refine and update the mathematical kinematic models used. The various calibration techniques attempt to identify and compensate for errors in the robotic system. The errors typically fall into two general categories. The first kind of error that occurs in robotic systems is geometric error, such as an incorrectly defined link length in the kinematic model. The second type of error is called non-geometric error, which may include temperature effects, gear backlash, loading, and the un-modeled dynamics of the robotic system. While both types of errors may have a significant effect on the positioning accuracy of the system, geometric errors are typically the easiest to identify and correct. Non-geometric errors may be difficult to compensate for, due to being linked to the basic mechanical structure of the robot and the possibility that some of the non-geometric errors may change rapidly and significantly during robot 102 operation (e.g., temperature effects, loading effects, etc.).
Robot 102 calibration is typically divided into four steps: selection of the kinematic model, measurement of the robot's 102 pose, identification of the model parameters, and compensation of robot 102 pose errors. The measurement phase is typically the most critical, and affects the result of the entire calibration. Many different devices have been used for the measurement phase, including Coordinate Measuring Machines (CMMs), theodolites, lasers, and visual sensors. Visual sensors, in particular Charge-Coupled Device (CCD) array cameras, have the advantage of being relatively inexpensive, flexible, and widely available. It is important to note that in order to use a camera as a measuring device, the camera may also need to be calibrated correctly.
Overall kinematic calibration of the robotic manipulator 102 is very important for positioning accuracy, and may include tool-frame 120 calibration. However, there may also be a need for independently calibrating the tool-frame 120, which arises from a number of sources. First, many robotic systems come with pre-calibrated kinematic models that do not include the actual tool 112 that will be used. Also, some commercial robot controllers do not allow access to the kinematic parameters, or modifying the kinematic parameters is beyond the expertise of the average robot user. Additionally, in some systems the tool 112 is often changed, or may be damaged or bent if the robot 102 crashes into a fixed object. For many applications (e.g., welding), it is critically important to have a good definition of the tool-frame 120. The method and system of the various embodiments provides for quick and accurate calibration of the tool-frame 120 without performing a full kinematic calibration of the robot 102 such that the tool-frame 120 is independently calibrated. The basic issue addressed by the various embodiments is, assuming that the wrist 110 pose in the world-frame 114 is correct, what is the position and orientation of the tool-frame 120 relative to the wrist 110? For the various embodiments, the wrist 110 pose is assumed to be accurate. The method of the various embodiments is generally concerned with computing an accurate tool-frame 120 relative to the wrist-frame 118, which means that the rest of the robot 102 pose may become irrelevant.
The remainder of the Detailed Description of the Embodiments is organized into five main sections. The first section deals with the methods used by various embodiments to calibrate the tool-frame 120 assuming that the wrist 110 position is correct. In particular, an analysis of the tool-frame 120 calibration problem and methods for tool-frame 120 calibration are described. The second section describes vision and camera calibration. The third section describes the application of a vision system to enforce a constraint on the tool so that the previously developed methods may be used for tool-frame calibration. The fourth section describes the results of simulations and testing with a real robotic system. The fifth section describes Appendices for supporting concepts including some properties of homogeneous difference matrices (Appendix A), as well as detailing the use of Singular Value Decomposition (SVD) for least-squares fitting (Appendix B).
Calibrating the Tool-Frame
Calibrating the tool-frame of the tool 214 may be divided into two separate stages. First the Tool Center Point (TCP) 216 is found. Next the orientation of the tool 214 relative to the wrist 220 may be computed if the TCP location is insufficient to properly model the tool. For some tools, a third calibration stage may be added to address properly situating the tool for an operation direction (e.g., a two-wire welding torch that should have the two wires aligned along a weld seam).
Tool-Frame Cal. Stage 1: Calibrating the Tool Center Point (TCP)
A technique is described below for computing the three-dimensional (3-D) vector from the origin of the wrist-frame to the origin of the tool-frame, given that the TCP 216 is physically constrained in the world-frame Rw 202. The specific constraints that are used are typically simple and geometric, including constraints that the TCP 216 be at a point or lie on a line. To say that the TCP 216 is physically constrained means that the wrist 220 of the robot will be moved to different poses Wri 218 while the TCP 214 remains at a point or on a line. This technique will work for any tool 214, as long as the TCP 216 location may be measured and the geometric constraint may be enforced. The calibration of the TCP 216 to the wrist 220 may be accomplished by a number of methods, including torque sensing, touch sensing, and visual sensing.
To calculate the TCP 216, something may need to be known about the position of the TCP 216 or the pose of the wrist Wri 218. For example, constraining the wrist 220 and measuring the movement of the TCP 216 would provide enough information to accomplish the tool-frame calibration. However, with the TCP as the variable in the calibration 224, it is assumed that nothing is known about the tool 214 before calibration 224. Modern robot 222 controllers allow full control of the position and orientation of the wrist 220, so it makes more sense to constrain the TCP 216 and use the full pose information of the wrist poses Wri 218 to calibrate 224 the TCP 216.
The problem of finding 224 the TCP 216 may be examined in both two and three dimensions (2-D and 3-D), although in practice the three-dimensional case is typically used. However, the two-dimensional case provides valuable insight into the problem. To discuss the two-dimensional TCP 216 calibration 224 problem, several variables must be defined. In two dimensions, the TCP 216 is denoted as in Eq. 1.
t=(tx ty 1)T Eq. 1
And in three dimensions the TCP 216 is denoted as in Eq. 2.
t=(tx ty tz 1)T Eq. 2
Note that the vector t is specified with respect to the wrist 220 coordinate frame 218. Homogeneous coordinates are used so that the homogeneous transformation representation of the wrist-frames Wri 218 may be used. The ith pose of the robot wrist-frame Wri 218 may be denoted as in Eq. 3.
Where Ti is the translation from the origin of the world-frame Rw 202 to the origin of the ith wrist-frame Wri 218, and Ri is the rotation from the world-frame Rw 202 to the ith wrist-frame Wri 218. In two dimensions the Wi matrix is of size 3×3, while in three dimensions the Wi matrix is of size 4×4. The ith wrist-frame Wri 218 pose information is available from the kinematics 226 of the robot 222, which is computed in the robot controller.
The position pi of the TCP 216 in the world coordinate system Rw 202 for the ith wrist pose Wri 218 may be computed as in Eq. 4.
pi=Wit i=1, . . . , N Eq. 4
Where Wi is the transformation from the ith wrist-frame Wri 218 to the world coordinate frame Rw 202.
A point constraint means that the position of the TCP 216 in the world-frame Rw 202 is the same for each wrist pose Wri 218, as shown in Eqs. 5 and 6.
p1=p2= . . . =pN Eq. 5
or
W1t=W2t= . . . =WNt Eq. 6
Meaning that any two of the points pi of the TCP 216 are equal as in Eq. 7.
W
i
t−W
j
t=(Wi−Wj)t=0 i≠j Eq. 7
To obtain information from a point constraint, at least two wrist poses Wri 218 are needed. If more than two wrist poses Wri 218 are available, the constraints may be stacked together into a matrix equation of the form shown in Eq. 8.
Where the matrix is called the constraint matrix and is denoted by A.
Stacking the constraints as in Eq. 8 prevents duplication while covering the possible combinations. In fact, each additional wrist pose Wri 218 provides an increasing number of constraints that may be used to increase accuracy when there are small errors in Wri 218 as may appear in a real world system. Because the order of the terms in each constraint is unimportant (i.e., W1−W2 is equivalent to W2−W1), the number of constraint equations, denoted M, may be determined as the number of combinations of wrist poses Wri 218 taken two at a time from the set of all available wrist poses Wri 218 as described in Eq. 9.
For example, when N=3, the number of constraints in Eq. 8 is shown in Eq. 10.
In Eq. 8, it may be seen that t is in the null space of the constraint matrix. Because t is specified in homogeneous coordinates, the last element of t must be equal to one. Therefore, as long as the dimension of the null space of the constraint matrix is less than or equal to one, the solution may be recovered by scaling the null space. If the dimension of the null space is zero, then t is the null vector of the constraint matrix. If the dimension of the null space is one, then t may be recovered by scaling the null vector of the constraint matrix so that the last element is equal to one.
To find the null space of the constraint matrix, the Singular Value Decomposition (SVD) may be used. Applying the SVD yields Eq. 11.
A=UΣVT Eq. 11
Where Σ is a diagonal matrix containing the singular values of A. U and V contain the right and left singular directions of A, and the null space of A is the span of the right singular vectors corresponding to the singular values of A that are zero because the singular values represent the scaling of the matrix in the corresponding singular direction, and the null space contains all vectors that are scaled by zero. Note that in practice the minimum singular values will likely never be exactly zero, so the null space will be approximated by the span of the singular directions corresponding to the singular values of A that are close to zero.
Using the SVD to find the null space and then scaling the singular direction vector appropriately to recover t works as long as the dimension of the null space of the constraint matrix is less than or equal to one. It is clear that the dimension of the null space is related to the number of poses Wri 218 used to build the constraint matrix, and that a minimum number of poses Wri 218 will be required in order to guarantee that the dimension of the null space is less than or equal to one.
The minimum number of poses Wri 218 depends on the properties of the matrix that results from subtracting two homogeneous transformation matrices (see Appendix A section below). For convenience, the matrix resulting from subtracting two homogeneous transformation matrices will be called a homogeneous difference matrix. The constraint matrix is a composition of M of the homogeneous difference matrices. Because the Wi's are homogeneous transformation matrices, the last row of each Wi is (0, 0, . . . , 1)T. Therefore, when two homogeneous transformation matrices are subtracted, the last row of the resulting matrix is zero as in Eq. 12.
It is clear that the matrix of Eq. 12 will not be of full rank. For example, in the two-dimensional case, with two wrist poses Wri 218, the dimension of the constraint matrix is 3×3, but the maximum rank of the matrix of Eq. 12 is two. However, it turns out that the rank of the constraint matrix in the case of Eq. 12 is always two as long as the two wrist poses Wri 218 have different orientations, which means that the dimension of the null space is guaranteed to be at least one. Therefore, the minimum number of wrist poses Wri 218 to obtain a unique solution for t in the two-dimensional point constraint case is two.
In the three-dimensional point constraint case the situation is more complicated. For two wrist poses Wri 218, the dimension of the constraint matrix is now 4×4. The last row of the constraint matrix is zero, as in the two-dimensional point constraint case. Therefore, the rank of the constraint matrix cannot be more than three. However, the rank of the constraint matrix is in fact only two, because all four columns in the three-dimensional homogeneous difference matrix are coplanar. To help understand, first note that Property A2 (see Appendix A section below) states that the vectors in the upper left 3×3 block of the difference matrix are coplanar. To show that the fourth column is contained in the same plane, it is helpful to draw a picture.
Because t is specified in homogeneous coordinates, any vector in the null space may be scaled so that the last element is one, which reduces the solution space to a line instead of a plane. However, reducing the solution space to a line is still insufficient to determine a unique solution for t, meaning that an additional wrist pose is needed. Adding a third wrist pose increases M to three, and increases the dimension of the constraint matrix A of Eq. 14 to 12×4
As long as none of the wrist poses are equal, the rank of the constraint matrix A increase to three, which enables a unique solution for t to be found. Therefore, the minimum number of wrist poses to obtain a unique solution for t in the three-dimensional point constraint case is three.
Using the set of points Wit 506, 510, 514, 518, the condition that connecting lines between points Wit 506, 510, 514, 518 are parallel is described by Eq. 14 below.
(Wit−Wjt)∥(Wjt−Wkt) Eq. 14
Using the dot product, the parallel condition in Eq. 14 may be expressed as in Eq. 15.
((Wi−Wj)t)T((Wj−Wk)t)=C Eq. 15
Where the raising to the power of T is an indication of the transposition of the matrix and C is a constant related to the magnitude of the differences between Wit, Wjt, and Wkt. In particular, the constant C is shown in Eq. 16.
C=∥(Wi−Wj)t∥∥(Wj−Wk)t∥ Eq. 16
If the transposed term in Eq. 15 is expanded, the resulting expression is a quadratic form as shown in Eq. 17 below.
t
T(Wi−Wj)T(Wj−Wk)t−C=0 Eq. 17
Eq. 17 is a quadratic form because it is of the form shown in Eq. 18.
t
T
Qt+b
T
t+c=0 Eq. 18
Where Q in Eq. 18 may be defined by Eq. 19.
Q=(Wi−Wj)T(Wj−Wk), b=0, and c=−C Eq. 19
Each additional wrist pose introduces an additional quadratic constraint of the form shown in Eq. 17. Even though Eq. 9 shows that the number of combinations of wrist poses 508, 512, 516, 520 taken two at a time increases significantly with each additional wrist pose, most of the combinations are redundant when the parallel lines constraint is used. For example, for wrist poses W1 508, W2 512, W3 516, if (W1-W2)t is parallel to (W2-W3)t, then (W1-W2)t is also parallel to (W1-W3)t. Therefore, each additional wrist pose 508, 512, 516, 520 only adds one quadratic constraint.
The matrix Q in the Eqs. 18 and 19 determines the shape of the conic representing the quadratic constraint. If Q is full rank, the conic is called a proper conic. If the rank of Q is less than full, the conic is called a degenerate conic. Proper conics are shapes such as ellipses, circles, or parabolas. Degenerate conics are points or pairs of lines. To determine what sort of conic is represented for the case of the condition that lines connecting the points Wit 506, 510, 514, 518 are parallel, the rank of Q must be known. Eqs. 20 and 21 are arrived at using the properties of the rank of a matrix.
rank(Wi−Wj)T=rank(Wi−Wj) Eq. 20
rank((Wi−Wj)T(Wj−Wk))≦min(rank(Wi−Wj) rank(Wj−Wk)) Eq. 21
As shown above, in two dimensions, the rank of Wi−Wj is no more than two, which would seem to mean that the conic represented by Q for the parallel line condition would always be degenerate, but because homogeneous coordinates are being used, the conic represented by Q for the parallel condition only results in a degenerate shape if the rank of Q is strictly less than two. In three dimensions, less may be said about the rank of Q because the homogeneous difference matrices could be of rank two or three. So the conic shape could either be a proper conic in three variables or a degenerate conic.
The properties of Q for the parallel condition may be used to determine the minimum number of wrist poses 508, 512, 516, 520 required for a unique solution for t in the line constraint 504 case. As described above, it was observed that at least three wrist poses are needed to obtain one quadratic constraint. The rank of Q is at most two, meaning that the shape of the curve is some sort of quadratic curve in two variables (e.g., a circle or an ellipse). To ensure a discrete number of solutions another wrist pose 508, 512, 516, 520 must be added to introduce a second constraint. Hence, the minimum number of wrist poses 508, 512, 516, 520 required for a solution for t in the line constraint case is four in both two and three dimensions.
It is interesting to note that for any two wrist poses 508, 512, 516, 520 in two dimensions a TCP may be found which satisfies the point constraint, meaning that for any three wrist poses 508, 512, 516, 520, a point constraint solution may be found for two of the wrist poses 508, 512, 516, 520, causing two of the world coordinate points to be the same. This reduction in the number of available points from three to two causes the solution for the line constraint problem to be trivial, also indicating that a fourth wrist pose 508, 512, 516, 520 is needed.
As described above, the location of the TCP relative to the wrist-frame (i.e., the translation of TCP to the wrist-frame) may be performed with a minimum of three wrist poses 508, 512, 516, 520 for a 3-D point constraint or four wrist poses 508, 512, 516, 520 for a 3-D line constraint. Although the TCP relative to the wrist-frame may be calculated with the minimum number of required wrist poses 508, 512, 516, 520, it may be beneficial to use more wrist poses 508, 512, 516, 520. For some embodiments, the number of wrist poses 508, 512, 516, 520 may exceed the minimum number of wrist poses 508, 512, 516, 520 by only a few wrist poses 508, 512, 516, 520 and still provide reasonable results. The more wrist poses 508, 512, 516, 520 obtained, the more room for error there is in enforcing the specific geometric constraint (e.g., point/line constraints). Thus, an embodiment may use a large number of wrist poses 508, 512, 516, 520 to alleviate the need for an embodiment to make minute corrections to individual wrist poses 508, 512, 516, 520. Thus, an embodiment may be preprogrammed to automatically perform the large number (30-40) of wrist poses 508, 512, 516, 520 with only corrective measurements from the camera needed to obtain a sufficiently accurate TCP translational relationship to the robot wrist. Automatically performing a large number (30-40 of wrist poses 508, 512, 516, 520 permits an embodiment to avoid a need for an operator to manually ensure that the TCP is properly constrained within the image captured by the camera. An automatic embodiment may also evenly space the wrist poses 508, 512, 516, 520 rather than using random wrist poses 508, 512, 516, 520. Using many evenly spaced wrist poses 508, 512, 516, 520 permits an embodiment to relatively easily generate the desired wrist poses 508, 512, 516, 520 as well as permitting greater control over the robot movement as whole. As may be self-evident, the wrist position and orientation for each wrist pose 508, 512, 516, 520 may be recorded in/on a computer readable medium for later use by the TCP location computation algorithms.
While the point constraint formulation in Eq. 8 may be used to solve for t by computing the SVD of the constraint matrix and then scaling the null vector, the current line constraint formulation in Eq. 17 cannot be used to solve for t because C is unknown. Therefore an iterative method was implemented to solve for t in the line constraint case. The iterative algorithm is based on the method of Nelder and Mead. For more information on the method of Nelder and Mead see W. H. Press, B. P. Flannery, and S. A. Teukolsky “Downhill simplex method in multidimensions,” Section 10.4 in Numerical Recipes in C: The Art of Scientific Computing, Cambridge University Press, pp 408-412, 1992. The Nelder and Mead method requires an initial approximation (i.e., guess) for t, and computes a least-squares line fit using the SVD (see Appendix B section below). The sum of the residuals from the least-squares fit is used as the objective function, and approaches zero as t approaches the true TCP. A version of the main TCP calibration method described above may be used to generate the initial approximation for t if no approximation exists. The main difference between the method to obtain an initial approximation for t and the method to obtain the TCP location relative to the wrist-frame is that the method to obtain an initial approximation for t moves wrist poses 508, 512, 516, 520 about the center of the robot wrist rather than the TCP of the tool because the TCP of the tool is unknown.
It is important to note that the TCP calculation algorithm described above requires that wrist pose 508, 512, 516, 520 information be gathered and a corresponding TCP translation relationship to the robot wrist-frame be performed only once to arrive at a final TCP relationship to the robot wrist-frame. That is, it is not necessary to iteratively repeat a process of performing a number of wrist poses 508, 512, 516, 520, correcting the TCP location (i.e., correcting the TCP relationship to the robot wrist-frame) over and over until the TCP location is “close enough.” An embodiment performs the desired number of wrist poses 508, 512, 516, 520 while maintaining the specified geometric constraint (e.g., point/line constraint) for the TCP location in the camera image and then calculates the TCP location relative to the robot wrist-frame using the computational algorithms described above. Also, only a single point on the tool need be identified in the image to implement the TCP calculation algorithm. Thus, it is not necessary to locate multiple useful features in the image of the tool, only a single point.
Tool-Frame Cal. Stage 2 (Optional): Calibrating the Tool Orientation
For some tools and processes, finding only the TCP relationship to the wrist frame is adequate. For example, if a touch probe extends directly along the joint axis of the last joint (i.e., the wrist), the orientation of the tool may be assumed to be equal to the orientation of the wrist. However, for many tools additional information is needed about the orientation of the tool. Welding processes, for example, have strict tolerances on the angle of the torch. For example, errors in the torch angle may cause undercut, a condition where the arc cuts too far into the metal. For the robot to have the ability to position the torch within the process tolerances, it is desirable for the orientation component of the tool-frame to be accurately calibrated.
One method of finding the tool orientation is to move the tool into a known orientation in the world coordinate frame. The wrist pose may then be recorded and the relative orientation between the tool and the wrist may be computed. However, the method of moving the tool into a known orientation in the world coordinate frame often requires a jig or other special fixture and is also typically very time consuming.
Another option is to apply the method described above for computing the tool center point a second time using a point on the tool other than the TCP. For example, the orientation of a tool may be found by performing the TCP calibration procedure using another point along the tool direction. A new point in the wrist-frame would then be computed, and the tool direction would then be the vector between this new point and the previously found TCP. Calibrating using the method described above for the TCP calibration, but for a different point on the tool has the advantage of using previously developed techniques, which also do not require specialized equipment.
Tool-Frame Cal. Stage 3 (Optional): Calibrating Tool Operation Direction
To calibrate a tool operation direction for a two-wire welding torch tool 602, an embodiment rotates and tilts the tool with the robot 608 until the front wire 604 and the back wire 606 appear as a single wire 610 in the image captured by the camera. It is not important which wire is the front wire 602 or the back wire 606, just that one wire 604 eclipses the other wire 606 making the two wires 604, 606 appear as a single wire 610 in the image captured by the camera. The position and orientation of the robot and robot wrist are recorded when the two wires 604, 606 appear as a single wire 610 in the camera image and the recorded position and orientation are built into the kinematic model of the robotic system to define an axis of the tool-frame.
Vision Concepts
Before vision may be applied to calibration of the unknown tool-frame relative to the known wrist-frame, it is desirable to understand some concepts about camera models and calibration techniques. This section on Vision Concepts presents a brief overview of the pinhole camera model, followed by a description of some techniques for calibrating a camera.
Pinhole Camera Model
To use a camera to measure objects in the real world, it is desirable to know the parameters of the camera relative to the real world. Camera calibration involves accurately finding the camera parameters, which include the parameters of the pinhole projection model (e.g., the camera-centered coordinate frame 710 of the image plane 708 and the relationship to the two-dimensional grid 704 of the frame buffer 702) as well as the position and orientation of the camera in some world-frame 718. Many methods exist for calibrating the camera parameters, but probably the most widespread and flexible calibration method is the self-calibration technique, which provides a way to calibrate the camera without the need for expensive and specialized equipment. For further information on the self-calibration technique see Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 11, pp. 1330-1334, November 2000; R. Tsai, “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses,” IEEE Journal of Robotics and Automation, Vol. 3, No. 4, pp. 323-344, August 1987; and/or R. K. Lenz and R. Y. Tsai, “Techniques for calibration of the scale factor and image center for high accuracy 3-D machine vision metrology,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 10, No. 5, pp. 713-720, September 1988.
The effect of lens distortion is often included in the projection model to increase accuracy. Lens distortion may include radial and tangential components, and different models may include different levels of complexity. Most calibration techniques, including self-calibration, can identify the parameters of the lens distortion model and correct the image to account for them.
Two-Dimensional Camera Calibration
If the measurements of interest are only in two dimensions, then the camera calibration procedure becomes relatively simple. If perspective errors and lens distortion are ignored, the only calibration that is typically necessary is a scaling factor between the pixels of the image in the frame buffer 702 and whatever units are being used in the real world (i.e., the world-frame 718). This scaling factor is based on the intrinsic camera parameters and on the distance from the camera to the object (e.g., point 716). If perspective effects and lens distortion are included, the model becomes slightly more burdensome but still avoids most of the complexity of full three-dimensional calibration. Two-dimensional camera calibrations are often used in systems with a camera mounted at a fixed distance away from a conveyor.
Three-Dimensional Camera Calibration
Full three-dimensional (3-D) calibration typically includes finding both the parameters of the pinhole camera model (intrinsic parameters) and the location of the camera in the world-frame 718 (extrinsic parameters). Intrinsic camera calibration typically includes finding the parameters of the pinhole model and of the lens distortion model. Extrinsic camera calibration typically includes finding the six parameters that represent the rotation and translation between the camera-centered coordinate frame 710 and the world-frame 718. These two steps may often be performed simultaneously, but performing the steps simultaneously is not always necessary.
In the full camera calibration, the relationship between three-dimensional points X, Y,Z,1)T in the world-frame and two-dimensional points (u, v, 1)T in the image may be expressed by Eq. 22 below.
Where R and t are the extrinsic parameters that characterize the rotation and translation from the robot world-frame 718 to the camera-centered frame 710. The parameter s is an arbitrary scaling factor. A is the camera intrinsic matrix, described by Eq. 23 below.
Where α and β are the scale factors in the image u and v axes of the two-dimensional (2-D) pixel grid 704 of the frame buffer 702, and u0 and v0 are the coordinates of the image center. In the described camera model, there are six extrinsic and four intrinsic parameters.
Partial Three-Dimensional Camera Calibration
In the interest of avoiding the use of special calibration tools (see
Because the tool itself is being used to generate the planar point correspondences, it is impossible to determine t in Eq. 22 if the TCP is unknown. Therefore, the translation portion of the extrinsic relationship is unknown and only the rotational parameters may be computed. However, for the partial 3-D camera calibration to work it is desired that the robot still be constrained to move in a plane.
It is clear that a translation of the robot wrist results in the same translation for the tool center point, regardless of the tool geometry. Thus, the wrist of the robot may be translated in a known plane and the corresponding tool center points in the image may be recorded using an image processing algorithm. The translation of the robot wrist and recording of tool center points in the image results in a corresponding set of planar 3-D points, which are obtained from the robot controller, and 2-D image points, which may then be used to compute the rotation from the camera-centered coordinate system 710 to the robot world coordinate system 718 using standard numerical methods. It is important to note that the 3-D planar points and the 2-D image points do not necessarily correspond in the real world, but in fact may differ by the uncalibrated translational portion of the tool-frame. However, this translational difference does not affect the rotation.
Another way to view the partial 3-D camera calibration is that a plane in world coordinates 718 is computed that corresponds to the image plane 708 of the camera. While the translation between the image plane 708 and the world-frame 718 cannot be found because the TCP is unknown, a scaling factor can be incorporated in a similar fashion to the 2-D camera calibration so that image information may be converted to real-world information that the robot can use. Including the scaling factor yields Eq. 24, which is a simplified relationship between image coordinates 710 and robot world coordinates 718.
Where α and β are the scaling factors from pixels in the frame buffer 702 to robot world units in the u and v directions of the frame buffer 708, respectively. R is a rotation matrix representing the rotation from the camera-centered coordinate frame 710 to the robot world coordinate frame 718. The parameters for the image center 712 are omitted in the intrinsic matrix because this type of partial calibration is only useful for converting vectors in image space to robot world space. Because the full translation is unknown, no useful information is gained by transforming only a single point from the image into robot space. The vectors of interest in the image are independent of the origin 712 of the image frame 710, so the image center 712 is not important and need not be calibrated for the vision-based tool center point calibration application.
In Eq. 24, the rotation matrix is calibrated using the planar point correspondences described above. The scale factors are usually found by translating the wrist of the robot a known distance and measuring the resulting motion in the image. The desired directions for the translations of the wrist of the robot are the u and v directions of the frame buffer 702 of image plane 708, which may be found in robot world coordinates 718 through the previously computed rotation matrix of the partial 3-D camera calibration. This simplified extrinsic relationship allows vectors in the image frame 710 to be converted to corresponding vectors in robot world coordinates 718.
Note that in the partial 3-D camera calibration process, there are only three extrinsic and two intrinsic parameters that must be calibrated, which is a significant reduction from the full 3-D camera calibration. Also note that the vectors in robot world coordinates 718 will all lie in a plane. Because of this, the partial 3-D camera calibration is only valid for world points 718 in a plane. As soon as the robot moves out of the plane, the scaling factors will change slightly. However, it turns out that the partial 3-D camera calibration gives enough information about the extrinsic camera location to perform several interesting tasks, including calibrating the TCP.
Using Vision to Calibrate a Tool
With the framework for calibrating a tool with a simple geometric constraint described above, the use of a vision system to actually perform this calibration may be described. The camera may be used to continuously capture an image of the target tool in real-time. Embodiments may store an image and/or images at desired times to perform calculations based on the stored image and/or images.
Extraction of TCP Data
A process for extracting the two-dimensional tool center point and orientation from the camera image may be as follows and as shown in
Enforcing Point and Line Constraints
It is important to note that if the partial 3-D calibration method is used for calibrating the camera, the extrinsic parameters of the camera calibration are only valid for a single plane in the robot world coordinates. In practice, however, the TCP 1204 of the robot's tool 1202 will move out of the designated plane. Therefore, care should be taken when using image vectors to generate 3-D motion commands for the robot, because the motion of the robot will not always exactly correspond to the desired motion of the TCP 1204 in the image. To overcome the non-correspondence between the motion of the TCP 1204 and the motion of the robot, a kind of visual servoing technique may be used. In the visual servoing technique, the TCP 1204 of the tool 1202 is successively moved closer 1206 to the desired point 1208 in the image until the TCP 1204 is within a specified tolerance of the desired point 1208. The shifts 1206 in the TCP 1204 location in the image should be small so that the TCP 1204 location in the image is progressively moved closer to the desired image point 1208 without significantly going past the desired point 1208. Various schemes may be used to adjust the shift 1206 direction and sizes that would achieve the goal of moving the TCP 1204 in the image to the desired image point 1208. A more proper statement of how the shifts 1206 are implemented may be a shift in the robot wrist pose that causes a corresponding shift 1206 in the TCP 1204 location in the image.
Various embodiments may choose to increase the number of wrist poses to 30-40 wrist poses and collect correction measurements of the location of the TCP 1204 in the image with regard to the desired image point 1208 and then apply the correction measurements from the camera to the wrist pose position and orientation data that generated the TCP 1204 locations. While the correction measurements from the camera may not be as accurate as moving the wrist pose until the TCP 1204 is at the desired point 1208 on the image, the large number of wrist poses provides sufficient data to overcome the small accuracy problems introduced by not moving the TCP 1204 to the desired image point 1208.
Using Vision to Compute Tool Orientation
One way of computing the tool orientation using a vision system is to measure the angle between the tool 1202 and the vertical direction in the image. Using the partial 3-D camera calibration, the robot may be commanded to correct the tool orientation by a certain amount in the image plane. The tool 1202 may then be rotated 90 degrees about the vertical axis of the world-frame and the correction may be repeated. This ensures that the tool direction is vertical, which allows computation of the tool orientation relative to the wrist-frame. However, this method is iterative and time-consuming. A better method would use the techniques already developed for finding the TCP 1204 relative to the robot wrist-frame with a second point on the tool 1202.
The information gained from the image processing algorithm includes the TCP 1204 relative to the wrist-frame and the tool direction in the image. The TCP 1204 relative to the wrist-frame and the tool direction in the image may be used to find a second point on the tool that is along the tool direction. If the constraints from the Calibrating the Tool-Frame section of this disclosure are applied to the new/second point, the TCP calibrating method described in the Calibrating the Tool-Frame section may be used to find the location of the new/second point relative to the wrist-frame. The tool orientation may then be found by computing the vector between this new/second point and the previously calculated TCP relative to the wrist-frame.
To implement an embodiment an external camera may be used to capture the image of the tool. Some embodiments may have a separate camera and a separate computer to capture the image and to process the image/algorithm, respectively. The computer may have computer accessible memory (e.g., hard drive, flash drive, RAM, etc.) to store information and or programs needed to implement the algorithms/processes to find the tool-frame relative to the wrist-frame of the robot. The computer may send commands to and receive data from the robot and robot controller as necessary to find the relative tool-frame. While the computer and camera may be separate devices, some embodiments may use a “smart camera” that combines the functions of the camera and computer into a single device. The computer may be implemented as a traditional computer or as a less programmable firmware (i.e., FPGA, ASIC, etc.) device.
In order to provide a better and/or clearer image from the camera, additional filters may be added to deal with reflections and abnormalities seen in the image (e.g., scratches in the lens cover, weld splatter, etc.). One example filter that may be implemented is to reject portions of the image that are close to the edges of the image.
Results
In order to gain some insight into the tool calibration method and verify the analysis from the Tool-Frame Cal. Stage 1: Calibrating the Tool Center Point (TCP) section above, simulations in two and three dimensions were performed. Data was also collected using a real robotic system.
Two-Dimensional Simulation Results
In the two-dimensional case, it is useful to visualize the possible solutions by varying the TCP over a particular range and performing a least-squares fit for the particular constraint (see Appendix B section below). If the TCP is a solution, the sum of the residuals in the least-squares fit will be zero. The error for the solution may be written as in Eq. 25.
Where ci is the point on the constraint geometry that is closest to pi. In the point case, ci is the centroid, and in the line case ci is the point on the line closest to pi.
For example, in the point constraint case, the TCP was varied over a two-dimensional range, and the set of points pi in the world-frame were computed for each possible TCP. The least-squares fit was then computed, and the residuals were computed as the magnitude of the difference between the point pi and the centroid of the points p0. When t is close to the true TCP, the sum of the residuals is very small. The wrist poses were manually positioned by the user in these simulations, introducing some error into the wrist pose data. In the simulations the true TCP was set to be (50,50,1)T.
For a simulation of a point constraint with two wrist poses, the result agrees with the result from the analysis in the Tool-Frame Cal. Stage 1: Calibrating the Tool Center Point (TCP) section above, which concluded that two wrist poses is sufficient for a unique solution for t. To analyze the simulation, the solution is the value of t for which ε is small, which corresponds to a single minimum in a plot of the results. To solve for the TCP, first the constraint matrix is formed. From Eq. 8 A (the constraint matrix) is computed to be:
Note that the last row in A is zero, indicating that the matrix is singular. The singular value decomposition is:
Therefore, the null space of A is spanned by the third singular vector of V, (0.702,0.712,0.014)T, which corresponds to the zero singular value. Because homogeneous coordinates are used, the correct TCP will be a scaled version of vector V so that the last element is one. For 2-D two wrist pose example, scaling the vector V appropriately yields (50.14,50.86,1)T. The actual TCP was (50,50,1)T, so the algorithm returned a substantively correct solution. The difference between the two vectors calculated and actual vectors may be due to both round off errors and errors in the wrist pose data.
In the line constraint case, a similar procedure may be followed to verify the analysis in the Tool-Frame Cal. Stage 1: Calibrating the Tool Center Point (TCP) section above. For a line constraint with three wrist poses, the analysis the Tool-Frame Cal Stage 1: Calibrating the Tool Center Point (TCP) section above indicated that the solutions for the line constraint case with three wrist poses satisfied a single quadratic constraint in two dimensions. A plot in of the solutions clearly showed that the solutions lied on a circle resulting from the quadratic constraint. Thus an incorrect solution that still satisfies the quadratic constraint may be found.
After adding a fourth wrist pose the solutions plot looked similar to the three wrist pose line constrain case, but does indeed have only a single solution. In the plot, the minimum is not very clearly defined and has the potential to cause numerical problems that could affect the solution. However, the definition problems may be caused by the fact that the wrist poses were all fairly similar in orientation, differing by at most 90 degrees. A solution to the conditioning problem is to change the relative orientations of the wrist poses. A plot of a simulation that radically changes the orientation of one of the wrist poses has a minimum that is much more clearly defined. Therefore, the problem is better conditioned indicating that the difference between the wrist poses is important, and that a wide variety of wrist poses may help the conditioning of the problem.
It may also be useful to examine the effect of errors or noise in the wrist pose data on the final solution for the TCP. In practice, the wrist poses will have errors resulting from several sources, including vision and kinematic errors. To simulate this effect, Gaussian noise of zero mean and variable magnitude was added to the wrist pose data before the TCP was computed. A plot of the error in the TCP computation as the noise level increases suggested two practical tips for using the TCP calibration algorithm. First, using more wrist poses than necessary helps to decrease the effect of errors in the wrist pose data. Second, it is important for the numerical conditioning of the problem to have the wrist poses be as different as possible. However, in practice using the tips may not always be achievable because the robot's work cell may have other obstacles to the robot's potential motion. Also, if vision is used, the TCP must remain in the field of view of the camera.
Three-Dimensional Simulation Results
Visualizing the solutions to the problem in three dimensions is harder, but may be accomplished through the use of volume rendering and contour surfaces. In a volume plot, the contour levels of the function are seen as surfaces, while the actual function value is represented by a color. The data for a three-dimensional simulation was generated for using a similar program as for the two-dimensional case, in which the user manually positioned a number of wrist-frames on the computer screen. The TCP was then varied over a specified range, and the sum of the residuals was computed in a least-squares fit of the constraint to the points pi. The error, ε, was then visualized as a volume rendering.
As with the two-dimensional case, the point constraint was considered first. As described in the Tool-Frame Cal. Stage 1: Calibrating the Tool Center Point (TCP) section above, for a three-dimensional embodiment, the point constraint was deemed to require three wrist poses to obtain a TCP relationship to the robot's wrist-frame. The color of the volume rendering plot for the three-dimensional point constraint simulation with only two wrist poses showed the magnitude of the objective function (i.e., error in least-squares fit). The contour surfaces of the function gave some idea of where the solutions were. Because the contour surfaces in the plot were becoming smaller and smaller cylinders, the solutions lied on a line. Having a line of solutions agrees with the three-dimensional point constraint analysis in the Tool-Frame Cal. Stage 1: Calibrating the Tool Center Point (TCP) section above because there was an incorrect solution that still satisfies the constraint equations confirming that more than two wrist poses are needed for a three-dimensional point constraint. When three wrist poses were used for the three-dimensional point constraint, the volume plot illustrated that with the additional wrist pose the contour surfaces converged to a point, meaning that there is a single solution. Thus, it was confirmed that at least three wrist poses are needed to find the TCP when a three-dimensional point constraint is used. Similar to the two-dimensional case, more than three poses may be used to reduce the effect of errors in the wrist pose data.
The line constraint in three dimensions was examined next. A simulation with only three wrist poses was performed with the 3-D line constraint. The contour surfaces of the plot of the simulation of the 3-D line constraint with three wrist poses showed that the solutions lied on a curve in space. From the analysis in the Tool-Frame Cal. Stage 1: Calibrating the Tool Center Point (TCP) section it is apparent that the solution curve in space shows the solution curve is a quadratic sort of curve, which may be proper or degenerate. Thus, the results showed that an incorrect solution may result from using only three wrist poses, illustrating the need for an additional wrist pose.
When four wrist poses were used for the simulation of the 3-D line constraint the contour surfaces of the plot showed a closed shape indicating that there is only one solution for the 3-D line constraint with four wrist poses. Thus, it is clear that no fewer than four wrist poses are required to compute the TCP if the three-dimensional line constraint is used, which confirms the analytical result in the Tool-Frame Cal. Stage 1: Calibrating the Tool Center Point (TCP) section.
Real System Testing Results
After the three-dimensional simulation and analysis, a preferred method was chosen for implementation on a real system. In particular, the three-dimensional line constraint was easy to apply with a vision-based tool calibration and was chosen for a real world implementation. The TCP calibration method described in the disclosure above was implemented and tested using an industrial welding robot with a standard MIG welding gun. The tool calibration software was implemented on a separate computer, with communication to the robot controller occurring over standard communication link. With the aid of some short programs written for the robot, the calibration software was able to command the robot to move to the positions required. A black and white, digital camera was used, which was interfaced to the calibration software through a standard driver.
First the intrinsic calibration of the camera was performed using the self-calibration method with a checkerboard calibration pattern. The pattern was attached to a rigid metal plate to ensure its planar nature. Table 1 shows the calibrated intrinsic parameters of the camera. Because the camera is only used to ensure that the TCP's are at the same point in the image, it is not necessary to consider lens distortion for the TCP calibration application. Lens distortion is a more important issue when the camera is to be used for making accurate measurements over a large area in the image.
After the intrinsic parameters of the camera were calibrated, the partial 3-D calibration procedure discussed in the Partial Three-Dimensional Camera Calibration section above was performed. A manual tool calibration was also carried out for the tool, using the software in the robot controller. The value of the TCP obtained through the manual method was (−125.96,−0.55,398.56)T, measured in millimeters. The orientation of this particular tool was the same as the orientation of the wrist.
It is important to note that without specialized measuring equipment, it is impossible to determine the tool center point relative to the robot world-frame for comparison to the tool center point calculated by the invention as was done for the simulation discussed in the disclosure above. Computer Aided Design (CAD) models are inadequate representations of the real tool, and all of the tool calibration methods available have some error. Therefore, the analyses presented here used a rough vision-based measure of error to give an idea of the performance of the method.
In an attempt to assess the true accuracy of the automatic method compared to the manual method, a vision-based measure of error was applied. It may be observed that if the robot has an incorrect tool definition and the robot is then commanded to rotate about the TCP, the tip of the real tool will move in space by an amount related to the error in the tool definition. To measure this error, the tool is moved to an arbitrary starting location and the image coordinates of the TCP are recorded. The tool is then rotated about each axis of the tool-frame individually by some amount, and the image coordinates of the TCP are recorded after each rotation. The image coordinates of the TCP for the starting location is then subtracted from the recorded TCP's, and the norm of each of the three difference vectors is computed. The error measure is then defined as the sum of the norms of the difference vectors. Note that the error measurement does not provide specific information about the direction or real world magnitude of the error in the tool definition, but instead provides a quantity that is correlated to the magnitude of the true error.
To assess the validity of the error measurement, the error measurement was applied to a constant tool definition 30 times and the results were averaged. A plot showing the average error for the particular TCP and the standard deviation for the data was created. The standard deviation is the important result from the experiment because the standard deviation gives an idea of the reliability of the error measurement. The standard deviation was just over one pixel, which means that the errors found in subsequent experiments were probably within one or two pixels of the true error. The one or two pixel deviation is most likely due to the image processing algorithm, which does not return exactly the same result for the image TCP every time. However, a standard deviation of one pixel is considered acceptable and shows that the results obtained in the subsequent experiments are valid.
To measure the performance of the TCP calibration algorithm, the TCP was calculated at increasing envelope angles 1308 within the usable range. The average of three trials was taken, and the results were plotted. While the data was somewhat erratic, the plot still generally trended downward, which means that larger envelope angles 1308 do, in fact, reduce the error in the computation. In fact, increasing the envelope angle 1308 from ten to twenty degrees reduced the error by a factor of two. A conclusion from the real world experiment is that, in the interest of accuracy and consistency, it is better to use as large an angle as possible given the field of view of the camera, agreeing with the results obtained through simulation.
Given the real world experiment data, it is reasonable to conclude that an effective technique for increasing the accuracy of the TCP is to use a large envelope angle 1308 in order to maximize the difference between the wrist poses 1302. To avoid issues with the camera's field of view, the method could also be performed once with a small envelope angle 1308 to obtain a rough TCP, and then repeated with a large envelope angle 1308 to fine-tune the result.
The vision-based error measurement was also applied in order to compare the manually and automatically defined TCP's. The automatic method used four wrist poses with an envelope angle 1308 of twenty degrees. The TCP was defined ten times with each method (automatic and manual) to obtain a statistical distribution.
The average TCP's for each method (manual or automatic) are very similar, which means that the automatic method is capable of determining the correct TCP. The standard deviations for the automatic method are generally around 0.5 millimeters, which is a good result because the result indicates that the automatic method is consistent and reliable.
Table 2 shows the result of applying the error measurement to both the automatic TCP and the manually defined TCP. The errors in the automatic and manual methods are almost identical. This means that the automatic method does not offer an accuracy improvement over the manual method, but that it is capable of delivering comparable accuracy. While accuracy is an important factor, there are also other advantages to the automatic method.
One of the primary advantages of the visual tool-frame (TCP) calibration is speed. Even with a skilled operator, manual methods and some other automatic methods may take ten minutes to fully define an unknown tool, while the vision-based method yielded calibration times of less than one minute. Incorporating an initial approximation (i.e., guess) or increasing the robot's velocity between wrist poses 1302 may further reduce calibration time.
Various embodiments have been described for calibrating the tool-frame of a robot quickly and accurately without performing the full kinematic calibration. The accuracy of the vision-based TCP calibration method is comparable to other methods, and the techniques of the various embodiments provide an order of magnitude in speed improvement. The TCP computation method described herein is robust and flexible, and is capable of being used with any type of sensing system, vision-based or otherwise.
A challenging portion of this application is the vision system itself. Using vision in uncontrolled industrial environments can present a number of challenges, and the best algorithm in the world is useless if reliable data cannot be extracted from the image. A big problem for vision systems in industrial environments is the unpredictable and often hazardous nature of the environment itself. Therefore, the calibration systems must be robust and reliable, a task which is difficult to achieve. However, with careful use of robust image processing techniques, controlled backgrounds and lighting, reliable performance may be achieved.
The TCP calibration method of the various embodiments may be used in a wide variety of real world robot applications, including industrial robotic cells, as a fast and accurate method of keeping tool-frame definitions up to date in the robot controller. The speed of the various embodiments allows for a reduction in cycle times and/or more frequent tool calibrations, both of which may improve process quality overall and provide one more small step toward true offline programming.
Appendix A—Properties of Homogeneous Difference Matrices
Two homogeneous transformation matrices are defined as in Eq. 26.
If the difference of the two homogenous matrices is taken, the result is obviously not a homogeneous transformation anymore, but still has some interesting properties that stem from the properties of the original matrices. The resulting homogeneous difference matrix may be expressed as in Eq. 27.
The first interesting property of the homogeneous difference matrix may be stated as follows:
(0 . . . 0 1)T
is a basis vector for the null space, and the dimension of the null space is one.
In the illustration 1400, p1-p2 1406 is perpendicular to v 1410. The perpendicular nature of the difference vector 1406 is true of the difference vector 1406 between any point 1402 and the new rotated location 1404 of the point, meaning that subtracting two rotation matrices results in a new matrix consisting of the vectors between points on the old coordinate axes and points on the new coordinate axes. The difference vectors 1406 are coplanar, according to the argument given above. In fact, the difference vectors 1406 are contained in the plane whose normal is the equivalent axis of rotation 1410. One of the implications of the perpendicular property of the difference vectors 1406 is that only two of the three vectors in the difference of two rotation matrices are linearly independent. In fact, it turns out that only two of the columns in a three-dimensional homogeneous difference matrix are linearly independent (see the Tool-Frame Cal. Stage 1: Calibrating the Tool Center Point (TCP) section above).
Appendix B—SVD for Least-Squares Fitting
For example, suppose a straight line 1510 is to be fitted to a set of 3-D points. Let pi 1506 be the ith point in a data set that contains n points. Let v0 1502 be a point on the line, and let v 1504 be a unit vector in the direction of the line. A parametric equation (Eq. 28) may be written for the line 1510, based on the point v0 1502 and the vector v 1504:
p
i=αiv+v0 Eq. 28
The distance 1508 between a point 1506 and a line 1510 is usually defined as the distance 1508 between the point 1506 and the closest point 1508 on the line 1510. The value of αi may be found for the point 1512 on the line 1510 that is closest tops 1506, which yields an Eq. 29 for the distance di 1508.
d
i
2
=∥v
0+(pi−v0)Tv·v−p1∥2 Eq. 29
If d 1508 is considered to be the ith error in the line fit, a least-squares technique may be applied to find the line that minimizes the Euclidean norm of the error, denoted ε, which amounts to finding v0 1502 and v 1510 that solve the following optimization problem of Eq. 30.
For simplicity, define qi with Eq. 31.
q
i
=p
i
−v
0 Eq. 31
Then plug qi into the objective function and expand to obtain Eq. 32.
The first term in the minimization problem above may be re-written as a maximization problem, as in Eq. 33.
Now, the sum of Eq. 33 may be rewritten as Eq. 34 using the norm of a matrix Q, which is composed of the individual components of the qi's.
So the final optimization problem is given by Eq. 36.
max∥Qv∥ Eq. 36
In the singular value decomposition of Q, the maximum singular value corresponds to the maximum scaling of the matrix in any direction. Therefore, because Q is constant, the objective function of the maximization problem is at a maximum when v 1510 is along the singular direction of Q corresponding to the maximum singular value of Q. Because all of the pi's 1506 are translated equally by the choice of v0 1502, the choice of v0 1502 does not change the SVD of Q.
For the second term in Eq. 32, in order for the sum of qi2 to be a minimum v0 1502 must be the centroid of the points because the centroid is the point that is closest to all of the data points, in a least-squares sense. Any other choice of v0 1502 would result in a larger value for the second term in Eq. 32.
Using the SVD in as described above may be applied to 2-D and 3-D lines, as well as 3-D planes. First the centroid is computed, which is a point on the line or plane. Then the singular direction corresponding to the maximum singular value of Q is computed. In the line case, the direction is a unit vector 1504 in the direction of the line 1510. In the plane case, the vector lies in the plane. For the plane case, the singular direction corresponding to the minimum singular value is the normal, which is a more convenient way of dealing with planes.
The foregoing description of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and other modifications and variations may be possible in light of the above teachings. The embodiment was chosen and described in order to best explain the principles of the invention and its practical application to thereby enable others skilled in the art to best utilize the invention in various embodiments and various modifications as are suited to the particular use contemplated. It is intended that the appended claims be construed to include other alternative embodiments of the invention except insofar as limited by the prior art.
This application is based upon and claims priority to: U.S. provisional application Ser. No. 60/984,686, filed Nov. 1, 2007, entitled “A System and Method for Vision-Based Tool Calibration for Robots,” which is specifically incorporated herein by reference for all that it discloses and teaches.
| Number | Date | Country | |
|---|---|---|---|
| 60984686 | Nov 2007 | US |
