Patent Application
20110087360
This invention relates to the use of robots for assembly of parts on a workpiece that is moving randomly on an assembly line.
Assembly tasks using industrial robots have increased in both number and complexity over the years because of the increasing requirements of product quality and quantity. However, robots performing assembly tasks are still a small portion of total robot sales each year. One of the main reasons is that it is difficult for conventional industrial robots to adjust to any sort of change such as those that may occur when the workpiece on which the parts are to be assembled is moving on an assembly line. Therefore, more intelligent industrial robotic systems are rapidly expanding the realms of possibility in assembly applications because they can perform assembly tasks with high autonomy and adaptability to the environments.
Moving line assembly, that is an assembly of parts performed while the workpiece is moving, is based on information from vision, force and position sensors. In moving line assembly, the robot can track the moving workpiece while performing the assembly process. Compared to stop-station assembly, that is, an assembly which is performed when the parts are stationary, moving line assembly can save huge amounts of time and resources.
U.S. Pat. No. 7,138,780 describes a method of processing a workpiece with a tool during a continuous movement of the workpiece, with the workpiece and tool both moving with a constant velocity. This makes the synchronization of the workpiece and tool much simpler because it can be done in the robot programming and external sensors are riot needed. U.S. Pat. No. 7,236,850 describes a method for tracking the motion of a moving object. The tracking is only performed along one direction (the moving direction). This is similar to the well known one directional conveyor tracking that uses the encoder method. Also the tracking is based on the displacement. This could cause tracking errors and eventually the failure of an assembly process.
U.S. Pat. No. 6,886,231 describes a stop-assembly-station system that includes a vision system. The vision system is used to identify the position/orientation of the part. The assembly is carried out by using the vision system alone. A vision and force control based stop-station assembly system is described in U.S. Pat. No. 6,141,863. The force control is used to execute the insertion task on the stationary workpiece and the vision system is used to identify the position/orientation of a part. Therefore, no speed control is involved in the process since the workpiece is stationary.
U.S. Pat. No. 5,040,056 describes a part picking up system. The part position/orientation is identified by the vision system and the data is transferred to the robot to pick up the part. Although the conveyor is moving, the vision system is used only for position/orientation identification. Therefore, no visual serving is involved.
There are many research papers about moving line tracking, such as Changhyun Cho, Sungchul Kang, Munsang Kim, and Jae-Bok Song, “Macro-Micro Manipulation with Visual Tracking and its Application to Wheel Assembly”, The International Journal of Control, Automation, and Systems, vol. 3, no. 3, pp. 461-468, September 2005 and Ik Sang Shin, Sang-Hyun Nam, Hyun Geun Yu, Rodney G. Roberts, Seungbin B. Moon1, “Conveyor Visual Tracking using Robot Vision”, 2006 Florida Conference on Recent Advances in Robotics, FCRAR 2006. These systems use only visual serving to track the motion of the parts to perform the assembly tasks.
A 3D vision system, that is, stereo vision, can be used to identify the position errors during the tracking of the moving workpiece. The processing of stereo vision information enables a robot to determine the position and orientation of an object in the robot coordinate frame. However, an accurate calibration between the camera coordinate system and robot coordinate system has to be realized. Because the stereo vision system requires high quality cameras and accurate calibrations, these systems are costly, error prone and not robust enough for daily use in the workshop. The complicated computation for determining the position/orientation of an object makes this system difficult to implement in real time applications, especially for fast motion tracking. Also visual serving alone could cause damage to the final products if the stereo vision system malfunctions.
An apparatus for controlling an industrial robot to assemble a part to a predetermined location on a randomly moving workpiece. The robot has a force sensor and a vision system both mounted on the robot. The apparatus has:
means for causing the robot to grip the part; and
means for simultaneously using signals from the vision system and the force sensor to control movement of the robot to a position where the gripped part can be assembled by the robot to the predetermined location on the randomly moving workpiece.
A method for using an industrial robot to assemble a part to a predetermined location on a randomly moving workpiece. The method:
causes the robot to grip the part; and
simultaneously uses signals from a vision system and a force sensor both mounted on the robot to control movement of the robot to a position where the gripped part can be assembled by the robot to the predetermined location on the randomly moving workpiece.
A computer program product for controlling an industrial robot to assemble a part to a predetermined location on a randomly moving workpiece. The computer program product includes:
computer usable program code configured to cause the robot to grip the part; and
computer usable program code configured to simultaneously use signals from a vision system and a force sensor both mounted on the robot to control movement of the robot to a position where the gripped part can be assembled by the robot to the predetermined location on the randomly moving workpiece.
A system controlling an industrial robot to assemble a part to a predetermined location on a randomly moving workpiece. The control system includes:
a computing device having therein program code usable by the computing device, the program code comprising:
code configured to cause the robot to grip the part; and
code configured to simultaneously use signals from a vision system and a force sensor both mounted on the robot to control movement of the robot to a position where the gripped part can be assembled by the robot to the predetermined location on the randomly moving workpiece.
A system for controlling an industrial robot to assemble a part to a predetermined location on a randomly moving workpiece. The system includes:
an industrial robot comprising a force sensor and a vision system both mounted on the robot; and
apparatus connected to the robot for controlling the robot, the apparatus responsive to signals simultaneously from the force sensor and the vision system to control movement of the robot after the robot has gripped the part to a position where the part can be assembled by the robot to the predetermined location on the randomly moving workpiece.
There is described herein a method and apparatus for controlling an industrial robot to perform the assembly of a part to a workpiece based on the synergistic combination of vision, force and position as the feedback information to control the robot motion while the workpiece is moving randomly. The speed errors are computed and the speed of the robot is controlled such that the tool that holds the part to be assembled to the moving workpiece tracks the motion of the randomly moving workpiece. A single camera with simple calibration combined with force control is used. As is described in more detail below, force control is applied to control the motion of the robotic system to perform the assembly task on the randomly moving workpiece.
The method and apparatus is described below, by way of example and not of limitation, in connection with the process of loading a wheel on the hub of a motor vehicle moving on an assembly line. As is described below, in this example:
1. A 2D vision system, that is, a single camera, identifies the position/orientation of the wheel and the robot tool is controlled to pick up the wheel.
2. The robot moves to a first position that is taught manually to the robot. The first position is related to a first trigger on the moving assembly line. The camera takes an image when the moving assembly line, that is, the conveyor that moves the workpiece to the workstation where the robot is located, triggers the first trigger. After image processing, the orientation pattern of the bolts or studs on the hub is identified. The robot additional axis rotates to match the holes on the wheel to the bolts on the hub.
3. While the additional axis is rotating, the robot moves to the second position that is also taught manually to the robot. When a second trigger that is also on the moving assembly is triggered, the robot controller or a computing device connected to the controller computes the speed errors of the moving workpiece and tracks the workpiece. At the same time, force control is enabled along all directions. The robot moves towards the hub because of force control along the Z-axis.
4. After the contact force along the Z-axis reaches a certain value, the assembly of the wheel to the hub is completed and the force control along the negative Z-axis retracts the tool.
5. The force control along all directions also keeps the system safe under abnormal conditions. For example, when the vision system sends the wrong signals, the force control balances the wrong signals and will not cause damage to the system and the product.
In this example, the assembly is the mounting of the wheel 34 on hub 40 of the work piece 24. The wheel is shown for ease of illustration in
As is shown in
When the moving work piece 24 enables trigger 1 this is an indication that the hub 40 can be seen by camera 28. The camera 28 takes an image and as is described in block 206 the position/orientation of the hub 40 onto which the wheel 34 is to be mounted by the robot 12 is identified by the robot controller 104. The additional axis, which is used to rotate the wheel 34 to match the bolts on the hub 40 and the holes on the wheel 34, rotates the held wheel 34 a certain angle to align the bolts on the hub 40 and the holes on wheel 34 while the robot 12 moves to the second position. The second position, which is related to trigger 2, is used to synchronize the robot 12 with the moving line 26.
When the moving work piece enables trigger 2, the camera 28 takes images continuously and the position errors of a pre-defined feature on the car body 24 is computed by the robot controller 104. One example of such a feature 44 is shown in
Using the Jacobian matrix described below, the speed errors are calculated (block 208) and at block 210 the robot 12 is controlled to follow the motion of the conveyor 26 while force control is enabled to perform the assembly task. After the wheel 34 is inserted into the hub 40, the gripper 22 releases and force control along the reverse insertion direction retracts the gripper 22 from the hub 40.
v
ref
=v+Δv
I
+Δv
f
Where
Δv
I
=J(Xd−X)
Δv
f
=K
−1(Fd−F)
Fd and F are the desired force and actual measured contact force respectively. K is the damping coefficient. Xd and X are respectively the desired position of the pre-defined feature which can be a feature, such as feature 44 of
Since the camera 28 is moving towards the insertion direction and is getting closer and closer to the feature, the desired feature position and Jacobian matrix are changing. Therefore, the desired feature position and Jacobian matrix is updated in real time.
To simplify the computation, it is assumed that the camera frame is parallel to the frame attached to the moving assembly line 26. Therefore, there is only translation movement involved. As shown in
A 3D or 2D calibration method resident either in the controller 104 or a separate computing device 102 such as a PC could be used to identify the position of the pre-defined feature 44 of
The feature 44 is measured at two positions: the start position and the end position. At both positions, the camera 28 takes images. After processing the images, the feature size (R0 and R1) and the center (X0 and X1) can be identified at the image frame. When the camera 28 moves from the start position to the end position, the desired feature center can be calculated as:
where X and R are the desired feature center and actual feature size respectively. R is computed based on real time image processing.
The Jacobian should be updated in real time. Since R can be computed in real time, the image Jacobian can be updated:
J=CJ0
Where C is the scale. Assume J0 is calibrated when the feature size is R0, then
Therefore, the Jacobian is calculated in real time.
In the example described above, the camera frame is assumed to be parallel to the frame attached to the moving assembly line 26. However, the moving line assembly methodology described herein can also be used when the camera frame is not parallel to the frame attached to the moving assembly line 26. In this case, the well known to those of ordinary skill in the art 2D/3D calibration method can be used to calibrate the relationship between the image frame and the work object frame. A transformation matrix is then obtained to transfer the coordinates in the images frame to those in the work object frame. Examples of this well known 2D/3D calibration method are described in R. J. Tsai, A Versatile Camera Calibration Technique for High Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses, IEEE Journal of Robotics and Automation, Vol. 3, No. 4, 1987, pp. 323-344 and in Z. Zhang, A flexible new technique for camera calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 22, Issue 11, Nov 2000 Page(s): 1330-1334.
As is shown in
As will be appreciated by one of skill in the art, the method and apparatus described herein may be embodied as a method, system, or computer program product. Accordingly, that method and apparatus may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.”
Furthermore, that method and apparatus may take the form of a computer program product on a computer-usable or computer-readable medium having computer-usable program code embodied in the medium. The computer-usable or computer-readable medium may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device and may by way of example but without limitation, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium or even be paper or other suitable medium upon which the program is printed. More specific examples (a non-exhaustive list) of the computer-readable medium would include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a transmission media such as those supporting the Internet or an intranet, or a magnetic storage device.
Computer program code for carrying out operations of the method and apparatus described herein may be written in an object oriented programming language such as Java, Smalltalk, C++, C# or the like, or may also be written in conventional procedural programming languages, such as the “C” programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
The method and apparatus herein is described above in connection with the process of loading a wheel on the hub of a motor vehicle moving on an assembly line. As was described above, in that embodiment the camera image is used to determine the orientation of the wheel to the studs on the hub and the robot reorients the wheel so that the holes on the wheel will line up with the studs on the hub when the robot mounts the wheel to the hub. It should be appreciated that there are other embodiments for this method and apparatus where it is not necessary to determine the orientation of the part with respect to the workpiece. For example, where the part is round in geometry and is to be mounted into a round receptacle in the workpiece either as a press fit part or onto a single central hub in the receptacle. It should also be appreciated the vision system may also be a 3D vision system that in combination with the force control is used to control movement of the robot to a position where the part can be assembled to the randomly moving workpiece.
As was described above, the method and apparatus herein may be implemented in the controller for the robot 12 or may be implemented in a system that includes a computing device that is connected to the controller. One example of such a system 100 is shown in
As can be appreciated by those of ordinary skill in the art, when the method is implemented in software in computing device 102 or controller 104, the computing device or controller functions to execute the software to thereby make the calculations of the method and system described above. The controller 104 is connected to robot 12 which is used to perform the assembly 106 of a part on a randomly moving workpiece that is described above. Thus, if the software is executed by controller 104 or if the controller 104 receives commands from computing device 102 that executes the software for the technique, the robot 12 is controlled to perform the assembly process 106 in accordance with the method and system described herein. It should be appreciated that the technique described herein can be implemented on the robot controller 104 as a software product, or implemented partly or entirely on the computing device 102 which communicates with the robot controller 104 via a communication network, such as, but not limited to, the Internet.
It is to be understood that the description of the foregoing exemplary embodiment(s) is (are) intended to be only illustrative, rather than exhaustive, of the method and system described herein. Those of ordinary skill will be able to make certain additions, deletions, and/or modifications to the embodiment(s) of the disclosed subject matter without departing from the spirit of the invention or its scope, as defined by the appended claims.
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/US09/38754 | 3/30/2009 | WO | 00 | 12/8/2010 |
| Number | Date | Country | |
|---|---|---|---|
| 61040736 | Mar 2008 | US |
