Patent Application
20120165986
The field is the use of a robot to pick parts from a bin that holds the parts to be picked by the robot.
Existing systems have used various contact or other non-vision sensing means such as force or proximity sensors to detect if one or more parts have been successfully picked from a bin that is holding the parts to be picked by the robot. These techniques can detect if only a single part has been picked by the robot from the bin, but cannot determine the orientation of the picked part or the nature of the interlocking if multiple entangled parts have been picked.
The term “bin” as used herein means, without limitation, any container, carton, box, tray or other structure that can receive and hold the parts to be picked by the robot.
A method for using a robot to pick parts from a bin. The method uses an image from one or more cameras to determine if the robot is holding one or more parts after the robot is commanded to pick one part from the bin.
A method for using a robot to pick parts from a bin. The method uses an image from one or more cameras to determine the position/orientation of a part held by the robot after the robot is commanded to pick one part from the bin.
A robotic system for picking parts from a bin. The system has a robot having a gripper for picking a part from the bin when the robot is so commanded; one or more cameras mounted to provide one or more images that show what is held by the gripper after the robot is commanded to pick a part from the bin; a computing device programmed to use the one or more images to determine if the robot is holding one or more parts after the robot is commanded to pick a part from the bin; and
the computing device further programmed to command the robot when the it is determined that the gripper is holding more than part to move the gripper and attempt to return to the bin all but one of the parts held by the gripper.
A robotic system for picking parts from a bin. The system has:
a robot having a gripper for picking a part from the bin when the robot is so commanded;
One or more cameras mounted to provide one or more images that show what is held by the gripper after the robot is commanded to pick a part from the bin;
a computing device programmed to use the one or more image to determine if the position/orientation of the part held by the gripper meets a predetermined criteria for the position/orientation of the part held by the gripper; and
the computing device further programmed to command the robot to move the gripper and return the picked part to the bin when it is determined that the part held by the gripper does not meet the predetermined criteria for the position/orientation of the part held by the gripper.
A robotic system for picking parts from a bin. The system has:
a robot having a gripper for picking a part from the bin when the robot is so commanded;
one or more cameras mounted to provide one or more images that show what is held by the gripper after the robot is commanded to pick a part from the bin;
a computing device programmed to use the one or more images to determine if the position/orientation of the part held by the gripper meets a predetermined criteria for the position/orientation of the part held by the gripper; and
the computing device further programmed to command the robot either by itself or together with the gripper or only the gripper to adjust the part's position/orientation to be within the predetermined criteria when it is determined that the part held by the gripper does not meet the predetermined criteria for the position/orientation of the part held by the gripper.
Referring now to
Robot systems such as system 10 typically include a manipulator assembly 12 and a computer-based controller 14. The robot manipulator assembly 12 includes an upper arm 16a and lower arm 16b. The manipulator assembly 12 has one end mounted through a first joint 18 to a base 20, and a wrist 22 on the opposite end. A grasping mechanism 24, also known as a gripper, is mounted to wrist 22 and is configured to receive a part. The grasping mechanism 24 and other devices such as a work-piece that are mounted to the robot wrist 22, are together known generally as an end-effector.
The vision system 36 is used to determine the part location and orientation in the bin 40. The images from cameras 38 have not been previously used for post pick analysis such as determining if the robot has picked more than one part 41 from the bin 40 or the orientation of the picked part 41. The vision system 36 shown in
Many types of errors can occur when using an industrial robot 12 and machine vision system 36 to automate bin picking. Because parts 41 are randomly distributed in a bin 40, parts could become interlocked with each other, entangled, and/or obstructed. When such interlocked, entangled and/or obstructed parts 41 are removed from the bin 40, several errors can occur. For example, a part 41 could be completely pulled out of the grasping mechanism (gripper) 24 during the extraction. Even if a part 41 is not pulled out of the gripper 24, its position and/or orientation could shift while it remains in the gripper 24. Such a shift can cause collision issues and other problems when the picked part is presented for the next robotic procedure, such as an incorrect placement of the gripped part 41 on a conveyer.
Another error that can be caused by interlocking and entangled parts 41 is the accidental picking of extra parts 41. This error can also cause problems for the next procedure that uses the picked part 41, because two or more parts 41 would be delivered to the station that handles the picked part. For example, if the next procedure is for the robot that has picked the part to place a single picked part 41 on a conveyer, the accidental placement of additional parts 41 on the conveyer would likely cause a jam to occur. The additional parts 41 can also be a safety concern since the robot's quick motions could cause a partially entangled part 41 to loosen and fly in a random direction in the robot cell, causing damage and possible human injury.
The system in previously described
The systems in
In the systems shown in
If a part 41 is not visibly in the gripper 24 after the extraction motion is complete, then another pick 41 would be attempted. This check that the picked part 41 is visibly in the gripper 24 can be done immediately after the extraction of the part 41 from bin 40 should have been completed, and repeated just before the part 41 is used for the next operation (such as placement on a conveyor). The performing of this check for a second time is to ensure that a loosely grasped part 41 was not accidentally dropped during the robot motions between the extraction of the part 41 from bin 40 and the start of the next operation.
The same vision software components used to locate parts 41 in the bin 40 or additional vision software components can also use the images from the cameras 38 to detect the presence of additional parts 41 attached to the gripper 24 and/or a part 41 that was intentionally picked by robot 12 from bin 40. If the robot 12 has picked additional parts 41 from bin 40, then the grasped parts 41 are moved over the bin 40 and then rotated or shaken over the bin 40 to thereby return the extra picked parts 41 to the bin 40. After this operation, the additional vision software components need to recheck how many parts 41 are in the gripper 24, and, if necessary, repeat either the pick procedure or the procedure to return the extra picked parts 41 to the bin 40. This rechecking of how many parts 41 are in gripper 24 is stopped if a predetermined drop attempt limit has been reached. This determination prevents the robot 12 from continuously repeating the foregoing operation. The predetermined drop attempt limit could, for example, be based either on the expiration of a predetermined period of time after the parts 41 were initially picked from bin or reaching a predetermined number of attempts to return the excess picked parts 41 to the bin 40.
Referring now to
At block 502, the vision system 36 finds a part 41 that can be picked from bin 40. The process proceeds to decision block 504 where it is asked if the vision system 36 has a found a part 41 that can be picked from bin 40. If the answer is no, a “cannot pick” signal is output to the controller 14 or other computing device. If the answer to the question in decision block 504 is yes, the process proceeds to block 506 where the gripping mechanism 24 is commanded to move to and grasp the part 41.
At block 508, the gripper 24 is retracted from bin 40. The gripper 24 should be grasping a part 41 and thus at decision block 510 it is asked if the vision system sees that the gripper 24 is holding at least one part 41. If the answer is no, this means that gripper 24 has not grasped a part 41 either because of a bad pick or that the picked part 41 was dropped by the gripper 24. Therefore the procedure returns back to block 502 to begin again the picking process.
If the answer to the question in decision block 510 is yes, then the gripper 24 is holding at least one part 41. The process then proceeds to decision block 512 where it is asked if the gripper 24 is either holding more than one part 41 or are other parts 41 attached to the gripped part 41. The answer to this question based on processing the image from the cameras 38 determines if the gripper 24 is holding only one part 41 or has gripped two or more parts 41. If the answer to the question in decision block 512 is no, then the gripper 24 is holding only one part 41 that has no other parts attached to the gripped part and the process proceeds to block 514 where the gripped part 41 proceeds to where it can be used in another operation. The process then returns from block 514 to block 502 to find another part 41 to pick from bin 40.
If the answer to the question in decision block 512 is yes, then the process proceeds to decision block 516 where it is asked if the predetermined drop attempt limit has been reached. This question is asked to prevent the robot 12 from continuously repeating the process to drop off extra part(s). The predetermined drop attempt limit could, for example, be timed based or based on the number of attempts to shake the parts free. If the answer to question in block 516 is no, the process proceeds to block 518 where the grasped parts are moved or rotated or shaken over the bin 40 so that the extra parts can be dropped back into bin 40. If the answer to the question of decision block 516 is yes, the process proceeds to block 520 where all of the gripped parts 41 are released back into bin 40.
While the flowchart of
In addition to the above, the position and/or orientation of a picked part 41 may shift in the gripper due to some obstructions during the extraction process, an imperfect grasp, or other errors. Cameras 38, mounted either on the robot shown in
Referring now to
At decision block 602, the procedure 600 asks if the position/orientation of the picked part 41 is within predetermined acceptable limits. This could be determined by comparing the difference between the current position and a pre-defined position of the picked part 41. The system will verify that this difference does not exceed predetermined acceptable limits. If the answer to the question asked in decision block 602 is no, then a pick error has occurred and the picked part 41 will be dropped back into the bin 40 and an appropriate error message can be displayed on the display terminal associated with the controller 14 or other computing device.
If the answer to the question asked in decision block 602 is yes, then the procedure 600 proceeds to decision block 604 where it is asked if it is required to change the final position/orientation of the robot 12. This can be determined by comparing the difference between the current position/orientation of the picked part 41 and a pre-defined position/orientation of the picked part 41. The system verifies that this difference is within a predefined range of a predetermined required position/orientation. If the answer to the question asked in decision block 604 is no, then the gripped part 41 proceeds to where it can be used in another operation. If the answer to this question is yes, then the procedure 600 proceeds to block 606 where the difference between the current and a pre-defined position/orientation of the picked part 41 is applied to cause the robot 12 either by itself or together with the gripper 24 or only by the gripper 24 to change the position/orientation of the picked part 41 so that the picked part 41 proceeds to the desired position/orientation for the next operation.
Some or all of the cameras 38 can be used for one or more of the vision-based aspects of the system, including: finding a part 41 in the bin 40, detecting if a part 41 was picked, detecting if extra parts 41 were picked, and/or detecting if a picked part's position/orientation has shifted significantly.
Part presence and part location can be implemented using industry standard vision libraries, or using commercially available vision applications. Examples of vision libraries include, but are not limited to, the libraries available from Matrox Graphics and Cognex. An example of a vision application, includes but is not limited to, ABB TrueView.
It is to be understood that the description of the foregoing exemplary embodiment(s) is (are) intended to be only illustrative, rather than exhaustive, of the present invention. Those of ordinary skill will be able to make certain additions, deletions, and/or modifications to the embodiment(s) of the disclosed subject matter without departing from the spirit of the invention or its scope, as defined by the appended claims.
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/US2010/046819 | 8/26/2010 | WO | 00 | 3/14/2012 |
| Number | Date | Country | |
|---|---|---|---|
| 61237420 | Aug 2009 | US |
