Patent Application
20080181485
1. Field
This disclosure generally relates to robotic systems, and more particularly to robotic vision systems that detect objects.
2. Description of the Related Art
There are many object recognition methods available for locating complex industrial parts having a large number of detectable features. A complex part with a large number of features provides redundancy, and thus can be reliably recognized even when some fraction of its features are not properly detected.
However, many parts that require a bin picking operation are simple parts which do not have a required level of redundancy in detected features. In addition, the features typically used for recognition, such as edges detected in captured images, are notoriously difficult to extract consistently from image to image when a large number of parts are jumbled together in a bid. The parts therefore cannot be readily located, especially given the potentially harsh nature of the environment, i.e., uncertain lighting conditions, varying amounts of occlusions, etc.
The problem of recognizing a simple part among many parts lying jumbled in a storage bin, such that a robot is able to grasp and manipulate the part in an industrial or other process, is quite different from the problem of recognizing a complex part having many detectable features. Robotic systems recognizing and locating three-dimensional (3D) objects, using either (a) two-dimensional (2D) data from a single image or (b) 3D data from stereo images or range scanners, are known. Single image methods can be subdivided into model-based and appearance-based approaches.
The model-based approaches suffer from difficulties in feature extraction under harsh lighting conditions, including significant shadowing and specularities. Furthermore, simple parts do not contain a large number of detectable features, which degrades the accuracy of a model-based fit to noisy image data.
The appearance-based approaches have no knowledge of the underlying 3D structure of the object, merely knowledge of 2D images of the object. These approaches have problems in segmenting out the object for recognition, have trouble with occlusions, and may not provide a 3D pose accurate enough for grasping purposes.
Approaches that use 3D data for recognition have somewhat different issues. Lighting effects cause problems for stereo reconstruction, and specularities can create spurious data both for stereo and laser range finders. Once the 3D data is generated, there are the issues of segmentation and representation. On the representation side, more complex models are often used than in the 2D case (e.g., superquadrics). These models contain a larger number of free parameters, which can be difficult to fit to noisy data.
Assuming that a part can be located, it must be picked up by the robot. The current standard for motion trajectories leading up to the grasping of an identified part is known as image based visual serving (IBVS). A key problem for IBVS is that image based servo systems control image error, but do not explicitly consider the physical camera trajectory. Image error results when image trajectories cross near the center of the visual field (i.e., requiring a large scale rotation of the camera). The conditioning of the image Jacobian results in a phenomenon known as camera retreat. Namely, the robot is also required to move the camera back and forth along the optical axis direction over a large distance, possibly exceeding the robot range of motion. Hybrid approaches decompose the robot motion into translational and rotational components either through identifying homeographic relationships between sets of images, which is computationally expensive, or through a simplified approach which separates out the optical axis motion. The more simplified hybrid approaches introduce a second key problem for visual serving, which is the need to keep features within the image plane as the robot moves.
Conventional bin picking systems are relatively deficient in at least one of the following: robustness, accuracy, and speed. Robustness is required since there may be no cost savings to the manufacturer if the error rate of correctly picking an object from a bin is not close to zero (as the picking station will still need to be manned). Location accuracy is necessary so that the grasping operation will not fail. And finally, solutions which take more than about 10 seconds between picks would slow down entire production lines, and would not be cost effective.
A system and method for identifying objects using a robotic system are disclosed. Briefly described, in one aspect, an embodiment may be summarized as a method that captures an image of at least one object with an image capture device that is moveable with respect to the object, processes the captured image to identify at least one feature of the at least one object, and determines a hypothesis based upon the identified feature. By hypothesis, we mean a correspondence hypothesis between (a) an image feature and (b) a feature from a 3D object model, that could have given rise to the image feature.
In another aspect, an embodiment may be summarized as a robotic system that identifies objects comprising an image capture device mounted for movement with respect to a plurality of objects to capture images and a processing system communicatively coupled to the image capture device. The processing system is operable to receive a plurality of images captured by the image captive device, identify at least one feature for at least two of the objects in the captured images, determine at least one hypothesis predicting a pose for the at least two objects based upon the identified feature, determine a confidence level for each of the hypotheses, and select the hypothesis with the greatest confidence level.
In another aspect, an embodiment may be summarized as a method that captures a first image of at least one object with an image capture device that is moveable with respect to the object; determines a first hypothesis based upon at least one feature identified in the first image, wherein the first hypothesis is predictive of a pose of the feature; captures a second image of the at least one object after a movement of the image capture device; determines a second hypothesis based upon the identified feature, wherein the second hypothesis is predictive of the pose of the feature; and compares the first hypothesis with the second hypothesis.
In another aspect, an embodiment may be summarized as a method that captures an image of a plurality of objects, processes the captured image to identify a feature associated with at least two of the objects visible in the captured image, determines a hypothesis for the at least two visible objects based upon the identified feature, determines a confidence level for each of the hypotheses for the at least two visible objects, and selects the hypotheses with the greatest confidence level.
In another aspect, an embodiment may be summarized as a method that captures a first image of at least one object with an image capture device that is moveable with respect to the object, determines a first pose of at least one feature of the object from the captured first image, determines a hypothesis that predicts a predicted pose of the feature based upon the determined first pose, captures a second image of the object, determines a second pose of the feature from the captured second image, and updates the hypothesis based upon the determined second pose.
In another aspect, an embodiment may be summarized as a method that captures a first image of at least one object with an image capture device that is moveable with respect to the object, determines a first view of at least one feature of the object from the captured first image, determines a first hypothesis based upon the first view that predicts a first possible orientation of the object, determines a second hypothesis based upon the first view that predicts a second possible orientation of the object, moves the image capture device, captures a second image of the object, determines a second view of the at least one feature of the object from the captured second image, determines an orientation of a second view of the at least one feature, and compares the orientation of the second view with the first possible orientation of the object and the second possible orientation of the object, in order to determine which orientation is the correct one.
In the drawings, identical reference numbers identify similar elements or acts. The sizes and relative positions of elements in the drawings are not necessarily drawn to scale. For example, the shapes of various elements and angles are not drawn to scale, and some of these elements are arbitrarily enlarged and positioned to improve drawing legibility. Further, the particular shapes of the elements as drawn, are not intended to convey any information regarding the actual shape of the particular elements, and have been solely selected for ease of recognition in the drawings.
In the following description, certain specific details are set forth in order to provide a thorough understanding of various embodiments. However, one skilled in the art will understand that the invention may be practiced without these details. In other instances, well-known structures associated with robotic systems have not been shown or described in detail to avoid unnecessarily obscuring descriptions of the embodiments.
Unless the context requires otherwise, throughout the specification and claims which follow, the word “comprise” and variations thereof, such as, “comprises” and “comprising” are to be construed in an open sense, that is as “including, but not limited to.”
The illustrated embodiment of the robot camera system 102 comprises an image capture device 114, a base 116, and a plurality of robot camera system members 118. A plurality of servomotors and other suitable actuators (not shown) of the robot camera system 102 are operable to move the various members 118. In some embodiments, base 116 may be moveable. Accordingly, the image capture device 114 may be positioned and/or oriented in any desirable pose to capture images of the pile of objects 112.
In the exemplary robot camera system 102, member 118a is configured to rotate about an axis perpendicular to base 116, as indicated by the directional arrows about member 118a. Member 118b is coupled to member 118a via joint 120a such that member 118b is rotatable about the joint 120a, as indicated by the directional arrows about joint 120a. Similarly, member 118c is coupled to member 118b via joint 120b to provide additional rotational movement. Member 118d is coupled to member 118c. Member 118c is illustrated for convenience as a telescoping type member that may be extended or retracted to adjust the pose of the image capture device 114.
Image capture device 114 is illustrated as physically coupled to member 118c. Accordingly, it is appreciated that the robot camera system 102 may provide a sufficient number of degrees of freedom of movement to the image capture device 114 such that the image capture device 114 may capture images of the pile of objects 112 from any pose (position and/or orientation) of interest. It is appreciated that the exemplary embodiment of the robot camera system 102 may be comprised of fewer, of more, and/or of different types of members such that any desirable range of rotational and/or translational movement of the image capture device 114 may be provided.
Robot tool system 104 comprises a base 122, an end effector 124, and a plurality of members 126. End effector 124 is illustrated for convenience as a grasping device operable to grasp a selected one of the objects from the pile of objects 112. Any suitable end effector device(s) may be automatically controlled by the robot tool system 104.
In the exemplary robot tool system 104, member 126a is configured to rotate about an axis perpendicular to base 122. Member 126b is coupled to member 126a via joint 128a such that member 126b is rotatable about the joint 128a. Similarly, member 126c is coupled to member 126b via joint 128b to provide additional rotational movement. Also, member 126c is illustrated for convenience as a telescoping type member that may extend or retract the end effector 124.
Pose of the various components of the robot camera system 100 described above is known. Control system 106 receives information from the various actuators indicating position and/or orientation of the members 118, 126. When the information is correlated with a reference coordinate system 130, control system 106 may computationally determine pose (position and orientation) of every member 118, 126 such that position and orientation of the image capture device 114 and the end effector 124 is determinable with respect to a reference coordinate system 130. Any suitable position and orientation determination methods and system may be used by the various embodiments. Further, the reference coordinate system 130 is illustrated for convenience as a Cartesian coordinate system using an x-axis, an y-axis, and a z-axis. Alternative embodiments may employ other reference systems.
Image capture device controller logic 214, residing in memory 204, is retrieved and executed by processor 202 to determine control instructions for the robot camera system 102 such that the image capture device 114 may be positioned and/or oriented in a desired pose to capture images of the pile of objects 112 (
Similarly, robot tool system controller logic 216, residing in memory 204, is retrieved and executed by processor 202 to determine control instructions for the robot tool system 104 such that the end effector 124 may be positioned and/or oriented in a desired pose to perform a work operation on an identified object in the pile of objects 112 (
The hypothesis determination logic 218 resides in memory 204. As described in greater detail hereinbelow, the various embodiments determine the pose (position and/or orientation) of an object using the hypothesis determination logic 218, which is retrieved from memory 204 and executed by processor 202. The hypothesis determination logic 218 contains at least instructions for processing a captured image, instructions for determining a hypothesis, instructions for hypothesis testing, instructions for determining a confidence level for a hypothesis, instructions for comparing the confidence level with a threshold(s), and instructions for determining pose of an object upon validation of a hypothesis. Other instructions may also be included in the hypothesis determination logic 218, depending upon the particular embodiment. By hypothesis, we mean a correspondence hypothesis between (a) an image feature and (b) a feature from a 3D object model, that could have given rise to the image feature.
Database 220 resides in memory 204. As described in greater detail hereinbelow, the various embodiments analyze captured image information to determine one or more features of interest on one or more of the objects in the pile of objects 112 (
Operation of an exemplary embodiment of the object identification system 100 will now be described in greater detail. Processor 202 determines control instructions for the robot camera system 102 such that the image capture device 114 is positioned and/or oriented to capture a first image of the pile of objects 112 (
The control system 106 processes a series of captured images of the two objects 302. Using a suitable edge detection algorithm or the like, the robot control system 106 determines a model for the geometry of at least one of the circular features 304. For convenience, this simplified example assumes that geometry models for both of the features 304 are determined since the feature 304 is visible on both objects 302.
From the determined geometry models of the ellipses (graphically illustrated as ellipses 308a, 308b in
In one embodiment, a hypothesis pertaining to at least object pose is then determined for a selected object. In other embodiments, a plurality of hypotheses are determined, one or more for each object having a visible feature of interest. A hypothesis is based in part upon a known model of the object, and more particularly, a model of its corresponding feature of interest. To determine a hypothesis, geometry of the feature of interest determined from a captured image is compared against known model geometries of the reference feature of the reference object to determine at least one predicted aspect of the object, such as the pose of the object. That is, a difference is determined between the identified feature and a reference feature of a known model of the object. Then, the hypothesis may be based at least in part upon the difference between the identified feature and the reference feature once the geometry of a feature is determined from a captured image. In another embodiment, the known model geometry is adjusted until a match is found between the determined feature geometry and the reference model. Then, the object's pose or orientation may be hypothesized for the object.
In other embodiments, the hypothesis pertaining to object pose is determined based upon detection of multiple features of interest. Thus, the first captured image or another image may be processed to identify a second feature of the object. Accordingly, the hypothesis is based at least in part upon the difference between the identified second feature and the second reference feature. In some embodiments, a plurality of hypotheses are determined based upon the plurality of features of interest.
In the above simplified example where the feature of interest for the objects 302a, 302b is circular, various perspective views of a circular feature are evaluated at various different geometries (positions and/or orientations) until a match is determined between the detected feature and the known model feature. In the above-described simplified example, the model geometry of a perspective view of a circular reference feature is compared with one or more of the determined geometries of the ellipses 308a, 308b. It is appreciated from
As noted above, pose of the image capture device 114 at the point where each image is captured is known with respect to coordinate system 130. Accordingly, the determined hypothesis can be further used to predict one or more possible poses for an object in space with reference to the coordinate system 130 (
Because object pose is typically indeterminable from the first captured image, another image is captured and analyzed. Changes in the detected feature of interest in the second captured image may be compared with the hypothesis to resolve the pose question described above. Accordingly, the image capture device 114 (
In selected embodiments, the image capture device 114 is dynamically moved in a direction and/or dynamically moved to a position as described herein. In other embodiments, the image capture device 114 is moved in a predetermined direction and/or to a predetermined position. In other embodiments, the objects are moved in a predetermined direction and/or to a predetermined position. Other embodiments may use a second image capture device and correlate captured images to resolve the above-described indeterminate object pose problem.
In yet other embodiments, the determined hypothesis is further used to determine a path of movement for the image capture device, illustrated by the directional arrow 136 (
In some situations, detected features of interest will become more or less discernable for the selected object in the next captured image. For example, in the event that the hypothesis correctly predicts pose of the object, the feature of interest may become more discernable in the second captured image if the image capture device is moved in a direction that is predicted to improve the view of the selected object. In such situations, the detected features of interest will be found in the second captured image where predicted by the hypothesis in the event that the hypothesis correctly predicts pose of the object. If the hypothesis is not valid, the detected feature of interest will be in a different place in the second captured image.
From the determined geometry models of the ellipses (graphically illustrated as ellipses 308a, 308b in
The initial hypothesis determined from the first captured image may be used to predict the expected geometry models of the ellipses (graphically illustrated as ellipses 314a, 314b in
In one exemplary embodiment, the identified feature in the second captured image is compared with the hypothesis to determine a first confidence level of the first hypothesis. If the first confidence level is at least equal to a threshold, the hypothesis may be validated. A confidence value may be determined which mathematically represents the comparison. Any suitable comparison process and/or type of confidence value may be used by various embodiments. For example, but not limited to, a determined orientation of the feature may be compared to a predicted orientation the feature based upon the hypothesis and the known movement of the image capture device relative to the object to compute a confidence value. Thus, a difference in actual orientation and predicted orientation could be compared with a threshold.
For example, returning to
Other confidence levels could be employed to invalidate a hypothesis. For example, a second threshold confidence value could require that a geometry of the area of the selected feature of interest in the captured image be less than a second threshold. If the geometry of the feature of interest in the captured image was outside the second threshold, then that hypothesis would be determined to be invalid.
It is appreciated that a variety of aspects of a feature of interest could be selected to determine a confidence level or value. Vector analysis is another non-limiting example, where the length and angle of the vector associated with a feature of interest on a captured image could be compared with a predicted length and angle of a vector based upon a hypothesis.
In some embodiments, the same feature on a plurality of objects may be used to determine a plurality of hypotheses for the feature of interest. The plurality of hypotheses are compared with corresponding reference model feature, and a confidence value or level is determined for each hypothesis. Then, one of the hypotheses having the highest confidence level and/or the highest confidence value could be selected to identify an object of interest for further analysis. The identified object may be the object that is targeted for picking from the bin 110 (
In an alternative embodiment, a hypothesis may be determined for the feature of interest in each captured image, where the hypothesis is predictive of object pose. The determined hypotheses between images may be compared to verify pose of the feature. That is, when the pose hypothesis matches between successively captured images, the object pose may then be determinable.
As noted above, movement of the image capture device 114 is known. In this simplified example, assume that the predicted geometry of the circular feature on a reference model is an ellipse that is expected to correspond to the illustrated ellipse 314a in
The process of moving the image capture device 114 incrementally along the determined path continues until the pose of one at least one of the objects is determinable. In various embodiments, the path of movement can be determined for each captured image based upon the detected features in that captured image. That is, the direction of movement of the image capture device 114, or a change in pose for the image capture device 114, may be dynamically determined. Also, the amount of movement may be the same for each incremental movement, or the amount of movement may vary between capture of subsequent images.
In another exemplary embodiment, a plurality of different possible hypotheses are determined for the visible feature of interest for at least one object. For example, a first hypothesis could be determined based upon a possible first orientation and/or position of the identified feature. A second hypothesis could be determined based upon a possible second orientation and/or position of the same identified feature.
Returning to
When the second captured image is analyzed, the two hypotheses are compared with a predicted pose (orientation and/or position) of the feature of interest. The hypotheses that fails to match or correspond to the view of detected feature would be eliminated.
Returning to
It is appreciated that the above-described approach of determining a plurality of possible hypotheses from the first captured image, and then eliminating hypotheses that are inconsistent with the feature of interest in subsequent captured images, may be advantageously used for objects having a feature of interest that could be initially characterized by many possible poses. Also, this process may be advantageous for an object having two or more different features of interest such that families of hypothesis are developed for the plurality of different features of interest.
At some point in the hypothesis elimination process for a given object, only one hypothesis (or family of hypotheses) will remain. The remaining hypothesis (or family of hypotheses) could be tested as described herein, and if validated, the object's position and/or pose could then be determined.
Furthermore, the above-described approach is applicable to a plurality of objects having different poses, such as the jumbled pile of objects 112 illustrated in
Such a plurality of hypotheses may be considered in the aggregate or totality, referred to as a signature. The signature may correspond to hypotheses developed for any number of characteristics or features of interest of the object. For example, insufficient information from one or more features of interest may not, by themselves, be sufficient to develop a hypothesis and/or predict pose of the object. However, when considered together, there may be sufficient information to develop a hypothesis and/or predict pose of the object.
For convenience of explaining operation of one exemplary embodiment, the above-described example (see
Accordingly, the detected features (whether true detection of a feature of interest or a false detection of other edges or artifacts) are analyzed to initially identify a plurality of most-likely detected features. If a sufficient number of features are not initially detected, subsequent images may be captured and processed after movement of the image capture device 114. Any suitable system or method of initially screening and/or parsing an initial group of detected edges into a plurality of most-likely detected features of interest may be used by the various embodiments. Accordingly, such systems and method are not described in detail herein for brevity.
Once the plurality of most-likely detected features of interest are initially identified, the image capture device 114 is moved and the subsequent image is captured. Because real-time processing of the image data is occurring, and because the incremental distance that the image capture device 114 is moved is relatively small, the embodiments may base subsequent edge detection calculations on the assumption that the motion of the plurality of most-likely detected features from image to image should be relatively small. Processing may be limited to the identified features of interest, and other features may be ignored. Accordingly, relatively fast and efficient edge detection algorithms may be used to determine changes in the plurality of identified features of interest.
In other embodiments, one detected feature of interest (corresponding to one of the objects in the pile of objects 112) is selected for further edge detection processing in subsequently captured images. That is, one of the objects may be selected for tracking in subsequently captured images. Selection of one object may be based on a variety of considerations. For example, one of the detected features may correlate well with the reference model and may be relatively “high” in its position (i.e., height off of the ground) relative to other detected features, thereby indicating that the object associated with the selected feature of interest is likely on the top of the pile of objects 112. Or, one of the detected features may have a relatively high confidence level with the reference model and may not be occluded by other detected features, thereby indicating that the object associated with the selected feature of interest is likely near the edge of the pile of objects 112. In other embodiments, a selected number of features of interest may be analyzed.
Once the second captured image has been analyzed to determine changes in view of the feature(s) of interest, the hypothesis may be validated. That is, a confidence level or value is determined based upon the hypothesis and the detected feature. The confidence level or value corresponds to a difference between the detected feature and a prediction of the detected feature (which is made with the model of the reference object based upon the current hypothesis). If the confidence level or value for the selected feature is equal to at least some threshold value, a determination is made that the pose of the object associated with the selected feature can be determined.
Returning to the simplified example described above (see
On the other hand, the confidence level or value may be less than the threshold, less than a second threshold, or less than the first threshold by some predetermined amount, such that a determination is made that the hypothesis is invalid. Accordingly, the invalid hypothesis may be rejected, discarded or the like. The process of capturing another first image and determining another first hypothesis would be restarted. Alternatively, if captured image data is stored in memory 204 (
Assuming that the current hypothesis is neither validated or invalidated, a series of subsequent images are captured. Edge detection is used to further track changes in the selected feature(s) of interest in the subsequently captured images. At some point, a correlation will be made between the determined feature of interest and the corresponding feature of interest of the reference object such that the hypothesis is verified or rejected. That is, at some point in the process of moving the image capture device 114 (or moving the objects), and capturing a series of images which are analyzed by the control system 106 (
Once the pose of the object is determined, control instructions may be determined such that the robot tool system 104 may be actuated to move the end effector 124 in proximity of the object such that the desired work may be performed on the object (such as grasping the object and removing it from the bin 110). On the other hand, at some point in the process of moving the image capture device 114 and capturing a series of images which are analyzed by the control system 106 (
At some point in the process of capturing a series of images after movement of the image capture device 114 (or movement of the objects), a second hypothesis may be determined by alternative embodiments. For example, one exemplary embodiment determines a new hypothesis for each newly captured image. The previous hypothesis is discarded. Thus, for each captured image, the new hypothesis may be used to determine a confidence level or value to test the validity of the new hypothesis.
In other embodiments, the previous hypothesis may be updated or revised based upon the newly determined hypothesis. Non-limiting examples of updating or revising the current hypothesis include combining the first hypothesis with a subsequent hypothesis. Alternatively, the first hypothesis could be discarded and replaced with a subsequent hypothesis. Other processes of updating or revising a hypothesis may be used. Accordingly, the updated or revised hypothesis may be used to determine another confidence level to test the validity of the updated or revised hypothesis.
Any suitable system or method of hypothesis testing may be used by the various embodiments. For example, the above-described process of comparing areas or characteristics of vectors associated with the captured image of the feature of interest could be used for hypothesis testing. Accordingly, such hypothesis testing systems and method are not described in detail herein for brevity.
Another simplified example of identifying an object and determining its pose is provided below.
Continuing with the second example,
In this simplified example of determining the pose of the lag screw 500a, it is assumed that the identified feature of interest 502a (graphically represented by the black bar) does not provide sufficient information to determine the pose of the lag screw 500a. That is, a hypothesis may be determined by comparing the feature of a reference model of a lag screw (the shank of a lag screw) with the determined feature 502a. However, because of the occlusion of a portion of the lag screw 500a by lag screw 500b, the length of the identified feature 502a will be less than the length of the feature in the absence of the occlusion. (On the other hand, an alternative hypothesis could assume that the lag screw 500a is at some angle in the captured image to account for the relatively short length of the identified feature 502a.)
In some embodiments the identified feature 502a and/or the other identified features 502b-e are used to determine movement of the image capture device 114 (
Here, the completely visible lag screw 500a will result in a determined feature 502a that substantially corresponds to the reference feature (the shank) of a reference model of a lag screw. Since the lag screw 500a is illustrated as laying in a slightly downward angle on the pile of lag screws 500a-e, the perspective view of the feature of the reference model will be adjusted to match up with the determined feature 502a. Accordingly, the pose of the lag screw 500a may be reliably and accurately determined. That is, given a hypothesis that the expected pose of a completely visible reference lag screw now reliably matches the determined feature 502a, the pose of the lag screw 500a is determinable.
The process illustrated in
The process illustrated in
The process illustrated in
The process illustrated in
The process illustrated in
In the above-described various embodiments, image capture device controller logic 214, hypothesis determination logic 218, and database 220 were described as residing in memory 204 of the control system 106. In alternative embodiments, the image capture device controller logic 214, hypothesis determination logic 218 and/or database 220 may reside in another suitable memory (not shown). Such memory may be remotely accessible by the control system 106. Or, the image capture device controller logic 214, hypothesis determination logic 218 and/or database 220 may reside in a memory of another processing system (not shown). Such a separate processing system may retrieve and execute the hypothesis determination logic 218 to determine and process hypotheses and other related operations, may retrieve and store information into the database 220, and/or may retrieve and execute the image capture device controller logic 214 to determine movement for the image capture device 114 and control the robot camera system 102.
In the above-described various embodiments, the image capture device 114 was mounted on a member 118c of the robot tool system 104. In alternative embodiments, the image capture device 114 may be mounted on the robot tool system 104 or mounted on a non-robotic system, such as a track system, chain/pulley system or other suitable system. In other embodiments, a moveable mirror or the like may be adjustable to provide different views for a fixed image capture device 114.
In the above-described various embodiments, a plurality of images are successively captured as the image capture device 114 is moved until the pose of an object is determined. The process may end upon validation of the above-described hypothesis. In an alternative embodiment, the process of successively capturing a plurality of images, and the associated analysis of the image data and determination of hypotheses, continues until a time period expires, referred to as a cycle time or the like. The cycle time limits the amount of time that an embodiment may search for an object of interest. In such situations, it is desirable to end the process, move the image capture device to the start position (or a different start position), and begin the process anew. That is, upon expiration of the cycle time, the process starts over or otherwise resets.
In other embodiments, if hypotheses for one or more objects of interest are determined and/or verified before expiration of the cycle time, the process of capturing images and analyzing captured image information continues so that other objects of interest are identified and/or their respective hypothesis determined. Then, after the current object of interest is engaged, the next object of interest has already been identified and/or its respective hypothesis determined before the start of the next cycle time. Or, the identified next object of interest may be directly engaged without the start of a new cycle time.
In yet other embodiments, if hypotheses for one or more objects of interest are determined and/or verified before expiration of the cycle time, a new starting position for the next cycle time for the image capture device 114 may be determined. In embodiments where the image capture device 114 is not physically attached to the device that engages the identified object of interest, the image capture device 114 may be moved to the determined position in advance of the next cycle time.
As noted above, in some situations, a hypothesis associated with an object of interest may be invalidated. Some embodiments determine at least one hypothesis for two or more objects using the same captured image(s). A “best” hypothesis is identified based upon having the highest confidence level or value. The “best” hypothesis is then selected for validation. As described above, motion of the image capture device 114 for the next captured image may be based on improving the view of the object associated with the selected hypothesis.
In the event that the hypothesis that was selected is invalidated, the process continues by selecting one of the remaining hypotheses that has not yet been invalidated. Accordingly, another hypothesis, such as the “next best” hypothesis that now has the highest confidence level or value, may be selected for further consideration. In other words, in the event that the current hypothesis under consideration is invalidated, another object and its associated hypothesis may be selected for validation. The above-described process of hypothesis validation is continued until the selected hypothesis is validated (or invalidated).
In such embodiments, additional images of the pile of objects 112 may be captured as needed until the “next best” hypothesis is validated. Then, pose of the object associated with the “next best” hypothesis may be determined. Furthermore, the movement of the image capture device 114 for capture of subsequent images may be determined based upon the “next best” hypothesis that is being evaluated. That is, the movement of the image capture device 114 may be dynamically adjusted to improve the view of the object in subsequent captured images.
In some embodiments, the feature on the object of interest is an artificial feature. The artificial feature may be painted on the object of interest or may be a decal or the like affixed to the object of interest. The artificial feature may include various types of information that assists in the determination of the hypothesis.
In the above-described various embodiments, the control system 106 (
The above description of illustrated embodiments, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Although specific embodiments of and examples are described herein for illustrative purposes, various equivalent modifications can be made without departing from the spirit and scope of the invention, as will be recognized by those skilled in the relevant art. The teachings provided herein can be applied to other object recognition systems, not necessarily the exemplary robotic system embodiments generally described above.
The foregoing detailed description has set forth various embodiments of the devices and/or processes via the use of block diagrams, schematics, and examples. Insofar as such block diagrams, schematics, and examples contain one or more functions and/or operations, it will be understood by those skilled in the art that each function and/or operation within such block diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. In one embodiment, the present subject matter may be implemented via Application Specific Integrated Circuits (ASICs). However, those skilled in the art will recognize that the embodiments disclosed herein, in whole or in part, can be equivalently implemented in standard integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more controllers (e.g., microcontrollers) as one or more programs running on one or more processors (e.g., microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of ordinary skill in the art in light of this disclosure.
In addition, those skilled in the art will appreciate that the control mechanisms taught herein are capable of being distributed as a program product in a variety of forms, and that an illustrative embodiment applies equally regardless of the particular type of signal bearing media used to actually carry out the distribution. Examples of signal bearing media include, but are not limited to, the following: recordable type media such as floppy disks, hard disk drives, CD ROMs, digital tape, and computer memory; and transmission type media such as digital and analog communication links using TDM or IP based communication links (e.g., packet links).
Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present systems and methods. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Further more, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
From the foregoing it will be appreciated that, although specific embodiments of the invention have been described herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the invention.
These and other changes can be made to the present systems and methods in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the invention to the specific embodiments disclosed in the specification and the claims, but should be construed to include all power systems and methods that read in accordance with the claims. Accordingly, the invention is not limited by the disclosure, but instead its scope is to be determined entirely by the following claims.
This application claims the benefit under 35 U.S.C. § 119(e) of U.S. provisional patent application Ser. No. 60/875,073, filed Dec. 15, 2006, the content of which is incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 60875073 | Dec 2006 | US |
