The invention generally relates to perception systems, and relates in particular to scanning systems for use in connection with robotic and other sortation systems that are intended to be used in dynamic environments requiring the robotic or other sortation system to accommodate processing a variety of types of objects.
For example many order fulfillment operations achieve high efficiency by employing a process called wave picking. In wave picking, orders are picked from warehouse shelves and placed at locations (e.g., into bins) containing multiple orders that are sorted downstream. At the sorting stage individual articles are identified, and multi-article orders are consolidated, for example into a single bin or shelf location, so that they may be packed and then shipped to customers. The process of sorting these articles has traditionally been done by hand. A human sorter picks an article from an incoming bin, finds a barcode on the object, scans the barcode with a handheld barcode scanner, determines from the scanned barcode the appropriate bin or shelf location for the article, and then places the article in the so-determined bin or shelf location where all articles for that order have been defined to belong. Automated systems for order fulfillment have also been proposed. See for example, U.S. Patent Application Publication No. 2014/0244026, which discloses the use of a robotic arm together with an arcuate structure that is movable to within reach of the robotic arm.
Other ways of identifying items by code scanning either require manual processing, or require that the code location be controlled or constrained so that a fixed or robot-held code scanner (e.g., barcode scanner) can reliably detect it. Manually operated barcode scanners are generally either fixed or handheld systems. With fixed systems, such as those used at point-of-sale systems, the operator holds the article and places it in front of the scanner so that the barcode faces the scanning device's sensors, and the scanner, which scans continuously and decodes any barcodes that it can detect. If the article is not immediately detected, the person holding the article typically needs to vary the position or rotation of the object in front of the fixed scanner, so as to make the barcode more visible to the scanner. For handheld systems, the person operating the scanner looks for the barcode on the article, and then holds the scanner so that the article's barcode is visible to the scanner, and then presses a button on the handheld scanner to initiate a scan of the barcode.
Automatic barcode scanners are similarly either fixed or hand-held systems, and the same principles apply. In the case of barcode scanners typically used in industrial applications, the possible positions of barcodes must be tightly controlled so that they are visible to the one or more scanners. For example, one or more barcode scanners may be placed in fixed locations relative to a conveyor so that they can scan items, typically boxes, as they pass by scanners. See, for example, U.S. Pat. No. 5,495,097. In these installations the range of placement of the barcodes is comparatively limited as the barcodes are on labels affixed to one of four sides or top or bottom (e.g., if upside down) of a box, which can be presented using simple mechanical means, at orientations optimal for scanning.
In all of these cases, the systems employ sensors, cameras or laser reflectivity sensors, as well as software to detect barcodes and decode them. These methods have inherent limitations that include the range of distances of orientations relative to the detection system, over which they are able to reliably scan barcodes. Firstly, the barcode must be facing the scanner; secondly the range to the barcode must be such that individual elements can be reliably distinguished; and, thirdly, the tilt and skew of the barcode must be such that individual elements can be reliably distinguished. The types of sensors employed, and the robustness of the software detection and decoding schemes determine these performance parameters.
There remains a need, therefore, for an object identification system for robotic and other sortation systems that is able to accommodate the automated identification and processing of a variety of objects in a variety of orientations.
In accordance with an embodiment, the invention provides a drop perception system that includes an open housing structure having an internal volume, an open top and an open bottom, and a plurality of perception units positioned to capture perception data within the internal volume at a plurality of locations between the open top and the open bottom of the open housing.
In accordance with another embodiment, the invention provides a perception system for assisting in identifying an object, the perception system including a plurality of perception units that are each positioned to be directed toward different portions of an object path that an object takes as the object travels through the perception system without assistance by any mechanical conveyance system in contact with the object.
In accordance with a further embodiment, the invention provides a drop perception system for identifying an object, the drop perception system including a plurality of perception units that are each positioned to be directed toward different portions of an object path that an object may take as the object falls through the drop perception system, and each perception unit is engageable to provide perception data regarding the object.
In accordance with a further embodiment, the invention provides a method of sorting objects. The method includes the steps of dropping an object into a perception system that includes a plurality of perception units that are each positioned to be directed toward different portions of a path that the object may take as the object falls through the perception unit, and engaging the perception units to capture perception data associated with the object.
The following description may be further understood with reference to the accompanying drawings in which:
The drawings are shown for illustrative purposes only.
In accordance with an embodiment, the invention provides a novel object perception system for the purposes of automatically sorting individual objects in a set. In applications such as order fulfillment, articles or goods are collected into heterogeneous sets and need to be sorted. Individual objects need to be identified and then routed to object-specific locations. The described systems reliably automate the identification of such objects by employing automated scanners. The scanners look for a variety of codes such as indicia (e.g., barcodes, radio frequency tags, Stock Keeping Unit (SKU), Universal Product Code (UPC), Digimarc DWCode, etc.).
Operating in conjunction with a robotic pick and place system, systems in accordance with various embodiments of the invention automate part of the sorting process, in particular the step of identifying picked objects. Instead of a person picking the object from a bin for example, a robotic arm picks an article from a bin. The object is passed in front of a plurality of barcode scanners, and then, having obtained identification codes for the object, the object is then routed to the appropriate bin or shelf location. Since barcode scanners employ cameras or lasers to scan 1D or 2D symbologies printed on labels affixed to objects, the barcodes must be visible to the scanner's sensors for successful scanning in order to automatically identifying items in a heterogeneous stream of arbitrary objects, as in a jumbled set of objects found in a bin.
Whereas fixed industrial scanners require that the object's barcode be situated so that its barcode is visible to a scanner, the robotic arm of the present invention may pick an object out of a heterogeneous collection of objects where the barcode is not visible and drop the object into a perception system of the present invention. In other embodiments, the system may provide that objects are dropped into the perception system by providing a feed conveyor positioned above the perception system, and providing that objects are singulated on the conveyor. The result is an automated barcode scanning system for arbitrary objects in a heterogeneous stream of objects that can be used to accurately and reliably identify the objects.
Sorting for order fulfillment is one application for automatically identifying objects from a heterogeneous object stream. Barcode scanners have a wide variety of further uses including identifying the stock keeping unit of an article, or tracking parcels. The described systems may have many uses in the automatic identification and sortation of objects.
In accordance with various embodiments, therefore, the invention provides a method for determining the identity of an object from a collection of objects, as well as a method for scanning the barcode of an object employing one or more scanners and a sortation system for differently processing different objects. The invention further provides a method for determining the placement of fixed barcode scanners so as to maximize the probability of successfully scanning an object selected by a robot end-effector in accordance with certain embodiments, as well as a method for determining whether multiple objects are dropped into the scanner at the same time.
An important aspect is the ability to identify via barcode or other visual markings of objects by employing a perception system into which objects may be dropped. Automated scanning systems would be unable to see barcodes on objects that are presented in a way that their barcodes are not exposed or visible. As shown in
The perception system may be used in certain embodiments, with a robotic system that may include a robotic arm equipped with sensors and computing, that when combined is assumed herein to exhibit the following capabilities: (a) it is able to pick objects up from a specified class of objects, and separate them from a stream of heterogeneous objects, whether they are jumbled in a bin, or are singulated on a motorized or gravity conveyor system; (b) it is able to move the object to arbitrary places within its workspace; (c) it is able to place objects in an outgoing bin or shelf location in its workspace; and, (d) it is able to generate a map of objects that it is able to pick, represented as a candidate set of grasp points in the workcell, and as a list of polytopes enclosing the object in space.
The allowable objects are determined by the capabilities of the robotic system. Their size, weight and geometry are assumed to be such that the robotic system is able to pick, move and place them. These may be any kind of ordered goods, packages, parcels, or other articles that benefit from automated sorting. Each object is associated with a stock keeping unit (SKU), which identifies the item.
The manner in which inbound objects arrive may be for example, in one of two configurations: (a) inbound objects arrive piled in bins of heterogeneous objects; or (b) inbound articles arrive by a moving conveyor. The collection of objects includes some that have exposed bar codes and other objects that do not have exposed bar codes. The robotic system is assumed to be able to pick items from the bin or conveyor. The stream of inbound objects is the sequence of objects as they are unloaded either from the bin or the conveyor.
The manner in which outbound objects are organized is such that objects are placed in a bin, shelf location or cubby, into which all objects corresponding to a given order are consolidated. These outbound destinations may be arranged in vertical arrays, horizontal arrays, grids, or some other regular or irregular manner, but which arrangement is known to the system. The robotic pick and place system is assumed to be able to place objects into all of the outbound destinations, and the correct outbound destination is determined from the SKU of the object.
It is assumed that the objects are marked in one or more places on their exterior with a visually distinctive mark such as a barcode or radio-frequency identification (RFID) tag so that they may be identified with a scanner. The type of marking depends on the type of scanning system used, but may include 1D or 2D barcode symbologies. Multiple symbologies or labeling approaches may be employed. The types of scanners employed are assumed to be compatible with the marking approach. The marking, either by barcode, RFID tag, or other means, encodes a symbol string, which is typically a string of letters and numbers. The symbol string uniquely associates the object with a SKU.
The operations of the systems described above are coordinated by the central control system 20. This system determines from symbol strings the SKU associated with an object, as well as the outbound destination for the object. The central control system is comprised of one or more workstations or central processing units (CPUs). The correspondence between SKUs and outbound destinations is maintained by the central control system in a database called a manifest. The central control system maintains the manifest by communicating with a warehouse management system (WMS).
During operation, the broad flow of work may be generally as follows. First, the system is equipped with a manifest that provides the outbound destination for each inbound object. Next, the system waits for inbound objects to arrive either in a bin or on a conveyor. The robotic system may pick one item at a time from the input bin, and may drop each item into the perception system discussed above. If the perception system successfully recognizes a marking on the object, then the object is then identified and forwarded to a sorting station or other processing station. If the object is not identified, the robotic system may either replace the object back onto the input conveyor and try again, or the conveyor may divert the object to a human sortation bin to be reviewed by a human.
The sequence of locations and orientations of the perception units are chosen so as to minimize the average or maximum amount of time that scanning takes. Again, if the object cannot be identified, the object may be transferred to a special outbound destination for unidentified objects, or it may be returned to the inbound stream. This entire procedure operates in a loop until all of the objects in the inbound set are depleted. The objects in the inbound stream are automatically identified, sorted, and routed to outbound destinations.
In accordance with an embodiment therefore, the invention provides a system for sorting objects that arrive inbound bins and that need to be placed into a shelf of outbound bins, where sorting is to be based on a unique identifier symbol. Key specializations in this embodiment are the specific design of the perception system so as to maximize the probability of a successful scan, while simultaneously minimizing the average scan time. The probability of a successful scan and the average scan time make up key performance characteristics. These key performance characteristics are determined by the configuration and properties of the perception system, as well as the object set and how they are marked.
The two key performance characteristics may be optimized for a given item set and method of barcode labeling. Parameters of the optimization for a barcode system include how many barcode scanners, where and in what orientation to place them, and what sensor resolutions and fields of view for the scanners to use. Optimization can be done through trial and error, or by simulation with models of the object.
Optimization through simulation employs a barcode scanner performance model. A barcode scanner performance model is the range of positions, orientations and barcode element size that a barcode symbol can be detected and decoded by the barcode scanner, where the barcode element size is the size of the smallest feature on the barcode. These are typically rated at a minimum and maximum range, a maximum skew angle, a maximum pitch angle, and a minimum and maximum tilt angle.
Typical performance for camera-based barcode scanners are that they are able to detect barcode symbols within some range of distances as long as both pitch and skew of the plane of the symbol are within the range of plus or minus 45 degrees, while the tilt of the symbol can be arbitrary (between 0 and 360 degrees). The barcode scanner performance model predicts whether a given barcode symbol in a given position and orientation will be detected.
The barcode scanner performance model is coupled with a model of where barcodes would expect to be positioned and oriented. A barcode symbol pose model is the range of all positions and orientations, in other words poses, in which a barcode symbol will expect to be found. For the scanner, the barcode symbol pose model is itself a combination of an article gripping model, which predicts how objects will be held by the robotic system, as well as a barcode-item appearance model, which describes the possible placements of the barcode symbol on the object. For the scanner, the barcode symbol pose model is itself a combination of the barcode-item appearance model, as well as an inbound-object pose model, which models the distribution of poses over which inbound articles are presented to the scanner. These models may be constructed empirically, modeled using an analytical model, or approximate models may be employed using simple sphere models for objects and a uniform distributions over the sphere as a barcode-item appearance model.
The LEDs and cameras therefore encircle the inside of the structure 32, and the cameras are positioned to view the interior via windows that may include a glass or plastic covering (e.g., 44). The structure 32 may be suspended by loop hooks 46 or placed over an opening and hang by brackets 48.
With further reference to
The above process may be repeated any number, m, of times (e.g., 50) (step 1036). After all m repeats have finished, the system confirms that the item has exited the scanner (step 1038). The system then determines whether any codes were found (step 1040), and if not reports and error that no codes were found (step 1046). If a codes was found, the system collects all codes found (step 1042) and determines whether all codes match each other (step 1044). If not, the system reports that more than one item was placed in the scanner (step 1050). If all codes found match each other, the system determines whether more than one item was placed in the scanner (step 1048) by determining whether too much space exists between regions of an item. If so, the system reports that more than one item was placed in the scanner (step 1050). If not, the system reports the identification of the code that was found (step 1052) and engages a sorting path associated with the code that was found (step 1054). If no code was found (step 1046) or if the system detects that more than item was in the scanner (step 1050), the system may ask whether the operator wishes to try the scan again (or may be programmed to do so) (step 1056). If yes, then the system returns the item(s) to the input stream ahead of the drop scanner (step 1058). If not, the system moves the item(s) to a manual sort location for sorting by a person (step 1060).
As discussed above, the output of the processor provides a signal indicative of the identified code of the item in the scanner, and based on this, a sortation system may immediately take action consistent with having the item routed in the desired direction or processing path. For example,
The scanner 32 is coupled to the processor 62 as discussed above, and an output sortation control signal 63 is provided to a sortation system, such as for example, a controller 96 of a conveyor 94 that provides direction routing of items (e.g., 92) to any of a plurality of bins, containers or locations 98, 104, for example by moving in either direction as indicated at C. Items 100 and 102, for example had been routed to location 98, and item 106 had been routed to location 104.
The system may also include an interrupting system that interrupts the falling of an object through the perception system. The interrupting system may be useful, for example, where the item to be scanned is a plastic bag (either opaque or transparent), and particularly, where the identifier code (such as a barcode) is not visible or readily visible by the perception units, for example if the bag is folded and obscures the barcode. With reference to
In other embodiments, the system may include an interrupting element that urges lighter items upward in a reverse direction for a short time. With reference to
Those skilled in the art will appreciate that numerous modifications and variations may be made to the above disclosed embodiments without departing from the spirit and scope of the present invention.
The present application is a continuation of U.S. patent application Ser. No. 16/407,965, filed May 9, 2019; which is a continuation of U.S. patent application Ser. No. 15/901,656, filed Feb. 21, 2018; which is a continuation of U.S. patent application Ser. No. 15/228,692, filed Aug. 4, 2016, now U.S. Pat. No. 9,937,532, issued Apr. 10, 2018; which claims priority to U.S. Provisional Patent Application Ser. No. 62/269,640 filed Dec. 18, 2015, the disclosures of which are hereby incorporated by reference in their entireties.
Number | Name | Date | Kind |
---|---|---|---|
3734286 | Simjian | May 1973 | A |
3864566 | Simpson et al. | Feb 1975 | A |
4186836 | Wassmer et al. | Feb 1980 | A |
4704694 | Czerniejewski | Nov 1987 | A |
4722653 | Williams et al. | Feb 1988 | A |
4759439 | Hartlepp | Jul 1988 | A |
4819784 | Sticht | Apr 1989 | A |
4846335 | Hartlepp | Jul 1989 | A |
4895242 | Michel | Jan 1990 | A |
5190162 | Hartlepp | Mar 1993 | A |
5495097 | Katz | Feb 1996 | A |
5648709 | Maeda | Jul 1997 | A |
5713473 | Satake | Feb 1998 | A |
5742420 | Peng | Apr 1998 | A |
5794788 | Massen | Aug 1998 | A |
5794789 | Payson et al. | Aug 1998 | A |
5839566 | Bonnet | Nov 1998 | A |
5875434 | Matsuoka et al. | Feb 1999 | A |
5996316 | Kirschner | Dec 1999 | A |
6059092 | Jerue et al. | May 2000 | A |
6060677 | Ulrichsen et al. | May 2000 | A |
6076023 | Sato | Jun 2000 | A |
6079570 | Oppliger et al. | Jun 2000 | A |
6087608 | Schlichter | Jul 2000 | A |
6124560 | Roos et al. | Sep 2000 | A |
6208908 | Boyd et al. | Mar 2001 | B1 |
6246023 | Kugle | Jun 2001 | B1 |
6323452 | Bonnet | Nov 2001 | B1 |
6390756 | Isaacs et al. | May 2002 | B1 |
6401936 | Isaacs et al. | Jun 2002 | B1 |
6505093 | Thatcher et al. | Jan 2003 | B1 |
6579053 | Grams et al. | Jun 2003 | B1 |
6685031 | Takizawa | Feb 2004 | B2 |
6688459 | Bonham et al. | Feb 2004 | B1 |
6705528 | Good | Mar 2004 | B2 |
6721444 | Gu et al. | Apr 2004 | B1 |
6762382 | Danelski | Jul 2004 | B1 |
6946612 | Morikawa | Sep 2005 | B2 |
7313464 | Perreault et al. | Dec 2007 | B1 |
7347376 | Biss et al. | Mar 2008 | B1 |
7474939 | Oda et al. | Jan 2009 | B2 |
7516848 | Shakes et al. | Apr 2009 | B1 |
8662314 | Jones et al. | Mar 2014 | B2 |
8718814 | Clark et al. | May 2014 | B1 |
8776694 | Rosenwinkel et al. | Jul 2014 | B2 |
8811722 | Perez Cortes et al. | Aug 2014 | B2 |
8823801 | Jacobson | Sep 2014 | B2 |
8874270 | Ando | Oct 2014 | B2 |
8972049 | Tidhar et al. | Mar 2015 | B2 |
9102053 | Suzuki | Aug 2015 | B2 |
9102055 | Konolige et al. | Aug 2015 | B1 |
9102336 | Rosenwinkel | Aug 2015 | B2 |
9120622 | Elazary et al. | Sep 2015 | B1 |
9227323 | Konolige et al. | Jan 2016 | B1 |
9256775 | Yasunaga | Feb 2016 | B1 |
9259844 | Xu et al. | Feb 2016 | B2 |
9266237 | Nomura | Feb 2016 | B2 |
9283680 | Yasuda et al. | Mar 2016 | B2 |
9364865 | Kim | Jun 2016 | B2 |
9381645 | Yarlagadda et al. | Jul 2016 | B1 |
9481518 | Neiser | Nov 2016 | B2 |
9486926 | Kawano | Nov 2016 | B2 |
9492923 | Wellman et al. | Nov 2016 | B2 |
9604363 | Ban | Mar 2017 | B2 |
9751693 | Battles et al. | Sep 2017 | B1 |
9821464 | Stiernagle et al. | Nov 2017 | B2 |
9878349 | Crest et al. | Jan 2018 | B2 |
9926138 | Brazeau et al. | Mar 2018 | B1 |
9937532 | Wagner et al. | Apr 2018 | B2 |
9962743 | Bombaugh et al. | May 2018 | B2 |
9975148 | Zhu et al. | May 2018 | B2 |
10007827 | Wagner | Jun 2018 | B2 |
10029865 | McCalib, Jr. et al. | Jul 2018 | B1 |
10058896 | Hicham et al. | Aug 2018 | B2 |
10127514 | Napoli | Nov 2018 | B2 |
10737299 | Wagner | Aug 2020 | B2 |
11046530 | Koga | Jun 2021 | B2 |
20010056313 | Osborne, Jr. | Dec 2001 | A1 |
20020092801 | Dominguez | Jul 2002 | A1 |
20020147568 | Wenzel et al. | Oct 2002 | A1 |
20020169698 | Chien | Nov 2002 | A1 |
20020179502 | Cerutti et al. | Dec 2002 | A1 |
20030029946 | Lieber et al. | Feb 2003 | A1 |
20030034281 | Kumar | Feb 2003 | A1 |
20030038065 | Pippin et al. | Feb 2003 | A1 |
20040261366 | Gillet et al. | Dec 2004 | A1 |
20050002772 | Stone | Jan 2005 | A1 |
20050268579 | Natterer | Dec 2005 | A1 |
20060022824 | Olsen, III et al. | Feb 2006 | A1 |
20060045672 | Maynard et al. | Mar 2006 | A1 |
20060070929 | Fry et al. | Apr 2006 | A1 |
20060182543 | Schaefer | Aug 2006 | A1 |
20060190356 | Nemet | Aug 2006 | A1 |
20070043468 | Schaefer et al. | Feb 2007 | A1 |
20070209976 | Worth et al. | Sep 2007 | A1 |
20080046116 | Khan et al. | Feb 2008 | A1 |
20080181485 | Beis et al. | Jul 2008 | A1 |
20080181753 | Bastian et al. | Jul 2008 | A1 |
20100125361 | Mougin et al. | May 2010 | A1 |
20100260380 | Kaeser et al. | Oct 2010 | A1 |
20100292841 | Wickham | Nov 2010 | A1 |
20100318216 | Faivre et al. | Dec 2010 | A1 |
20110144798 | Freudelsperger | Jun 2011 | A1 |
20110184555 | Kosuge et al. | Jul 2011 | A1 |
20110238207 | Bastian, II et al. | Sep 2011 | A1 |
20110243707 | Dumas et al. | Oct 2011 | A1 |
20110320036 | Freudelsperger | Dec 2011 | A1 |
20120118699 | Buchmann et al. | May 2012 | A1 |
20120177465 | Koholka | Jul 2012 | A1 |
20120219397 | Baker | Aug 2012 | A1 |
20130051696 | Garrett | Feb 2013 | A1 |
20130051969 | Takeuchi et al. | Feb 2013 | A1 |
20130202195 | Perez Cortes | Aug 2013 | A1 |
20130235372 | Voss | Sep 2013 | A1 |
20130245824 | Barajas et al. | Sep 2013 | A1 |
20140067127 | Gotou | Mar 2014 | A1 |
20140105719 | Mueller et al. | Apr 2014 | A1 |
20140166549 | Ito et al. | Jun 2014 | A1 |
20140305847 | Kudrus | Oct 2014 | A1 |
20140360924 | Smith | Dec 2014 | A1 |
20150057793 | Kawano | Feb 2015 | A1 |
20150073589 | Khodl et al. | Mar 2015 | A1 |
20150081090 | Dong | Mar 2015 | A1 |
20150081091 | Blomquist et al. | Mar 2015 | A1 |
20150217937 | Marquez | Aug 2015 | A1 |
20150224650 | Xu et al. | Aug 2015 | A1 |
20150283586 | Dante et al. | Oct 2015 | A1 |
20150306634 | Maeda et al. | Oct 2015 | A1 |
20150352721 | Wicks et al. | Dec 2015 | A1 |
20150375398 | Penn et al. | Dec 2015 | A1 |
20150378345 | Winkler | Dec 2015 | A1 |
20160136816 | Pistorino | May 2016 | A1 |
20160221762 | Schroader | Aug 2016 | A1 |
20160228921 | Doublet et al. | Aug 2016 | A1 |
20160243704 | Vakanski et al. | Aug 2016 | A1 |
20160244262 | O'Brien et al. | Aug 2016 | A1 |
20160379076 | Nobuoka et al. | Dec 2016 | A1 |
20170024896 | Houghton et al. | Jan 2017 | A1 |
20170043953 | Battles et al. | Feb 2017 | A1 |
20170050315 | Henry et al. | Feb 2017 | A1 |
20170066597 | Hiroi | Mar 2017 | A1 |
20170080566 | Stubbs et al. | Mar 2017 | A1 |
20170087731 | Wagner et al. | Mar 2017 | A1 |
20170128986 | Sterkel | May 2017 | A1 |
20170136632 | Wagner et al. | May 2017 | A1 |
20170137232 | Messner | May 2017 | A1 |
20170157648 | Wagner et al. | Jun 2017 | A1 |
20170157649 | Wagner | Jun 2017 | A1 |
20170173638 | Wagner et al. | Jun 2017 | A1 |
20170225330 | Wagner et al. | Aug 2017 | A1 |
20180075406 | Kingston et al. | Mar 2018 | A1 |
20180085788 | Engel et al. | Mar 2018 | A1 |
20180127219 | Wagner et al. | May 2018 | A1 |
20180148272 | Wagner et al. | May 2018 | A1 |
20180186572 | Issing | Jul 2018 | A1 |
20180265291 | Wagner et al. | Sep 2018 | A1 |
20180265298 | Wagner et al. | Sep 2018 | A1 |
20180265311 | Wagner et al. | Sep 2018 | A1 |
20180273295 | Wagner et al. | Sep 2018 | A1 |
20180273296 | Wagner et al. | Sep 2018 | A1 |
20180273297 | Wagner et al. | Sep 2018 | A1 |
20180273298 | Wagner et al. | Sep 2018 | A1 |
20180282065 | Wagner et al. | Oct 2018 | A1 |
20180282066 | Wagner et al. | Oct 2018 | A1 |
20180312336 | Wagner et al. | Nov 2018 | A1 |
20180327198 | Wagner et al. | Nov 2018 | A1 |
20180330134 | Wagner et al. | Nov 2018 | A1 |
20180333749 | Wagner et al. | Nov 2018 | A1 |
20190022702 | Vegh et al. | Jan 2019 | A1 |
20190047786 | Suzuki | Feb 2019 | A1 |
20190102712 | Duca | Apr 2019 | A1 |
20190329979 | Wicks et al. | Oct 2019 | A1 |
20210061563 | Ueda et al. | Mar 2021 | A1 |
Number | Date | Country |
---|---|---|
2006204622 | Mar 2007 | AU |
1033604 | Jul 1989 | CN |
1671489 | Sep 2005 | CN |
101482879 | Jul 2009 | CN |
101971221 | Feb 2011 | CN |
102363354 | Feb 2012 | CN |
102430530 | May 2012 | CN |
102621155 | Aug 2012 | CN |
202539084 | Nov 2012 | CN |
103129783 | Jun 2013 | CN |
103942518 | Jul 2014 | CN |
104093650 | Oct 2014 | CN |
104137051 | Nov 2014 | CN |
108700869 | Oct 2018 | CN |
3919865 | Dec 1990 | DE |
19510392 | Sep 1996 | DE |
102004001181 | Aug 2005 | DE |
102004013353 | Oct 2005 | DE |
102005061309 | Jul 2007 | DE |
102007023909 | Nov 2008 | DE |
102007038834 | Feb 2009 | DE |
102010002317 | Aug 2011 | DE |
102012102333 | Sep 2013 | DE |
102014111396 | Feb 2016 | DE |
0235488 | Sep 1987 | EP |
837415 | Apr 1998 | EP |
1995192 | Nov 2008 | EP |
2053350 | Apr 2009 | EP |
2511653 | Oct 2012 | EP |
2823899 | Jan 2015 | EP |
2084531 | Apr 1982 | GB |
2356383 | May 2001 | GB |
2507707 | May 2014 | GB |
S54131278 | Oct 1979 | JP |
S63310406 | Dec 1988 | JP |
H0395001 | Apr 1991 | JP |
H05324662 | Dec 1993 | JP |
H08157016 | Jun 1996 | JP |
2002175543 | Jun 2002 | JP |
2003150230 | May 2003 | JP |
2007182286 | Jul 2007 | JP |
2008037567 | Feb 2008 | JP |
101413393 | Jun 2014 | KR |
2005022076 | Mar 2005 | WO |
WO-2005022076 | Mar 2005 | WO |
2007009136 | Jan 2007 | WO |
2010034044 | Apr 2010 | WO |
2010099873 | Sep 2010 | WO |
2011038442 | Apr 2011 | WO |
2014166650 | Oct 2014 | WO |
2015118171 | Aug 2015 | WO |
2015162390 | Oct 2015 | WO |
2017036780 | Mar 2017 | WO |
Entry |
---|
Notice on the Second Office and the Second Office Action, along with its English translation, issued by the China National Intellectual Property Administration in related Chinese Patent Application No. 201680081764.5 dated Apr. 13, 2021, 14 pages. |
Examiner's Report issued by the Canadian Intellectual Property Office dated Dec. 7, 2020 in related Canadian Patent Application No. 2,998,544, 4 pages. |
Examiner's Report issued by the Innovation, Science and Economic Development Canada in related Canadian Patent Application No. 3,009,102 dated Jan. 12, 2021, 3 pages. |
International Preliminary Report on Patentability issued by the International Bureau of WIPO in related International Application No. PCT/US2016/050949 dated Mar. 13, 2018, 10 pages. |
International Search Report and Written Opinion issued by the International Searching Authority in related International Application No. PCT/US2016/050949 dated Dec. 8, 2016, 13 pages. |
International Search Report and Written Opinion issued by the International Searching Authority in related International Application No. PCT/US2016/066786 dated Mar. 20, 2017, 14 pages. |
International Preliminary Report on Patentability issued by the International Bureau of WIPO in related International Application No. PCT/US2016/066786 dated Jun. 19, 2018, 11 pages. |
Non-Final Office Action issued by the U.S. Patent and Trademark Office dated May 22, 2017 in related U.S. Appl. No. 15/228,692, 13 pages. |
Bohg, Jeannette, et al., “Data-Driven Grasp Synthesis—A Survey,” Transactions on Robotics, pp. 289-309, Apr. 14, 2016. |
Rembold, Derk et al., “Object Turning for Barcode Search,” Proceedings of the 2000 1EEE/RSK—Int'l Conf. on Intelligent Robots and Systems, p. 1267, Oct. 31, 2000. |
Cipolla, Roberto et al., “Visually Guided Grasping in Unstructured Environments,” Journal of Robotics and Autonomous Systems, pp. 337-346, Mar. 3, 2001. |
Klingbeil, Ellen et al., “Grasping with Application to an Autonomous Checkout Robot,” Proceedings—IEEE Int'l Conf. on Robotics and Automation, pp. 2837-2844, May 9, 2011. |
Non-Final Office Action issued by the U.S. Patent and Trademark Office dated Jun. 18, 2019 in related U.S. Appl. No. 15/982,238, 27 pages. |
Final Office Action issued by the U.S. Patent and Trademark Office dated Nov. 7, 2019 in related U.S. Appl. No. 15/982,238, 15 pages. |
First Examiner's Report issued by the Canadian Intellectual Property Office dated Jan. 21, 2019 in related Canadian Patent Application No. 2,998,544, 4 pages. |
Examiner's Report issued by the Canadian Intellectual Property Office on Jan. 6, 2020 in related Canadian Patent Application No. 2,998,544, 4 pages. |
Communication pursuant to Rules 161(1) and 162 EPC issued by the European Patent Office dated Apr. 18, 2018, in related European Patent Application No. 16778496.6, 3 pages. |
Communication pursuant to Rules 161(1) and 162 EPC issued by the European Patent Office dated Aug. 9, 2018, in related European Patent Application No. 16826518.9, 3 pages. |
First Examiner's Report issued by the Canadian Intellectual Property Office dated Apr. 15, 2019 in related Canadian Patent Application No. 3,009,102, 3 pages. |
Wikipedia, “Machine Vision,” Wikipedia.org, Mar. 1, 2017 (https://en.wikipedia.org/w/index.php?title=Machine_vision&oldid=768036938). |
Wikipedia, “Automatic Identification and Data Capture,” Wikipedia.org, Mar. 10, 2017 (https://en.wikipedia.org/w/index.php?title=Automatic-idenification_and_data_capture&oldid=0769563714). |
Non-Final Office Action issued by the U.S. Patent and Trademark Office dated Apr. 25, 2017 in related U.S. Appl. No. 15/260,837, 28 pages. |
Final Office Action issued by the U.S. Patent and Trademark Office dated Oct. 11, 2017 in related U.S. Appl. No. 15/260,837, 32 pages. |
Non-Final Office Action issued by the U.S. Patent and Trademark Office dated Mar. 21, 2019 in related U.S. Appl. No. 15/901,656, 10 pages. |
Non-Final Office Action issued by the U.S. Patent and Trademark Office dated Oct. 22, 2019 in related U.S. Appl. No. 15/901,656, 13 pages. |
Examiner's Report issued by the Canadian Intellectual Property Office dated Mar. 6, 2020 in related Canadian Patent Application No. 3,009,102, 3 pages. |
Examiner's Report issued by the Innovation, Science and Economic Development Canada (Canadian Intellectual Property Office) dated Nov. 5, 2021 in related Canadian Patent Application No. 2,998,544, 4 pages. |
Notice on the Third Office and the Third Office Action, along with its English translation, issued by the China National Intellectual Property Administration in related Chinese Patent Application No. 201680081764.5 dated Oct. 20, 2021, 12 pages. |
First Office Action, and its English translation, issued by the China National Intellectual Property Administration dated May 6, 2020 in related Chinese Patent Application No. 201680065881.2, 23 pages. |
First Office Action, and its English translation, issued by the China National Intellectual Property Administration dated Jun. 24, 2020 in related Chinese Patent Application No. 201680081764.5, 18 pages. |
Notice on the Second Office Action, and the Second Office Action, issued by the China National Intellectual Property Administration in related Chinese Patent Application No. 201680065881.2 dated Jan. 29, 2021, 7 pages. |
Communication pursuant to Article 94(3) EPC issued by the European Patent Office in related European Patent Application No. 16778496.6 dated Feb. 24, 2021, 6 pages. |
Communication pursuant to Article 94(3) EPC issued by the European Patent Office in related European Patent Application No. 16826518.9 dated Mar. 4, 2021, 10 pages. |
Examiner's Report issued by the Innovation, Science and Economic Development Canada (Canadian Intellectual Property Office) in related Canadian Patent Application No. 3,009,102 dated Dec. 31, 2021, 4 pages. |
Non-Final Office Action issued by the United States Patent and Trademark Office in related U.S. Appl. No. 16/800,587 dated Feb. 11, 2022, 27 pages. |
Notice on Grant of Patent Right for Invention and Search Report, along with its English translation, issued by the China National Intellectual Property Administration in related Chinese Patent Application No. 201680081764.5 dated Mar. 3, 2022, 8 pages. |
Communication pursuant to Article 94(3) EPC issued by the European Patent Office in related European Patent Application No. 16826518.9 dated Mar. 22, 2022, 8 pages. |
Number | Date | Country | |
---|---|---|---|
20200368785 A1 | Nov 2020 | US |
Number | Date | Country | |
---|---|---|---|
62269640 | Dec 2015 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 16407965 | May 2019 | US |
Child | 16900158 | US | |
Parent | 15901656 | Feb 2018 | US |
Child | 16407965 | US | |
Parent | 15228692 | Aug 2016 | US |
Child | 15901656 | US |