Patent Application
20230069565
Fulfillment centers require systems that enable the efficient storage, retrieval, picking, sorting, packing and shipment of a large number of items of diverse product types. Inventory is typically stored and organized within the fulfilment center within a storage structure. Once an order is placed, a warehouse worker or robot is tasked with retrieving the ordered items from the storage structure.
To minimize the number of trips and the total distance that the warehouse worker or robot must travel to retrieve items from the storage structure for a given number of orders, the warehouse worker or robot often retrieves items pertaining to multiple orders in a single trip. Retrieving items in this manner necessitates that the items be brought to a picking station where they are picked into individual order containers which are subsequently shipped to the consumer.
A traditional picking station includes a monitor that displays pick and place instructions received from Warehouse Software (WS). The pick and place instructions may direct an operator to pick and place an item of a particular type into an order container designated for a specific customer. Manually picking and placing each of the retrieved items is labor-intensive and expensive. While it is understood that replacing human workers with pick and place robots would lower operating costs, pick and place robots are often less efficient than human workers in performing pick and place tasks. For example, after reading the pick and place instructions displayed on the monitor, a human worker will be able to ascertain the item type and the quantity of that item type that he or she has been instructed to pick and will immediately and accurately be able to grasp the desired item and place it in the appropriate order container.
Conventional pick and place robots, on the other hand, are less efficient than humans at quickly and effectively grasping a wide variety of items ranging in size, shape, weight, material and stiffness. Contemporary pick and place robots have begun to utilize end effectors which “grasp” an item via a suction force. It has been found that these suction-based end effectors can successfully grasp a greater range of inventory items.
Despite the recent improvements that have been made to pick and place robots, various drawbacks remain. For example, when the item is stored within a flexible membrane, such as a polybag (as is common within the clothing industry), the suction force can draw the polybag of multiple items into the end effector of the robot, thereby causing the robot to unintentionally grasp two or more items. In another example, an item packaged in a small box may be on top of another item packaged in a larger box. When the item packaged in the larger box is grasped by the robot, the smaller box may be carried on the larger box. In yet another example, the packaging of an item may have exposed adhesive that may cause another item to stick to the exposed adhesive. As such, the item stuck to the adhesive may be carried along when the packaging is grasped by the robot.
If the unintentionally grasped items were subsequently packed into an order container, a customer would receive multiple items even though he or she had only paid for a single item. The unintentional picking of two or more items is referred to herein as “doubles picking,” “picking doubles,” “a double pick” and the like. It will be appreciated that doubles picking, when compounded, would result in large monetary losses to the retailer.
To avoid these losses, pick and place robots perform additional steps to ensure that only a single item has been picked. In one example, a robot may utilize a sensor that measures the weight of a picked item which the pick and place robot can then compare to an expected weight of the item to determine if it has grasped a single item, or more than one item, prior to placing the picked item(s) in the order container. This additional step, however, requires that the retailer keep and update a large database encompassing the weight of each product type stored within the fulfillment center. This is a cumbersome task. Moreover, the additional weighing step reduces the efficiency of the pick and place robot.
Aspects of the technology are directed to assisting robots in completing tasks through use of a teleoperator system. For example, one embodiment of the technology is directed to a method for training a system to generate pick instructions. The method may include receiving, by a teleoperator system, data corresponding to a robot attempting a picking task including picking an item of an identified product type from a container, the data including imagery of an end effector of the robot after the attempted picking task; displaying, by the teleoperator system, the imagery on a display; receiving, by the teleoperator system, an input indicating whether the picking task was successfully or unsuccessfully performed by the robot; labeling, by the teleoperator system, at least a portion of the data based on the received input; and transmitting, by the teleoperator system, the labeled data to a processor for training a learning algorithm for use in generating future pick instructions.
Another embodiment of the technology is directed to a system including a robot having an end effector; and a teleoperator system in communication with the robot. The teleoperator system may be configured to: receive data corresponding to a robot attempting a picking task including picking an item of an identified product type from a container, the data including imagery of an end effector of the robot after the attempted picking task; display the imagery on a display; receive an input indicating whether the picking task was successfully or unsuccessfully performed by the robot; label at least a portion of the data based on the received input; and transmit the labeled data to a processor for training a learning algorithm for use in generating future pick instructions.
In some embodiments the item is packaged in a flexible membrane.
In some embodiments the end effector of the robot is a suction-based end effector.
In some embodiments, the data further includes a grasping location corresponding to a location on a packaging of the item where the robot engaged the end effector during the attempted picking task.
In some embodiments, the picking task is performed in response to pick and place instructions transmitted to the robot, the pick and place instructions comprising the pick instructions and place instructions. In some examples, the teleoperator system instructs the robot to complete the transmitted place instructions after receiving an input indicating the picking task successful.
In some examples, the picking task is successful when the robot picks only an item of the identified product type.
In some examples, the picking task is unsuccessfully performed when the robot picks two or more items.
In some instances, the teleoperator system instructs the robot to return the two or more items to the container and further instructs the robot to perform an additional picking task.
In some examples, the additional picking task is the same task as the picking task, and the teleoperator system provides a grasping location to the robot, the grasping location corresponding to a location on a packaging of the item where the end effector of the robot will attempt to engage during performance of the additional picking task.
In some embodiments, the data further includes a set of requirements of the picking task, identification of product types in the container, location of an item within the container relative to the container and/or relative to another item within the container.
Various embodiments of the present disclosure are described herein with reference to the drawings. The figures depict embodiments of the present disclosure for purposes of illustration only. Alternative embodiments of the structures and methods illustrated herein may be implemented without departing from the principles or benefits of the disclosure as described herein.
The technology disclosed herein relates to assisting robots in completing tasks. For example, a robot may autonomously attempt to complete a task, such as a pick and place task, to move an item from a first location to a second location, such as from a picking container to an order container. A teleoperator may monitor the progress of the robot as it progresses through the pick and place task to determine whether the robot picked only the intended item. In the event the robot inadvertently picks one or more unintended items, the teleoperator may intervene and instruct the robot to return the picked item(s) to the picking container or call for onsite support. As is further described herein, information associated with the failed picking attempts, along with information associated with successful picking attempts, may be used to train machine learning algorithms to generate more accurate pick and place instructions for the robot to perform in the future, thereby significantly reducing, if not eliminating, double picking.
As used herein, the term “container” encompasses bins, totes, cartons, boxes, bags, auto-baggers, conveyors, sorters, and other such places an item could be picked from or placed. To distinguish between containers from which items are to be picked and containers in which picked items are to be placed, the term “picking container” will be used to identify containers from where items are to be picked, and the term “order container” will be used to identify containers in which items are to be placed. Also, as used herein, the terms “substantially,” “generally,” and “about” are intended to mean that slight deviations from absolute are included within the scope of the term so modified.
Although only one robot control system 101, one robot 111, one teleoperator system 121, one onsite operator system 131, and one storage device 150 are shown in
Some embodiments of system 100 may have different components than those described herein. For instance, some embodiments of the system 100 may include only teleoperator system 121 but not onsite operator system 131, or onsite operator system 131 but not teleoperator system 121. Similarly, some or all of the functions of some of the systems and storage devices may be distributed among the other systems. For example, the functions of teleoperator system 121 may be performed by onsite operator system 131. In another example, the functions of robot control system 101 may be performed by robot 111, teleoperator system 121, and/or onsite operator system 131.
Robot control system 101 includes one or more processors 102, memory 103, one or more input devices 106, one or more network devices 107, and one or more neural networks 108. The processor 102 may be a commercially available central processing unit (“CPU”), a System on a Chip (“SOC”), an application specific integrated circuit (“ASIC”), a microprocessor, microcontroller, or other such hardware-based processors. In some instances, robot control system 101 may include multiple processor types.
Memory, such as memory 103, may be configured to read, write, and store instructions 104 and data 105. Memory 103 may be any solid state or other such non-transitory type memory device. For example memory 103 may include one or more of a hard drive, a solid-state hard drive, NAND memory, flash memory, ROM, EEPROM, RAM, DVD, Blu-ray, CD-ROM, write-capable, and read-only memories, or any other device capable of storing data. Data 105 may be retrieved, manipulated, and/or stored by the processor 102 in memory 103.
Data 105 may include data objects and/or programs, or other such instructions, executable by processor 102. Data objects may include data received from one or more components, such as other systems, processor 102, input device 106, network device 107, storage device 150, etc. The programs can be any computer or machine code capable of being executed by a processor, such as processor 102, including the vision and object detection algorithms described herein. The instructions 104 can be stored in any format for processing by a processor or in any other computing device language including scripts or modules. The functions, methods, routines, etc., of the programs for vision and object detection algorithms are explained in more detail herein. As used herein, the terms “instructions,” “applications,” “steps,” “routines” and “programs” may be used interchangeably.
The robot control system 101 may include at least one network device 107. The network device 107 may be configured to communicatively couple robot control system 101 with the other systems, such as teleoperator system 121, robot 111, onsite operator system 131, and storage device 150 via the network 160. In this regard, the network device 107 may be configured to enable the robot control system 101 to communicate and receive data, such as data received from robot 111, and other such signals to other computing devices, such as teleoperator system 121 and onsite operator system 131, or storage device 150. The network device 107 may include a network interface card (NIC), Wi-Fi card, Bluetooth receiver/transmitter, or other such device capable of communicating data over a network via one or more communication protocols, such as point-to-point communication (e.g., direct communication between two devices), Ethernet, Wi-Fi, HTTP, Bluetooth, LTE, 3G, 4G, Edge, etc., and various combinations of the foregoing.
Robot control system 101 may include one or more input devices 106 for interacting with the robot control system, robot 111, or other systems, such as teleoperator system 121 and onsite operator system 131. Input devices 106 may include components normally used in connection with a computing device such as keyboards, mice, touch screen displays, monitors, controllers, joysticks, and the like.
Robot control system 101 may exchange data 105 via an internal bus (not shown), network device 107, direct connections, or other such connections. In this regard, data 105 may be exchanged between the memory 103, storage device 150, processor 102, input device 106, and other systems, such as robot 111, teleoperator system 121, and onsite operator system 131.
Network 160 may include interconnected protocols and systems. The network 160 described herein can be implemented using various protocols and systems, such that the network can be part of the Internet, World Wide Web, specific intranets, wide area networks, or local networks. The network can utilize standard communications protocols, such as Ethernet, Wi-Fi and HTTP, proprietary protocols, and various combinations of the foregoing.
In some instances, robot control system 101 may be connected to or include one or more data storage devices, such as storage device 150. Storage device 150 may be one or more of a hard drive, a solid-state hard drive, NAND memory, ROM, RAM, DVD, CD-ROM, write-capable, and read-only memories, or any other device capable of storing data. The storage device 150 may store data, including programs and data objects such as vision and object detection algorithms. Moreover, storage device 150 may log data, such as information related to the performance of the robots, completed tasks, assistance request histories, etc. Although
References to a processor, computer, or robot will be understood to include references to a collection of processors, computers, or robots that may or may not operate in parallel and/or in coordination. Furthermore, although the components of robot control system 101 are shown as being within the same block in
Each robot 111 may include one or more processors 112, memory 113 storing instructions 114 and data 115, input devices 116, network devices 117, sensors 118, mechanical devices 119, and neural networks 188. The processors 112, memory 113, input devices 116, network devices 117, and neural networks 188 may be similar to processors 102, memory 103, input devices 106, network devices 107, and neural networks 108 of the robot control system 101. The mechanical devices 119 may include mechanical components of the robot, such as wheels, picking arm, and end effectors, etc. As used herein, the terms “neural networks,” and “machine learning algorithms” will be understood to capture both the singular and plural. For instance, “neural networks” may include one neural network or a plurality of neural networks (e.g., two or more neural networks). Similarly, “machine learning algorithms” may include one machine learning algorithm or a plurality of machine learning algorithms.
As used herein, sensors 118 may include one or more image/video capture cards, cameras, including red-green-blue (RGB) or RGB-depth (D) cameras, video recorders, Light Detection and Ranging (LIDAR), sonar radar, accelerometers, depth sensors, etc. Such sensors may capture data in the environment surrounding the robot and/or information about the robot itself. The sensors 118 may be mounted to or within the robot 111. Such sensors 118 may also be spaced apart from the hardware of robot 111. In some instances, sensors spaced art from the hardware of robot 111 may be placed in the vicinity of the robot. As used herein, the terms “image” and “images” may include a single image, multiple images, videos, video stills, and/or multiple video stills.
Robot 111 may be a stationary or mobile, manipulator robot (sometimes referred to herein as a “manipulator robot” or “robot”), designed to perform pick and place tasks within a warehouse or fulfillment center (hereinafter simply “warehouse”).
Robot 211 may also include a camera 218, or other sensor, arranged to capture the environment of the picking arm 234 and end effector 242 as the robot performs a task such as a pick and place task. In this regard, the camera 218 may capture imagery of items as they are grasped by picking arm 234. The camera 218 may provide the imagery to robot control system 101 and/or directly to teleoperator system 121. Although
Robot 111 may operate in one of two modes: an autonomous mode, by executing autonomous control instructions, or a teleoperated mode, in which the control instructions are manually piloted (e.g., directly controlled) by a teleoperator, such as a remote teleoperator (e.g., a teleoperator located outside of the warehouse 10) or an onsite teleoperator (e.g., a teleoperator located within the warehouse 10). While the term “control instructions,” whether autonomous or piloted, is primarily described herein as instructions for assisting robot 111 in performing a pick and place task and, more specifically, determining if the robot correctly picked an item (e.g., did not inadvertently perform a doubles pick), it will be appreciated that the term “control instructions” may additionally refer to a variety of other robotic tasks such the recognition of an inventory item, the swapping of one end effector for another end effector, inventory counting, edge case identification, or any other robotic task that facilitates manipulation of objects or the environment in performing order fulfillment tasks, manufacturing tasks, assembly tasks, or other tasks. In one embodiment, robot 111 may be a machine learning robot capable of executing autonomous or piloted control instructions.
Robot 111 may send and/or receive processor readable data or processor executable instructions via communication channels, such as via network 160, to robot control system 101. In this manner, robot control system 101 can predict grasping poses (e.g., position and/or orientation and/or posture of the robotic picking arm) and send control instructions to robot 111 to execute the predicted grasping pose to autonomously grasp the item. Although robot 111 and robot control system 101 are illustrated as separate devices in
System 100 may further include one or more operator devices including a teleoperator system 121 and/or an onsite operator system 131. Teleoperator system 121 may be positioned within the warehouse in which the robots 111 are located or external to the warehouse, whereas onsite operator system 131 is positioned in the warehouse, such as in the vicinity of the robots 111. Each operator system, including teleoperator system 121 and onsite operator system 131 may include one or more processors 122,132; memory 123,133 storing instructions 124, 134 and data 125,135; network devices 127,137; and sensors 128,138; which may be similar to processors 102, memory 103, and network devices 107 of the robot control system 101, respectively. Sensors 128,138 may be similar to sensors 118 of robot 111.
Teleoperator system 121 and onsite operator system 131 may be personal computers, tablets, smart phones, wearable computers, or other such computing devices. Each of the operator systems may also include one or more input devices 126, 136 to capture control instructions from an operator. The one or more user input devices 126, 136 may be, for example, keyboards, mice, touch screen displays, displays, controllers, buttons, joysticks and the like.
A teleoperator may input synchronous (real-time) or asynchronous (scheduled or queued) control instructions to the teleoperator system 121. The control instructions may be, for example, click point control instructions, 3d mouse control instructions, click and drag control instructions, keyboard or arrow key control instructions, text or verbal instructions, action primitive instructions, etc. In some instances, sensors 128 may function as input devices, such as by capturing hand or body control instructions.
Each of the operator systems may also include one or more output devices 129, 139. Output devices may include displays, head mounted displays, such as smart glasses, speakers, and/or haptic feedback controllers (e.g., vibration element, piezo-electric actuator, rumble, kinesthetic, rumble motor). In some embodiments, the input and output devices of the operator systems may be the same device. For instance, the input and output devices of the teleoperator system 121 may be a touchscreen.
Teleoperator system 121 may be utilized by teleoperators, to monitor, control, and or assist the operation of robots 111. In this regard, teleoperators may visualize sensor data, such as imagery within images provided by robot sensors 118. As noted above, these images may include one or more individual images, videos, and/or stills from videos. The images may contain imagery of the robot and/or its environment, such as picking containers and order containers, as well as the items contained therein. These images may be replayed and/or viewed in real time. When robot 111 performs a task, an operator may utilize teleoperator system 121 or onsite operator system 131 to determine whether the task was performed successfully, thereby avoiding the need for the robot 111 or robot control system 101 to perform subsequent steps to determine if the task was successfully and correctly performed. By providing teleoperation capabilities via the teleoperator system 121, the neural networks 108 of robot control system 101 and/or neural networks 188 of robot 111 can be trained to generate improved control instructions (e.g., control instructions that result in more accurate picks and less double picking). Furthermore, if robot 111 is unsuccessful at autonomously performing the task, operators can utilize teleoperator system 121 to send control instructions to robot 111 to assist the robot in grasping an item and/or packing the item within the order container, for example, in a specific location of the order container or in a specific orientation. Teleoperator system 121 can thus be utilized by a teleoperator to assist robot 111 in properly performing certain edge case scenarios.
Other edge case scenarios may not be able to be corrected via teleoperation and/or cannot be corrected in an efficient manner via teleoperation. In these situations, onsite operator system 131 may provide an onsite operator with a notification that their assistance is required to handle these edge cases. As described herein, these notifications may be provided by teleoperator system 121, robot control system 101, and/or robot 111. By providing onsite operators with assistance notifications via onsite operator system 131, efficient handling of these edge cases is possible. For instance, onsite operator system 131 may receive a notification when maintenance of robot 111 is required, for example, when the system detects a failure in one of the hardware components of the robot such as the picking arm, the wheel, or the camera. In another example, onsite operator system 131 may receive a notification when an object is blocking the path that a robot needs to traverse to complete a task and that the object needs to be cleared. In yet another example, onsite operator system 131 may receive a notification that a robot needs assistance with inventory that has fallen outside the reach of the robot. In some embodiments, like teleoperator system 121, onsite operator system 131 may be used to determine whether robot 111 correctly performed a pick and/or to assist robot 111 in performing a task such as a pick and place task.
In addition to the operations described above and illustrated in the figures, various operations will now be described. The following operations do not have to be performed in the precise order described below. Rather, various steps can be handled in a different order or simultaneously, and steps may also be added or omitted.
As shown in step 403, robot 111 attempts to perform the picking portion, or the “picking task,” of the received pick and place instructions. The picking task may include picking an item from a picking container.
Upon completion of the picking task, the robot 111 may request approval from a teleoperator before proceeding with the remainder of the pick and place task, as shown in step 405. In this regard, the robot 111 or robot control system 101 may send a request or notification to a teleoperator system, such as teleoperator system 121, to confirm that the pick was successful. The request or notification may include an image feed that includes imagery captured by a sensor 118, such as a camera. The image feed may include imagery of the end effector of the robot and/or the item(s) picked by the end effector.
Upon receiving the request or notification through teleoperator system 121, the teleoperator may review the image feed to determine whether the correct item and only the correct item was picked by robot 111. In the event, the teleoperator determines the pick was successful (e.g., only the correct item was picked), the teleoperator may confirm that the pick task was successfully performed and instruct robot 111, using teleoperator system 121, to complete the pick and place task, as further illustrated by step 407 in
As illustrated by step 411 in
The robot control system 101, or other such computing devices, may process the collected data to train machine learning algorithms, such as the neural networks 108 shown in
Some or all of the items, including items 511-519, may be stored within flexible or rigid packaging, such as polybags, cardboard boxes, plastic wrap, etc. Alternatively, some items may not be packaged or partially packaged. In this example, each of the items 511-519 are of the same product type and are individually stored within a polybag. Picking container 503 may contain several layers of densely packed and overlapping items. Nevertheless, for clarity, picking container 503 is illustrated as only including a few items, and specifically items 511-519, with item 513 partially overlying item 511. It will be appreciated, however, that the packaging of some items may be folded underneath itself or may be located underneath the packaging of other items, and, for this reason, some of the items 511-519 appear in different sizes. In other examples, the items may be of different product types.
Per step 401 of the flow diagram of
To generate the pick and place instructions, the neural networks of robot control system 101 or robot 211 (e.g., neural network 108 or 188) may predict which item robot 211 has the highest likelihood of grasping and removing from picking container 503. In making such a determination, neural networks 108 or 188 may predict one or more grasping locations (e.g., areas of locations on the packing of an item accessible to the end effector 242 of robot 211) and/or one or more grasping pose candidates (e.g., position and/or orientation of the end effector 242) that it believes will lead to the end effector 242 of robot 211 successfully picking the item from container 503.
For example, neural network 108 of robot control system 101 or neural network 188 of robot 211 may determine that items 511 and 513 have the best chance of being picked based, in part, on the packing of items 515-519 overlying each other and/or the items' locations relative to sidewalls and/or partitions (not shown) of container 513. The neural networks 108 or 188 may predict that items 511 can be successfully picked if the item is grasped at grasping location 590A and that item 513 can be successfully picked if the item is grasped at grasping location 590B. The neural networks 108 or 188 may also predict one or more grasping pose candidates that may be sequentially performed as the end effector 242 of robot 211 approaches and engages items 511 or 513 at the predicted grasping locations.
The robot control system 101 and/or robot 211 may then implement a policy, which utilizes one or more metrics, checks, and filters to select one of the predicted grasping locations and/or one or more of the predicted grasping pose candidates for the robot 211 to execute. For example, based on the policy, the robot control system 101 or robot 211 may select grasping location 590B after determining that the probability of grasping and removing item 513 from container 503 by engaging the end effector 242 at grasping location 590B is higher than grasping and removing item 511 from the container by engaging the end effector at grasping location 590A. Robot control system 101 or robot 211 may then also select one or more of the predicted pose candidates that may be run in sequence to instruct the end effector 242 during the approach to grasping location 590B. The robot control system 101 or robot 211 may then generate pick and place instructions for the robot 211 to execute. The pick instructions may include the selected grasping location and the selected poses. The place instructions may include instructions on which order container to place the picked item.
As illustrated in step 403 of the flow diagram of
As illustrated in
As is further shown in
After performing the picking task, the robot 211 may request approval from a teleoperator before proceeding with the remainder of the pick and place instructions, as shown in step 405 of
The request may also include information associated with the requirements of the picking task or the entirety of the pick and place task. This information may include identification of the product type to be picked and the order container in which the products are to be placed.
After receiving the request or notification through teleoperator system 121 (or onsite operator system 131), a teleoperator may review the images and other such data included in the request to determine whether only the correct item was picked or whether a doubles pick occurred. Continuing the example illustrated in
Although the remedial measure is described above as returning the picked item(s) to the picking container, in some instances the remedial measures may include sending a message to warehouse system or robot controller so that other measures may be taken. Alternatively, or simultaneously, the remedial measure may include instructing the robot to continue with the next task, such as a placing the picked items into an order container. In another example, the remedial measure may include instructing the robot to attempt to fix the grasp on the picked items(s), such as by moving the grasping location such that only the intended items is picked and the other item or items is released.
In some instances, robot 211 or robot control system 101 may request that the teleoperator assist the robot in performing the picking task. In other instances, the teleoperator may input picking instructions to robot 211 before robot 211 or robot control system 101 requests assistance if, for example, the teleoperator determines that piloted instructions may be more efficient. In this regard, and as is further illustrated in
Although four end effectors are shown in the end effector selection section 510 of interface 501, the interface may display any number of end effectors, such as one, two, three, five, six, etc. The collection of end effectors shown on interface 501 may change based on the number of end effectors actually available to the robot 211 that is under control of teleoperator system 121. For instance, if a robot has two available end effectors, interface 501 may display only the two available end effectors. A teleoperator may select any of the end effectors presented in the interface for the robot to use to perform the task. In some examples, the interface may prevent or recommend to the teleoperator certain end effectors based on the capabilities of the end effectors to complete certain tasks. For instance, if robot control system 101 determines that a particular end effector is unlikely to successfully perform a certain picking task currently assigned to robot 211 based, for example, on the size, weight, material, or other properties of the product type or packaging of the item in which the robot is tasked with picking, the interface 501 may not display that particular end effector, or the interface may “gray out” the icon associated with that particular end effector. In another example, if a particular end effector is determined to be well suited for a particular task, interface 501 may provide a visual indication indicating such. For instance, the icon corresponding to the well-suited end effector may be highlighted or bolded. In some examples, the well-suited end effector may automatically be selected. Further, interface 501 may provide a visual indicator to illustrate which end effector is currently in use or otherwise attached to the robot 211.
The teleoperator may select a grasping location based on the imagery displayed within the interface of the teleoperator system 121, to assist robot 211 in completing the picking task. The selected grasping location may be provided to teleoperator system 121 via an input device 126, such as a mouse, keyboard, touchscreen, buttons, etc.
Irrespective of whether the picking task was autonomously performed, or performed with assistance from a teleoperator, as is shown in step 405 of
In accordance with step 411 in
The data and/or the additional data may be stored internally within robot 111, by robot control system 101, teleoperator system 121, onsite operator system 131, and/or storage device 150. The data corresponding to an unsuccessful picking task may be labeled as such. Similarly, data corresponding to a successful picking task, including picking tasks where teleoperator system 121 piloted instructions, may be labeled accordingly.
The robot control system 101 or other such computing devices may use the collected data to train a machine learning algorithm, such as neural networks 108, 188 shown in
The machine learning model 606 may also be trained to identify successful and unsuccessful picks, from imagery of the picked item(s) captured within the images provided by the sensors positioned around, on, and/or within the robot. For instance, the labeled data 602, 604 may be input into a machine learning model 606, which may use learning methods to train the machine learning model to detect an unintentional doubles pick, detect that the correct item was picked, detect that the item was grasped at the targeted grasping location, detect that the picked item is in an appropriate orientation etc.
After training, the updated machine learning algorithm 608 may be stored and used to predict and select future grasping locations and grasping poses that will increase the accuracy of the picking instructions and minimize doubles picking. Further, the updated machine learning algorithm 608 may be used to identify unintentional doubles picks without the need for teleoperator review. For instance, neural networks 108 stored in robot control system 101 may be updated to include the updated machine learning model 608. In some instances, the updated machine learning algorithm 608 may be transmitted to robot 111 and used to update neural network 188. In some instances, the updated machine learning model may replace all existing neural networks.
In some instances, the updated machine learning model 608 may be stored by robot control system 101 or robot 111, such that the robot control system or the robot may use the updated machine learning model to predict future grasping locations or grasping poses when completing picking tasks. Yet further, the robot control system or the robot may use the updated machine learning model 608 to determine whether a picking task was successfully completed.
Per step 401 of the flow diagram of
In instances where the robot control system 101 selects a grasping location and/or grasping pose, processor 102 may transmit a signal including processor-readable information that represents the selected grasping location 734 and/or the selected grasping pose over network 160 to robot 211. Alternatively, processors 112 of robot 211 may run part of, or the entirety of, the grasping model to predict the one or more grasping locations and/or the one or more gasping poses rather than relying upon robot control system 101.
As shown in
Although the technology herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present disclosure. It is, therefore, to be understood that numerous modifications may be made to the illustrative embodiments and that other arrangements may be devised without departing from the spirit and scope of the present disclosure as defined by the appended claims.
