Patent Application
20150331415
Traditionally, tasks to be performed by a robot are programmed by roboticists and are meant to be executed in controlled (caged) environments. More recently, “learning by demonstration” methods have been developed. Learning by demonstration is an important paradigm for teaching robots to carry out tasks. Many task learning systems (e.g., program synthesis) require a set of examples in which a human operator specifies the exact trajectories that are to be carried out by the robot for a particular workspace arrangement. This set of example tasks is accomplished via teleoperation and/or direct manipulation of the robot. Once these trajectories are learned, the robot essentially mimics the human operator's demonstrations.
Robotic task demonstration interface embodiments described herein generally employ a user interface to synthesize a robotic control program based on user demonstrations of object repositioning tasks, where the user manipulates objects in a displayed workspace. In one implementation, this is accomplished using a computer system that includes a user interface and a display device. First, the computer system receives object data. This data represents a collection of objects characterized by their orientation, location and perceptual properties in a workspace associated with a robot. A virtual rendering of the workspace is displayed in which the collection of objects is depicted so as to reflect the orientation, location and perceptual properties found in the object data. The displayed objects from the collection are capable of being repositioned so as to be depicted in a different orientation, or location, or both within the virtual rendering of the workspace. In one implementation, the virtual rendering of the workplace and objects are displayed on a touch screen display, and the displayed objects are repositioned via touch gestures on the touch screen display. The computer system next receives user instructions to reposition one or more of the depicted objects for the collection. The object repositioning instructions are indicative of tasks that it is desired for a robot to perform on an object or objects in an actual workspace associated with the robot. However, the tasks are demonstrated without manipulating real world objects. In the implementation employing a touch screen display, the user instructions are received in response to user interaction with the touch screen. A robotic control program is then synthesized for instructing the robot to perform the tasks indicated in the received object repositioning instructions.
Robotic task demonstration interface embodiments described herein also allow for multiple sets of repositioning demonstrations to be performed, and for the resulting learned robotic control program to be executed virtually for validation purposes, before applying it to the robot. In one implementation, one or more additional sets of repositioning demonstrations are received before the robotic control program is synthesized and in one implementation additional repositioning demonstrations are received after the robotic control program is synthesized and virtually executed. In these implementation involving additional repositioning instructions, a new virtual rendering of the workspace is displayed in which a collection of objects characterized by their orientation, location and perceptual properties are depicted. The additional user instructions to reposition one or more of the objects depicted in the new virtual rendering of the workspace are then received. Once again, the object repositioning instructions are indicative of tasks that it is desired for the robot to perform on an object or objects in the actual workspace. A robotic control program is then synthesized for instructing the robot to perform the tasks indicated in the additional object repositioning instructions and previously-received object repositioning instructions.
It should also be noted that the foregoing Summary is provided to introduce a selection of concepts, in a simplified form, that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
The specific features, aspects, and advantages of the disclosure will become better understood with regard to the following description, appended claims, and accompanying drawings where:
In the following description of robotic task demonstration interface embodiments reference is made to the accompanying drawings which form a part hereof, and in which are shown, by way of illustration, specific embodiments in which the technique may be practiced. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the interface.
It is also noted that for the sake of clarity specific terminology will be resorted to in describing the robotic task demonstration interface embodiments described herein and it is not intended for these embodiments to be limited to the specific terms so chosen. Furthermore, it is to be understood that each specific term includes all its technical equivalents that operate in a broadly similar manner to achieve a similar purpose. Reference herein to “one embodiment”, or “another embodiment”, or an “exemplary embodiment”, or an “alternate embodiment”, or “one implementation”, or “another implementation”, or an “exemplary implementation”, or an “alternate implementation” means that a particular feature, a particular structure, or particular characteristics described in connection with the embodiment or implementation can be included in at least one embodiment of the robotic task demonstration interface. The appearances of the phrases “in one embodiment”, “in another embodiment”, “in an exemplary embodiment”, “in an alternate embodiment”, “in one implementation”, “in another implementation”, “in an exemplary implementation”, and “in an alternate implementation” in various places in the specification are not necessarily all referring to the same embodiment or implementation, nor are separate or alternative embodiments/implementations mutually exclusive of other embodiments/implementations. Yet furthermore, the order of process flow representing one or more embodiments or implementations of the robotic task demonstration interface does not inherently indicate any particular order not imply any limitations of the interface.
The robotic task demonstration interface embodiments described herein generally employ a user interface to synthesize a robotic control program based on user demonstrations of object repositioning tasks (e.g. sorting, kitting, or packaging), where the user manipulates objects in a displayed workspace. The following sections will provided more detail about the environment and processes involved in implementing the embodiments of the robotic task demonstration interface.
Before the robotic task demonstration interface embodiments are described, a general description of a suitable environment in which portions thereof may be implemented will be described. Referring to
In view of the foregoing environment, one general implementation of the robotic task demonstration interface embodiments described herein is accomplished by using a computing device to perform the following process actions. Referring to
Referring again to
Next, user instructions to reposition one or more of the depicted objects of the collection are received (process action 204). The object repositioning instructions are indicative of tasks that it is desired for the robot to perform on an object or objects in an actual workspace associated with the robot. However, the tasks are demonstrated without manipulating real world objects. In one implementation, the object repositioning instructions are generated via a touch gesture on the aforementioned touch screen display, as will be described in more detail later.
Referring once again to
Robotic task demonstration interface embodiments described herein also allow for multiple sets of repositioning demonstrations to be performed during training, and for the resulting learned robotic control program to be validated virtually, before applying it to the robot. In one implementation, the user performs multiple sets of repositioning demonstrations before the robotic control program is synthesized. For instance, in one exemplary implementation this is accomplished by performing process actions 200 through 204 of
In another implementation, the robotic control program is synthesized and played virtually. In this latter implementation, if the user is satisfied, then the program is deemed validated. However, if the user is not satisfied with the operation of the program, additional repositioning demonstrations can be performed and the program re-synthesized. This can continue until the last-synthesized robotic control program is played and the user is satisfied with its operation. For instance, in one exemplary implementation this is accomplished by performing process actions 200 through 206 of
It is noted that the foregoing implementations can be combined in any order. Thus, for example, a user might provided multiple sets of object repositioning instructions prior to causing a robotic control program to be synthesized, and then upon virtually executing the program decide it is not yet correct and provide one of more additional sets of object repositioning instructions, until a resulting version of the program is deemed correct.
In foregoing implementations, repositioning demonstrations performed after the initial repositioning of one or more of the depicted objects were said to be performed using a new virtual rendering of the workspace. A new virtual rendering of the robot's workspace can be accomplished in a variety of ways. In one implementation, an image of an actual workspace associated with a robot is captured and employed to produce the new virtual workspace rendering. If a previous workspace rendering was generated using an image of the actual workspace, the objects could be physically rearranged (or new object introduced, or old objects removed) prior to the new image being captured. In another implementation, a newly generated or previously rendered virtual workspace could be modified to create a new virtual workspace rendering—thus negating the need to physically rearrange the workspace. For example, the orientation or location or perceptual properties (or any combination thereof) of one or more of the depicted objects can be changed using conventional methods. These changes can be prescribed or random. In a case where a new object is added to the virtual workspace rendering, it can be an object obtained from an actual image of a robot's workspace, or a synthesized object. Further, the new object can come from a previously-established database of virtual objects. These database objects can also be obtained from actual images of a robot's workspace, or synthesized. The new virtual rendering of the robot's workspace can also be a completely new synthesized workspace depicting either virtual objects derived from actual images of a robot's workspace or synthesized objects that where specifically generated for the new virtual rendering or obtained from database of virtual objects. The use of a new virtual rendering of the robot's workspace for repositioning demonstrations performed after the initial repositioning of one or more of the depicted objects has the advantage of generalizing the resulting robotic control program to never before seen inputs.
In foregoing implementations, the robotic control program was synthesized and then played virtually for validation by the user. This virtual playback can be implemented using the currently displayed virtual workspace rendering. However, this need not be the case. The virtual playback can also be implemented using a previously-created virtual rendering of the robot's workspace, or a new virtual rendering of the robot's workspace. In the case of a new virtual rendering, it can be obtained in the same ways as described previously.
It was described previously that the robotic control program is synthesized and can then be played virtually for validation by a user. The program was deemed validated if the user was satisfied. However, if the user was not satisfied with the operation of the program, additional repositioning demonstrations could be performed and the program re-synthesized. This would continue until the last-synthesized robotic control program is played and the user is satisfied with its operation. However, in addition to, or instead of, additional repositioning demonstrations being performed, in one implementation, the user can also designate the last-synthesized robotic control program as representing a negative example. In such a case, when the robotic control program is re-synthesized not only are the positive repositioning demonstrations taken into consideration, but also the designated negative examples. As a result, the re-synthesized program will not exhibit the behavior deemed incorrect by the user.
As described previously, one implementation of the robotic task demonstration interface embodiments described herein involves displaying a virtual rendering of a robot's workspace in which a collection of objects is depicted so as to reflect the orientation, location and perceptual properties characterized found in received object data. Each of the displayed objects from the collection is capable of being repositioned so as to be depicted in a different orientation, or location, or both within the virtual rendering of the workspace. In one implementation, the virtual workspace rendering is part of a virtual workspace training interface that a user can employ to move objects and demonstrate intended actions. To the user, this feels much like reaching into a photograph and manipulating the scene.
One exemplary implementation of the virtual workspace training interface 500 is shown in
The control bar 504 includes a number of control icons used in the training process. These include a “Clear” button 508 that when selected removes the currently displayed virtual workspace 502 from the interface 500. The “Cam” button 510 is used to input and display a new virtual workspace 502 derived from a currently captured image of the actual robot workspace. The “Shuffle” button 512 rearranges one or more of the displayed objects 506 by moving them to a new location in the virtual workspace 502, or reorienting the pose of an object, or both. Further, a “Simulate” button 514 is included which when selected generates and displays a simulated virtual workspace 502 as described previously.
Once a virtual workspace 502 is displayed in the interface 500, the user can manipulate the displayed objects 506 to demonstrate the type of actions the user wishes the synthesized robotic control program to accomplish via the robot. In other words, the user demonstrates examples of the intended actions in order to train the program. As described previously, in one implementation, at least one set of repositioning instructions is generated via the user demonstrating desired movements of one or more objects 506 via the interface 500, before a version of the robotic control program is synthesized. In one implementation this is accomplished via a touch screen. The user employs the touch screen to move objects, and multi-finger touch is employed to control of the pose of each object (e.g., it location and orientation). An object can be moved using one finger, and rotation requires two fingers. This is accomplished even with very tiny objects by first placing one finger on the object to select it, and causing an affordance “patch” to appear around the object. This patch has a size large enough to allow a move or rotation gesture to be performed by the user using one or two fingers. More particularly, referring to
Referring again to
As was described previously, if the last-synthesized program is deemed satisfactory by the user, it is considered validated and the training ends. However, if the user was not satisfied, additional repositioning demonstrations could be performed and the program re-synthesized in the manner described above. This would continue until the last-synthesized robotic control program is played and the user is satisfied with its operation. However, in addition to, or instead of, additional repositioning demonstrations being performed, the user can also designate the last-synthesized robotic control program as representing a negative example. In the example interface of
Once the user is satisfied with the last-synthesized robotic control program, he or she can select the “Do it!” button 528 to execute the program against the real world configuration using the actual robot.
The robotic task demonstration interface embodiments described herein are operational within numerous types of general purpose or special purpose computing system environments or configurations.
To allow a device to implement the robotic task demonstration interface embodiments described herein, the device should have a sufficient computational capability and system memory to enable basic computational operations. In particular, the computational capability of the simplified computing device 10 shown in
In addition, the simplified computing device 10 shown in
The simplified computing device 10 shown in
Retention of information such as computer-readable or computer-executable instructions, data structures, program modules, and the like, can also be accomplished by using any of a variety of the aforementioned communication media (as opposed to computer storage media) to encode one or more modulated data signals or carrier waves, or other transport mechanisms or communications protocols, and can include any wired or wireless information delivery mechanism. Note that the terms “modulated data signal” or “carrier wave” generally refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. For example, communication media can include wired media such as a wired network or direct-wired connection carrying one or more modulated data signals, and wireless media such as acoustic, radio frequency (RF), infrared, laser, and other wireless media for transmitting and/or receiving one or more modulated data signals or carrier waves.
Furthermore, software, programs, and/or computer program products embodying some or all of the various robotic task demonstration interface embodiments described herein, or portions thereof, may be stored, received, transmitted, or read from any desired combination of computer-readable or machine-readable media or storage devices and communication media in the form of computer-executable instructions or other data structures.
Finally, the robotic task demonstration interface embodiments described herein may be further described in the general context of computer-executable instructions, such as program modules, being executed by a computing device. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The robotic task demonstration interface embodiments may also be practiced in distributed computing environments where tasks are performed by one or more remote processing devices, or within a cloud of one or more devices, that are linked through one or more communications networks. In a distributed computing environment, program modules may be located in both local and remote computer storage media including media storage devices.
Additionally, the aforementioned instructions may be implemented, in part or in whole, as hardware logic circuits, which may or may not include a processor.
It is noted that any or all of the aforementioned embodiments throughout the description may be used in any combination desired to form additional hybrid embodiments. In addition, although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
