Patent Application
20170249561
The present disclosure relates to human-demonstrated learning of robotic applications, particularly those having force and position objectives.
Serial robots are electro-mechanical devices that are able to manipulate objects using a series of robotic links. The robotic links are interconnected by robotic joints, each of which is driven by one or more joint actuators. Each robotic joint in turn represents an independent control variable or degree of freedom. End-effectors disposed at the distal end of the serial robot are configured to perform a particular task, such as grasping a work tool or stacking multiple components. Typically, serial robots are controlled to a desired target value via closed-loop force, velocity, impedance, or position-based control laws.
In manufacturing, there is a need for flexible factories and processes that are able to produce new or more varied products with a minimum amount of downtime. To fully accomplish this goal, robotic platforms are required to quickly adapt to new tasks without time consuming reprogramming and code compilation. Traditionally, robots are programmed manually by coding the behavior in a programming language or through a teach pendent with pull-down menus. As the complexity of both the robot and the application increase, such traditional techniques have become unduly complex and time consuming. Therefore, an attempt to develop programs in a simpler, more intuitive way has developed known generally as “learning by demonstration” or “imitation learning”.
Using such methods, a human operator performs a task and a computer system learns the task by observing through the use of machine-learning techniques. Training operations are typically performed either by a human operator directly performing the task while a computer vision system records behaviors, or by the operator gripping the robot and physically moving it through a required sequence of motions. Such “learning by demonstration” techniques have the potential to simplify the effort of programming robotic applications with increased complexity. Robotic tasks typically have position or motion objectives that define the task. More so, these types of tasks have started to incorporate force or impedance objectives, i.e., objectives that specify the level of forces to be applied. When a task also requires force objectives, the use of position capture data alone is no longer sufficient. As a result, systems have evolved that attempt to learn such tasks by adding force sensors to the robot as the robot is moved or backdriven through a task demonstration. However, existing approaches may remain less than optimal for demonstration of certain types of dexterous tasks having both force and position objectives.
A system and accompanying method are disclosed herein for facilitating robotic learning of human operator-demonstrated applications having force and position objectives. The present approach is intended to greatly simplify development of complex robotic applications, particularly those used in unstructured environments and/or environments in which direct human-robot interaction and collaboration occurs. Unstructured environments, as is known in the art, are work environments that are not heavily configured and designed for a specific application. As the complexity of robots continues to increase, so too does the complexity of the types of robotic tasks that can be performed. For instance, some emerging robots use tendon-actuated fingers and opposable thumbs to perform tasks with human-like levels of dexterity and nimbleness. Traditional task programming and conventional backdriving task demonstration for such robots is thus complex to the point of being impracticable.
In an example embodiment, a system for demonstrating to a robot a task having both force and position objectives includes a glove that is wearable by a human operator. The system also includes sensors and one or more controllers, with the controller(s) in communication with the sensors. The sensors collectively measure task characteristics while the human operator wearing the glove actively demonstrates the task solely through the human operator's actions. The task characteristics include distributed forces acting on the glove, as well as a glove pose and joint angle configuration.
The controller may be programmed to apply machine learning logic to the task characteristics to thereby learn and record the demonstrated task as a task application file. The controller is also programmed to execute the task application file and thereby control an operation of the robot, i.e., the robot automatically executes the task that was initially demonstrated by the human operator wearing the glove.
A method is also disclosed for demonstrating a task to a robot using a glove on which is positioned the sensors noted above. The method may include measuring the set of task characteristics using the glove while a human operator wears the glove and demonstrates the task, and then transmitting the task characteristics to a controller. The method may include processing the task characteristics via the controller using machine learning logic to thereby learn and record the demonstrated task as a task application file and generating a set of control signals using the task application file. The set of control signals is transmitted to the robot to thereby cause the robot to automatically perform the demonstrated task.
The above features and other features and advantages of the present disclosure are readily apparent from the following detailed description of the best modes for carrying out the disclosure when taken in connection with the accompanying drawings.
Referring to the drawings, wherein like reference numbers correspond to like or similar components throughout the several figures, a glove 10 is shown schematically in
With respect to the glove 10 shown in
Unlike conventional methodologies using vision systems to determine position and teach pendants to drive a robot during a given task demonstration, the present approach instead allows the human operator 50 to perform a dexterous task directly, i.e., by the human operator 50 acting alone without any involvement of the robot 70 in the demonstration. As shown in
To address this challenge, the human operator 50 directly performs the task herein, with the demonstrated task having both force and position objectives as noted above. In order to accomplish the desired ends, the glove 10 may be equipped with a plurality of different sensors, including at least a palm pose sensor 20, joint configuration sensors 30, and an array of force sensors 40, all of which are arranged on the palm 17, fingers 12, and thumb 12T as shown in
The task characteristics may include a distributed force (arrow F10) on the glove 10 as determined using the array of the force sensors 40, as well as a palm pose (arrow O17) determined via the palm pose sensor 20 and a joint angle configuration (arrow J12) determined using the various joint configuration sensors 30. The first controller 60, which may be programmed with kinematics data (K10) describing the kinematics of the glove 10, may processes the task characteristics and output a task application file (TAF) (arrow 85) to a second controller (C2) 80 prior to the control of the robot 70, as described in more detail later below. While first and second controllers 60 and 80 are described herein, a single controller or more than two controllers may be used in other embodiments.
With respect to the array of force sensors 40 shown in
The joint configuration sensors 30 of
In an example embodiment, the joint configuration sensors 30 may be embodied as individual resolvers positioned at each joint, or as flexible strips as shown that are embedded in or connected to the material of the glove 10. The joint configuration sensors 30 determine a bending angle of each joint, and output the individual joint angles (arrow J12) to the first controller 60 of
The palm pose sensor 20 of
Referring to
Optionally, the system 25 may include a camera 38 operable for detecting a target, such as a position of the human operator 50 or the operator's hands, or an assembled or other object held by or proximate to the operator 50, during demonstration of the task and outputting the same as a position signal (arrow P50), in which case the position signal (arrow P50) may be received as part of the measured task characteristics. A machine vision module (MVM) can be used by the first controller 60 to determine position of the human operator 50 from the received position signal (arrow P50) for such a purpose, e.g., by receiving an image file and determining the position via the machine vision module (MVM) using known image processing algorithms, as well as to determine a relative position of the glove 10 with respect to the human operator 50.
The first controller 60 can thereafter apply conventional machine learning techniques to the measured task characteristics using a machine learning (ML) logic module of the first controller 60 to thereby learn and record the demonstrated task as the task application file 85. The second controller 80 is programmed to receive the task application file 85 from the first controller 60 as machine-readable instructions, and to ultimately execute the task application file 85 and thereby control an operation of the robot 70 of
The respective first and second controllers 60 and 80 may include such common elements as the processor (P) and memory (M), the latter including tangible, non-transitory memory devices or media such as read only memory, random access memory, optical memory, flash memory, electrically-programmable read-only memory, and the like. The first and second controllers 60 and 80 may also include any required logic circuitry including but not limited to proportional-integral-derivative control logic, a high-speed clock, analog-to-digital circuitry, digital-to-analog circuitry, a digital signal processor, and the necessary input/output devices and other signal conditioning and/or buffer circuitry. The term “module” as used herein, including the machine vision module (MVM) and the machine learning (ML) logic module, may be embodied as all necessary hardware and software needed for performing designated tasks.
Kinematics information K72 of the end-effector 72 and kinematics information (K10) of the glove 10 may be stored in memory M, such that the first controller 60 is able to calculate the relative positions and orientations of the human operator 50 and/or the glove 10 and a point in a workspace in which the task demonstration is taking place. As used herein, the term “kinematics” refers to the calibrated and thus known size, relative positions, configuration, motion trajectories, and range of motion limitations of a given device or object. Thus, by knowing precisely how the glove 10 is constructed and moves, and how the end-effector 72 likewise moves, the first controller 60 can translate the motion of the glove 10 into motion of the end-effector 72, and thereby compile the required machine-executable instructions.
With respect to machine learning in general, this term refers herein to the types of artificial intelligence that are well known in the art. Thus, the first controller 60 is programmed with the requisite data analysis logic for iteratively learning from and adapting to dynamic input data. For instance, the first controller 60 can perform such example operations as pattern detection and recognition, e.g., using supervised or unsupervised learning, Bayesian algorithms, clustering algorithms, decision tree algorithms, or neural networks. Ultimately, the machine learning module (ML) outputs the task application file 85, i.e., a computer-readable program or code that is executable by the robot 70 using the second controller 80. The second controller 80 ultimately outputs control signals (arrow CC70) to the robot 70 to thereby cause the robot 70 to perform the demonstrated task as set forth in the task application file 85.
Step S104 includes measuring the task characteristics (TC) using the glove 10 while the human operator 50 wears the glove 10 and demonstrates the task. The sensors 20, 30, and 40 collectively measure the task characteristics (TC) and transmit the signals describing the task characteristics, i.e., the forces F10, palm pose O17, and the joint configuration J12, to the first controller 60. The method 100 continues with step S106.
At step S106, the first controller 60 may determine if the demonstration of the task is complete. Various approaches may be taken to implementing step S106, including detecting a home position or calibrated gesture or position of the glove 10, or detection of depression of a button (not shown) informing the first controller 60 that the demonstration of the task is complete. The method 100 then proceeds to step S108, which may be optionally informed by data collected at step S107.
Optional step S107 includes using the camera 38 of
Step S108 includes learning the demonstrated task from steps S102-S106. This entails processing the received task characteristics during or after completion of the demonstration via the machine learning (ML) module shown in
Step S110 includes translating the demonstrated task from step S108 into the task application file 85. Step S110 may include using the kinematics information K10 and K72 to translate the task as performed by the human operator 50 into machine readable and executable code suitable for the end-effector 72 shown in
At step S112, the second controller 80 receives the task application file 85 from the first controller 60 and executes a control action with respect to the robot 70 of
While the best modes for carrying out the present disclosure have been described in detail, those familiar with the art to which this disclosure pertains will recognize various alternative designs and embodiments may exist that fall within the scope of the appended claims.
