Patent Application
20230373098
The present inventions relate generally to robots, and more particularly, to programming robots.
In order to configure a robot for a specific task, expert robot engineers are typically required to program it with dedicated languages and operating systems. For example, robots are currently programmed using special-purpose languages (e.g., ABB's RAPID language), which requires in-depth knowledge of the robot's technical capabilities (number of joints, degrees of freedom, range, coordinate systems, etc.). The engineer must also have sufficient knowledge of the dedicated language used for programming the robot. To build a robot application, it is necessary for the engineer to understand the capabilities and limitations of the specific target robot. This in turn requires an in-depth knowledge of the hardware implementation and mechanical properties of the robot. Further, when a robot is not able to perform a given task (e.g., due to kinematic limitations), numbered error codes are generated which are understandable only by expert users. Understanding such messages in order to be able to resolve an underlying problem can be difficult for non-experts. Conventional robots typically do not offer an interface for intuitive user instructions that allow clear feedback from the robot in case of errors or infeasible instructions. This makes it difficult for humans to understand how far a robot can reach and what its limitations are in terms of feasible movements. As a result, robot programming is considered a challenging task requiring software development and mechanical engineering skills, as well as specialized training.
Employing specialized engineers is costly for high-mix, low-volume enterprises where robots need to be reconfigured on a regular basis (e.g., every few minutes or hours in workshops, medical labs, restaurants, etc. where collaborative robots are employed for tasks with low batch sizes). The cost and delay associated with conventional robot programming may quickly become prohibitive where robots need to be frequently reprogrammed, which is typical, e.g., for collaborative robots. Such robots can be used in small workspaces, medical labs, personalized gift industries, restaurant kitchens, etc. where reconfiguration of the robot may be necessary multiple times per day. As robots (especially collaborative robots) gain quick adoption in many application domains, it is essential to ensure that programming robots to perform various tasks is intuitive and accessible to a wide range of users, not only to robotics experts. Thus, reducing costs and simplifying reconfiguration of robots would provide future automation applications, e.g., with collaborative robots in small workshops.
A method and system are described for programming robots. In order to program the robot, specialized software commands are needed. However, the operator may not have expertise with the specialized software commands used to program a robot. Thus, the system enables the operator to provide generic inputs describing the desired robot movement without needing to specify actual software commands. The system translates the operator inputs into software commands that direct the robot to perform the desired robot movement. The robot may then be programmed and operated using the translated software commands to perform the desired robot movement. The invention may also include any other aspect described below in the written description or in the attached drawings and any combinations thereof.
The system may be configured for programming the robot by the operator using any method described herein. The system may comprise an operator communication module configured for receiving the operator input; an environmental sensor module configured for capturing identification and/or location information; an actuation module configured for translating the operator input into the software commands; and possibly a controller module configured for coordinating. Also, a use of the system for programming the robot using any method described herein is provided.
The invention may be more fully understood by reading the following description in conjunction with the drawings, in which:
A new method and system are provided herein for enabling intuitive programming of robots. The method and system enables non-experts to co-create automation tasks with robots based on an intuitive programming method through natural interaction via multiple channels. Speech input/output, recognition of objects and hand gestures, and a graphical user interface guide users through the co-creation of robotic tasks and help them handle potential errors. Additional communications channels that may be used include augmented/mixed reality to overlay planned trajectories before execution, haptic gloves to provide tactile feedback and/or provide a direct sensation of kinetic limits, and thermal sensors for (scaled) temperature feedback. The fusion of multiple communication channels makes the interaction intuitive and natural, e.g., the operator can instruct the robot by either speech input or clicking a button on a touch screen. This way, robot programming becomes accessible to ordinary operators and the initial setup and reconfiguration of the robot's functionality can be completed within a short time. Thus, productivity may be increased and commissioning costs of robots may be decreased. In turn, this may open new market segments for collaborative robots in application areas where frequent robot reconfiguration is required.
The described method and system enables non-expert users to teach a robot a task by means of several input modalities (e.g., speech, vision, sensors, touch, showing a movement by lead-through, etc.) and provide immediate feedback in a user-friendly and understandable way (e.g., speech, display, acoustic signals, etc.). It is understood herein that references to an operator of the robot refers to a non-expert user with minimal or no programming skills using software commands of special-purpose robot programming languages. By using a modular design, an advantage of the system is its compatibility in the form of add-on packages with several communication channels as well as a wide range of robots.
A first step of the interaction involves the operator instructing the robot to perform a sequence of operations by providing high level concepts (e.g., pick, move, place) and object names known to humans (e.g., block, pencil, screw, this object here). The instructions may be enhanced with attributes (e.g., red pen, big screw, rotate clockwise, etc.). In a second step, if an instruction is not understood, then the operator is prompted for clarification (e.g., which object?) and several choices may be offered (e.g., by showing the objects that are visible to the robot's camera). The operator is then able to provide feedback to the robot using one of the several input modalities (e.g., speech input, touch screen, hand gesture) so that the intended task can be completed.
The improved robot programming method and system provides a natural interaction between the operator and the robot system which arises from the interplay of the described modules. The resulting interaction between non-expert users and the robot is similar to the way humans teach each other new skills. The interaction leverages multiple in- and output modalities such as speech recognition and synthesis, vision, gesture recognition, touch, lead-through mode (physically guiding the robot arms) and visualization on a display for intuitively teaching rather than programming the robot using software code.
A general architecture for the interactive programming framework is illustrated in
It is understood that the list of operator communication channels (i.e., inputs to the operator module 10) is not limited to the examples in
The actuation module 14 provides an interface towards the robot 18. On the one hand, the actuation module 14 translates task requests into software commands that are understandable by the robot 18. The robot 18 represents an existing robot programming interface, e.g. RobotWare or RAPID, which uses special-purpose robot programming software commands to direct robot movement. On the other hand, the actuation module 14 provides feedback from the robot 18 on the success or failure due to errors of the previously requested task. By changing the interface towards the robot 18 controller, the actuation module 14 may be integrated with robots 18 of different makes and types.
The controller 16 acts as a coordinator between the different modules 10, 12, 14. The controller 16 may be a centralized module 16 or may be distributed across multiple modules 10, 12, 14. The controller module 16 is responsible for fusing together the operator inputs, environment sensors and robot feedback. This fusion process of taking inputs from the operator through the operator module 10, sensors from the environmental sensor module 12, and feedback from the actuation module 14 allows a robot 18 to be taught skills like a human domain expert would teach an apprentice. The set of operator commands that are recognized by the controller module 16 may be called a skill set and may be dynamically extended over the robot's lifetime.
In an example of the method and system, an operator may desire the robot to “pick a tube”. The operator may also describe a sequence of robot movements and/or conditions for that robot movement to be executed. For example, the operator may desire the robot to “pick the tube and place it in the red bin” or “after a tube is available in the blue bin, pick it and shake it”. The operator may choose to command the system with an operator input either via speech recognition 10A or by employing a graphical user interface 10B. If the skill (i.e., pick) and object (i.e., tube) are known to the controller module 16, the controller module 16 may proceed to request a task (or set of tasks) execution(s) to the actuation module 14 which will translate the tasks into low-level robot software commands for the robot 18 controller. If a command is insufficient, e.g., the action is not understood or any information is missing, e.g. the object is unknown to the system, the controller module 16 may prompt one or more additional operator inputs to provide complementary input. An example interaction between the modules is shown in
Feedback requests and operator input may be provided by any of the operator communication and sensing modules 10, 12 best suited for a natural and intuitive information flow. For example, if the robot 18 does not know a certain movement, the robot 18 may be taught by example using a lead through mode where the operator moves the robot 18 and the robot 18 records the movement. In another example, the operator can register an object with the system by selecting it on an image on the touchscreen 10B. This example also highlights the importance of fusing multiple interaction modalities with the operator and environment. That is, it would be unnatural to identify an object via lead through if it can be easily identified on the touchscreen 10B. Likewise, it typically would require artifacts to teach 3-dimensional movements using a flat 2-dimensional screen.
The controller module 16 registers the additional operator inputs in an appropriate form for future use. For example, movements can be registered as path points and objects as visual fingerprints. Importantly, the format allows learned skills to be transferrable between robots 18. Thus, if a robot 18 is trained with a skill for an application domain, e.g., medical labs, all required information is encoded into the modules of the architecture of
Referring to
The operator inputs (and additional operator inputs) are then translated into software commands that will direct the robot 18 to perform the desired robot movement (26), for example, by an actuation module 10. Thus, it is understood that the operator input is not an actual software command, but instead, is a form of common text input, pointer, or other operator input that generically describes the desired robot movement. By contrast, the translated software commands are a form of specialized software code understood by engineers and the robot 18 but not readily used or understood by ordinary operators. It may be desirable to store the operator inputs, additional operator inputs and/or translated software commands in a library storage with data links therebetween so that future operator inputs may be able to be autonomously linked to such additional operator inputs and/or software commands (28). The library storage may be included as part of the actuation module 14 and may be a skill set library that recognizes operator inputs.
Once the operator inputs have been translated into software commands understandable to the robot 18, the robot 18 may be programmed with such software instructions (30). Subsequent to such programming, the robot 18 may be operated using the software commands in order to perform the desired robot movement (32). After the robot 18 has been operated using the software commands, it may be desirable to determine if the robot 18 successfully performed the desired robot movement (34). If it is determined that the robot 18 was unsuccessful, the operator may be prompted for additional operator input (20) and the method may be repeated using the original operator input and the additional operator input (34). A controller module 16 may coordinate between the operator communication module 10, environmental sensor module 12 and/or actuation module 14. Further, the controller module 16 may perform the determinations of whether the operator input is sufficient for translation and/or whether the operated robot movement successfully performed the desired robot movement, and may prompt the operator for additional inputs when needed. It may be desirable for the operator communication module 10, environmental sensor module 12, actuation module 14, and/or controller module 16 to be independent software or hardware modules that operate independently from each other. For example, the modules 10, 12, 14, 16 may be micro services that communicate with each other via messages. Thus, the modules 10, 12, 14, 16 may be implemented in a decentralized fashion.
While preferred embodiments of the inventions have been described, it should be understood that the inventions are not so limited, and modifications may be made without departing from the inventions herein. While each embodiment described herein may refer only to certain features and may not specifically refer to every feature described with respect to other embodiments, it should be recognized that the features described herein are interchangeable unless described otherwise, even where no reference is made to a specific feature. It should also be understood that the advantages described above are not necessarily the only advantages of the inventions, and it is not necessarily expected that all of the described advantages will be achieved with every embodiment of the inventions. The scope of the inventions is defined by the appended claims, and all devices and methods that come within the meaning of the claims, either literally or by equivalence, are intended to be embraced therein.
| Number | Date | Country | Kind |
|---|---|---|---|
| 20192460.2 | Aug 2020 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2021/072505 | 8/12/2021 | WO |
