Patent Application
20130085602
The present invention relates to a system of robots for office use.
There have been various publications about office robots. Office robots, as the name may suggest, perform jobs in the office environment. Autonomous office robots may be used as visitor guides and for running errands, promoting products, cleaning, etc. There are also semi-autonomous office robots that facilitate worker interaction via video-conferencing; in that case, tele-operators may remotely control the office robots to look for and interact with co-workers in offices. At the technical level, the office jobs described may require the robots to be capable of recognizing some people, communicating with the people, locating the people, moving from place to place in the office while avoiding obstacles, and collaborating with other robots. While prior-art inventions focus on capability of an individual robot, such as its artificial intelligence, its electro-mechanical design, and distributed computing within the robot, the present invention relates to a system where autonomous and semi-autonomous office robots can be cost-effectively deployed and managed through wireless networking and distributed computing.
The object of this invention is to provide a cost-effective system where various autonomous and semi-autonomous office robots can be deployed, controlled and managed. There are the capital expenditure (CAPEX) and operational expenditure (OPEX) aspects to be considered to make a system cost-effective. This invention can reduce both CAPEX and OPEX in deploying various office robots to achieve their desired functionalities. In this invention, we use a computing cluster that is capable of communicating to the robots via the corporate networking infrastructure. The computing cluster is to off-load the robots from the computation that is proved to be too burdensome to the robots. To achieve that, we architect the robot software into layers and functional modules within layers. Each robot is made capable of measuring the performance of executing the functional modules and the performance of passing information among the functional modules from one layer to another. The performance data is used for deciding which functional modules are to be executed on each robot and which complementary functional modules are to be executed on the computing cluster. There are a few advantages of this approach. Firstly, the many robots in the system can be built as low processing power and memory capacity mobile computing devices. The CAPEX on robots, as they are many in the system, can therefore be reduced. Secondly, there may be multiple versions of the robots, even of the same model, as technology is advanced along the years. Meanwhile, the software for the robots may also be upgraded along the years, hence demanding more processing power and memory capacity than before. It will be economical to keep even the older versions of the robots around. Our invention enables the older versions of the robots to execute as much as they can handle the upgraded software because the computing cluster off-loads the robots intelligently. Thirdly, the computing cluster centralizes the management and administration of the robots. Also, it enables collaborative learning and planning among the robots. For example, through collaborative planning, the robots are less likely getting into each other's way, hence requiring human intervention. All those translate into saving in OPEX. Lastly, the computing cluster can be built with state-of-the-art technologies in which computing power and memory capacity can be added to the computing cluster on demand. That is more manageable and more economical than constantly making sure the robots to be the same version or to run the same version of software.
Another object of this invention is disclosing how to partition the robot system software functional modules between the robots and the computing cluster. The functional modules that pertain specifically to the robot hardware are to be executed on the robots. Also, a safety monitoring module and the functional modules that the safety monitoring module depends on are to be executed on the robots. To enable the computing cluster to correctly interpret the data transferred from the robots, the robots should synchronize their clocks to the clock of the computing cluster and time-stamp the data.
Yet another object of this invention is disclosing how the performance data collected can be used to influence how the robots execute the functional modules and transfer the data to the computing cluster and the tele-operators' computers. The latency and bandwidth of the communications between the robots and the computing cluster may affect the image processing complexity. The latency and bandwidth of the communications between the robots and the tele-operators' computers may affect the actuation rates of robot actuators and the quality of the images transferred from the robots to the tele-operators' computers.
The present invention will be understood more fully from the detailed description that follows and from the accompanying drawings, which however, should not be taken to limit the disclosed subject matter to the specific embodiments shown, but are for explanation and understanding only.
This invention is expected to be deployed in a corporate environment similar to one depicted in
The computing cluster 24 in this invention comprises a set of computers and robot management software developed on a distributed processing framework and run over the set of computers. Although physically the same set of computers can support typical business applications, such as enterprise resource planning software, in the corporate computing infrastructure 25, we would like to make the computing cluster 24 a distinct logical entity. We shall refer to the suite of typical business applications running over a set of computers as the existing corporate computing infrastructure 25. In other words, in this invention, the computing cluster 24, which is related to the robot system, is logically distinguished from the existing corporate computing infrastructure 25. Such distinction may facilitate the actual deployment of the robot system without disruption on the existing corporate computing infrastructure 25. That said, the computing cluster 24 and the existing corporate computing infrastructure 25 can also be physically distinct.
The computing cluster 24 handles the robot software computation in a distributed manner and maintains a knowledge database 68 in a distributed manner. The knowledge database 68 comprises data, processed or unprocessed, gathered by the office robots and a priori knowledge provided by system administrators. For example, the knowledge database 68 may comprise data of facial and speech characteristics of various employees and visitors and data related to inventory and physical properties. A distributed processing framework addresses the scalability issues. The knowledge database 68 grows as more data are collected. Moreover, as the knowledge database 68 grows, the robot software computation load may also grow as database search may take longer. Also, when more robots are added to the system, the robot software computation load increases. The advantage of using distributed processing framework is the ability to add more computers to the computing cluster 24 as the computing requirements increase. That minimizes CAPEX as new resource is added to the computing cluster 24 only when needed.
Centralizing the knowledge database 68 in the computing cluster 24 enables collaborative collection of data and sharing of data by the office robots 26. The office robots 26 gather data in various office locations and at various times and contribute to the knowledge database 68. The process makes the knowledge database 68 richer and more trustworthy. The computing cluster 24 using a distributed processing framework also provides data storage redundancy. That again alleviates the memory requirements on the office robots 26.
The OPEX is reduced through centralized management of robots 26. The computing cluster 24 supports robot management applications. Software upgrade can be pushed from the robot management applications to the robots 26. Also, the robot management applications can analyze the status and utilization of the robots 26 so that the information technology department can justify the expense of the robot system to corporate executives. Also, remote access to control the semi-autonomous robots 26 can also be authenticated and authorized by the robot management applications.
The computing cluster 24 is not limited to information gathered by the robots 26 in the system. It can integrate information retrieved from the existing corporate computing infrastructure 25. For example, the computing cluster 24 can access the employee workplace location, employee picture, and employee contact information, all stored in the existing corporate computing infrastructure 25. The computing cluster 24, on the other hand, may have stored an employee's images of various viewpoints, captured by the office robots 26. Among those images of various viewpoints, there may be one employee front-face image. That can be used to match the employee front-face picture stored in the existing corporate computing infrastructure 25. In that way, all information about the employee whose image is captured by the office robots 26 can be linked to the employee information stored in the existing corporate computing infrastructure 25. From then on, a robot 26 can recognize an employee from various viewpoints and use his contact information with the help of the computing cluster 24.
The computing cluster 24 may send decisions of robot software computation to office robots 26 to direct the office robots 26 to perform actions. Also, the computing cluster 24 may send messages, such as emails and voice mails, to office workers depending on the applications.
The computing cluster 24 can support robot collaboration. For example, there are multiple floor sweeping robots on the same floor. The computing cluster 24 can coordinate the robots to cover the whole floor.
The robot system in this invention supports tele-operators remotely controlling the robots. The tele-operators may control some specific actions of the robots, or they may provide missions for the robots to carry out autonomously. The tele-operators would need to view the robot environment visually, for example, through images captured by the robots and observe sensory data. In our preferred embodiment, the images captured by the robots are conveyed to the computing cluster 24, for analysis, and then to the tele-operators' computers, terminals, browsers, or application software.
The robot system in this invention assumes certain corporate networking infrastructure. The corporate networking infrastructure assumed is typical of modern corporate network deployment and is optimal for addressing the security and computation load aspects of the robot system.
The robot system's networking infrastructure may comprise wireless Local Area Networks (wireless LANs), wired Local Area Networks (wired LANs), and Virtual Private Networks (VPNs).
The wireless LANs are needed as the office robots are considered to be light-duty mobile computing devices in the robot system. Robots have the ability to move around and should not be confined by wired connections. On the other hand, the computing cluster is usually on a wired LAN, i.e., the many computers in the computing cluster are connected via wired LAN. Wired LAN provides lower latency and higher bandwidth relative to wireless LAN, so wired LAN is more appropriate for the distributed processing nature of the computing cluster. When office robots and the computing cluster are co-located, they communicate via wireless LAN and wired LAN.
VPNs are needed when office robots and the computing cluster are connected by the Internet, or when tele-operators' computers and the computing cluster are connected by the Internet. VPN provides secure connectivity and, in some case, service level agreement on quality of service.
Let's assume that the computing cluster 24 resides in the main office 20. Office robots 26 in branch offices 10 also need to access the computing cluster 24 through the Internet 30. We may deploy IPSec (Internet Protocol Security) VPN or MPLS (Multi-Protocol Label Switching) VPN between branch offices 10 and the main office 20. Then the office robots 26 in a branch office 10 communicate to the computing cluster 24 via wireless LAN 23 in the branch office 10, IPSec VPN or MPLS VPN over the Internet 30, and wired LAN 21 in the main office 20.
The advantage of using IPSec VPN or MPLS VPN is the encryption of data that protects from eavesdropping over the Internet 30. Moreover, the encryption and decryption is performed using dedicated VPN gateways so that the office robots 26 are not burdened with the computation load. That again echoes our theme of enabling office robots 26 to be light-duty mobile computing devices.
The use of SSL VPN in this invention is mainly for tele-operators remotely controlling semi-autonomous office robots 26 via web applications. The web applications need to first contact the robot management applications on the computing cluster 24 to obtain authorization. Then they can set up separate SSL connections to the office robots 26 directly without the computing cluster 24 in the middle or control the office robots 26 indirectly with the computing cluster 24 as the middleman.
The computing cluster 24 does not need to be located physically in the main office 20. It can be hosted by the ISP (Internet Service Provider) that provides the VPN service to the main office 20.
The adaptive distribution of robot software computation is at the heart of this invention as it offers multiple advantages. Firstly, the office robots can be treated light-duty mobile computing devices. They can be built cheaply and without over-provisioning in their processing power and memory capacity to accommodate current software and future software upgrades. Secondly, the office robots may end up having a number of versions with different processing power and memory capacity along the years of deployment. Yet, they may deliver the same software features thanks for the computing cluster taking part of the computing responsibilities from the robots.
An embodiment of a robot 26 in the system is illustrated in
The sensor controller 51 is responsible for all sensors on the robot 50, e.g., infrared sensors, microphones, and cameras. The sensor controller 51 feeds data collected on the sensors into the instruction executor 53.
The instruction executor 53 is responsible for executing instructions and capable of interacting with the sensor controller 51, the actuator controller 55, and the instruction loader 54. The instructions may be codes written in an interpretive programming language and compiled codes, or a combination of both.
In our preferred embodiment, the instructions are written in Java, a programming language. The instruction executor 53 comprises the Java virtual machine and some software that enables the instruction executor 53 to interact with the sensor controller 51, the actuator controller 55, and the instruction loader 54.
The performance monitor 52 is capable of collecting performance data about the instruction executor 53 executing the instructions and also about the communications where the instruction executor 53 is relaying data to the server-side instruction executor 63 on the computing cluster 60 via the corporate networking infrastructure. The performance data is a key deciding factor on how to distribute the instructions among the instructor executor 53 and the server-side instruction executor 63.
The instruction loader 54 is responsible for providing instructions to the instruction executor 53. The instructor loader 54 is capable of loading instructions from the instruction server 64 on the computing cluster 60 and providing the instructions to the instructor executor 53. The instructor loader 54 uses performance data collected the performance monitor 52 and may communicate the performance data to the instruction server 64. Also, the instructor loader 54 may keep some instructions locally in a persistent storage so that it does not need to rely on the instruction server 64 all the time. In one embodiment, the instructor loader 54 may decide what subset of instructions to be loaded from the instructor server 64. In our preferred embodiment, the instructor loader 54 passes the performance data to the instructor server 64 and let the instructor server 64 decides what subset of instructions to be loaded onto the instructor loader 54 and what functionally complementary subset of instructions to be executed by the server-side instructor executor 63.
The computing cluster 60 may also monitor the performance of communications between the instructor executor 53 of the robot 50 and the server-side instructor executor 63 of the computing cluster 60, but the performance monitor 52 has exclusive knowledge about the performance of the instructor executor 53. Also, the knowledge about the processing capability and memory capacity of the robot can also be used to decide on the distribution of the instructions.
The actuator controller 55 is responsible for all actuators on the robot, e.g., motors, light-emitting diodes (LEDs), display screens, and speakers. The actuator controller 55 services the local decisions generated by the instruction executor 53 resulting from executing the instructions.
An embodiment of the computing cluster 24 is illustrated in
The instruction server 64 is responsible for storing the instructions of the robot system software. In our preferred embodiment, the instruction server 64 further decides how to divide the instructions among an instructor executor 53 of a robot and the corresponding server-side instructor executor 63.
On the computing cluster 60, there is one server-side instruction executor corresponding to one instruction executor of a robot. For example, if there are a hundred robots 50 in the system, then there are a hundred server-side instruction executors 63. The server-side instruction executor 63 executes instructions provided by the instruction server 64. There may be a few ways of implementing the server-side instruction executors 63. Each server-side instruction executor can be a separate process. Alternatively, each server-side instruction executor represents a separate context while some or all server-side instructor executors are run in one process.
We would like to present the notion that there is one server-side instruction executor corresponding to one instruction executor so that the possibility of scaling the computing cluster is clear. Also, depending on factors including the performance data collected by the performance monitors of the robots in the system, each pair of instruction executor and server-side instruction executor may divide the instructions of the robot software differently.
The instructions implement the robot software in the robot system. The instruction server decides on how to divide the instructions for each pair of instruction executor and server-side instruction executor. The instructions are designed to be divisible. The instructions are modularized and data are passed between the modules of the instructions. When the instruction server provides a subset of instructions to an instruction executor, the instruction server provides a complementary subset of instructions to the corresponding server-side instruction executor.
When some modules of instructions require intensive processing power or memory capacity, they are more suitable to be executed by the server-side instruction executor. When some modules of instructions trigger a high bandwidth of data passing among them, they are more suitable to be put in the same subset of instructions to be executed by the instruction executor or the server-side instruction executor. When some modules of instructions are supposed to produce time-critical decisions on a robot, they are more suitable to be executed by the instruction executor on the robot.
To facilitate division of the instructions, there are several provisions in this invention. Firstly, the robot software is to be architected with linear dependency. Secondly, the instructions are to be hardware independent and interpreted at run-time.
In an embodiment, the robot software stack is designed to have multiple layers with linear dependency. As in
The driver layer 130 handles the low-level functions required to operate the robots. The driver layer 130 is composed of device drivers for the sensors and actuators of a robot. A robot may have a number of sensors and actuators. Sensors allow robots to receive information about a certain measurement of the environment or internal components. They may include touch, vision, distance measurement, etc. Actuators are devices for moving or controlling something. They may include motors, artificial muscles, grippers, effectors, etc. We simply use sensor driver module 132 and actuator driver module 134 to represent a set of sensor drivers and a set of actuator drivers, respectively. The actual modules in this layer depend on the sensors and actuators used in a robot and the operating system that the driver software runs on. In general, modules in this layer take actuator control inputs, from the platform layer 120, in engineering units, e.g., positions, velocities, forces, etc. and generate the low-level signals that create the corresponding actuation. Similarly, this layer contains modules that take raw sensor data, convert it into meaningful engineering units, and pass the sensor values to the platform layer 120. The sensor driver module 132 may be associated with the sensor controller 51. The actuator driver module 134 may be associated with the actuator controller 55.
The platform layer 120 contains functional modules that correspond to the physical hardware configuration of the robot. This layer frequently translates between the driver layer 130 and the local intelligence layer 110 by converting low-level information into a more complete picture. The platform layer 120 is composed of functional modules that are specific to the robots. The platform layer 120 may include a sensing module 124, a steering module 126, an image processing module 122, a kinematics module 128, etc. The sensing module 124 is responsible for processing various sensor data. The image processing module 122 is responsible for processing the images or videos captured via camera. The steering module 126 is responsible for locomotion of the robot. The kinematics module 128 is responsible for operating the actuators. Different robots have different capabilities and characteristics. For example, some robots have cameras while some do not. Some robots have grippers while some do not. Therefore, not all the mentioned functional modules are relevant for some robots, and the same functional module is implemented differently for different robots.
The local intelligence layer 110 consists of functional modules of the high-level control algorithms for the individual robots. The functional modules take system information such as position, velocity, or processed video images and make control decisions based on all of the feedback. This layer might include a mapping and localization module 112, a path planning module 114, an obstacle avoidance module 116, a local goal setting module 118, a safety monitoring module 119, etc. The mapping and localization module 112 is responsible for identifying the location and position of the robots. The path planning module 114 is responsible for guiding the robots through their environments. The obstacle avoidance module 116 is responsible for guiding the robots around their obstacles. The local goal setting module 118 is responsible for defining the missions of the robots. The safety monitoring module 119 is responsible for aborting or reversing the actions that are causing problems or hazards. There may be some dependencies within the functional modules. For example, the local goal setting module 118 may send control signals into the path planning module 114.
The global intelligence layer 100 consists of functional modules of the high-level control algorithms for the system of robots. The functional modules may include a global goal setting module 102 a collaborative planning module 104, a recognition module 106, etc. The functional modules in this layer may contribute to and make use of the knowledge database 68, which contains information collected by the robots in the system and a priori knowledge provided by system administrators. The global intelligence layer 100 uses information from the local intelligence layer 110 and provide decisions and control signals to the local intelligence layer 110. The global goal setting module 102 defines the missions for all the robots in the system and use the local goal setting module 118. The global goal setting module 102 may get inputs from and provide outputs to the user interface layer 90. The recognition module 106 is responsible for object recognition, face recognition, speech recognition, etc. and may use the knowledge database 68.
The user interface layer 90 is responsible for presenting the robot system to the system administrators and tele-operators and receiving inputs or missions from them. The user interface layer may include a robot system display module 92, a robot system control module 94, and a mission definition module 96. In one embodiment, this layer provides web applications to the system administrations and tele-operators to interact with the robot system. The system administration may modify the data to and operate on the knowledge database 68.
The linear dependency among the layers facilitates the partition of the instructions representing the robot system software stack to be executed on the instruction executor and the corresponding server-side instruction executor.
In one embodiment, there are a number of pre-built packages where each package presenting a way of partitioning the instructions. One of the-built packages is selected by the instruction server 64 based on the performance data.
In another embodiment, the instruction server 64 partitions the instructions dynamically based on the performance data and compile the partitions of the instructions on-the-fly before providing the subset of instructions to the executor loader 54 and the complementary subset of instructions to the server-side instruction executor 63.
When it comes to what makes good partitions of instructions, the global intelligence layer 100 should be executed on the computing cluster, and the driver layer 130 should be executed on the robot. It is more of a matter of how to partition the functional modules inside the local intelligence layer 110 and the platform layer 120. In our preferred embodiment, the safety monitoring module 119 is assigned to the robot's instruction executor 53. It is because the operations initiated by the safety monitoring module 119 are probably time-critical. It is also because the safety monitoring function should be local to the robot in case of network failure. The safety monitoring module 119 usually depends on the sensing module 124, the steering module 126, the kinematics module 128, and perhaps also on the image processing module 122. The image processing module 122 may consist of many algorithms, and some algorithms may be computationally intensive and not required by the safety monitoring module 119. Therefore, the partition of instructions is really about where to execute the local intelligence layer modules (except the safety monitoring module) and, part or whole of, the image processing module 122.
Some image processing algorithms may be computationally intensive, but it may also be bandwidth intensive to transfer the images from the robot to the computing cluster 24. The network bandwidth capacity may depend on the location of the robot, i.e., whether the Internet 30 is involved. Sometimes, it is more desirable for the robot to compress the images and send them to the computing cluster 24 and, sometimes, more desirable to extract the features from the images and send the features information to the computing cluster 24.
Implementing the functional modules in the global intelligence layer 100, the local intelligence layer 110, and the platform layer 120 in a platform independent programming language also facilitates the partitioning the instructions. In our preferred embodiment, we use Java as the programming language. The instruction executor 53 and the server-side instruction executor 63 encompass the JVMs (Java Virtual Machine).
In partitioning the instructions, the performance monitor 52 plays an important role, as illustrated in
That said, the performance monitor 52 may estimate the instruction executor processing capacity and the network capacity by running engineered test instructions, as opposed to the instructions of the robot system software stack. Since the instruction executor processing capacity for a robot may seldom change, the performance monitor 52 may memorize about the instruction executor processing performance data and focus on assessing the run-time network performance.
The performance monitor 52 also collects performance data including network latency and bandwidth of the communications between the robot and the computing cluster 24 and also network latency and bandwidth of tele-operator sessions from the tele-operators' computers to the robot. Also, the performance monitor 52 exchanges keep-alives with the computing cluster 24 and synchronized timing information.
The synchronized timing information enables all robots in the system to synchronize their clocks. When images and sensory data are communicated across the robots and the computing cluster 24, the image processing module 122 and the sensing module 124 can correctly time-stamp the images and sensory data so that the instructions on the server-side instruction executors may be able to correlate the images and sensory data for correct analysis.
The keep-alives indicate the health of the communication channels. When a robot has lost contact with the computing cluster 24, the safety monitoring module 119 may pause the robot temporarily.
The network latency and bandwidth information, in addition to the processing capacity performance data, is useful for determining how the instructions should be partitioned. It is also useful for the image processing module 122 to scale down or up the degree of analysis. For example, it is used to control the degree of image compression, the rate of capturing images, and the image resolution so that the bandwidth required by the compressed images may fit into the networking infrastructure constraints.
Also, we may use the tele-operator session latency information to control the granularity of the mechanical movements, e.g., movements produced by the steering module 126 and the kinematics module 128. In a long latency environment, the tele-operator may not be able to observe the effect of the tele-operator's action, so it would be safer to slow down the mechanical movements; in a short latency environment, the tele-operator is able to adjust the effect of faster mechanical movements.
In our preferred embodiment, we have all communications from the robots to the computing cluster 24 all through direct peer-to-peer connections. Considering the presence of tele-operators, the partitioning of instructions has another aspect to consider. The deciding factors of what computation to be performed on the robots are application processing delay constraints and network bandwidth constraints in addition to computational complexity. For example, a robot is to capture video images and send them to a tele-operator. If the robot captures some video frames and sends them to the computing cluster 24 which in turns compresses the video frames and sends the compressed video stream to the tele-operator, that procedure may consume too much network bandwidth. It may be desirable that the robot first compresses the video frames to save some network bandwidth. Assuming that, if the computing cluster 24 processes the compressed video stream from the robots before relaying the compressed video stream to the tele-operators, that procedure may introduce a significant processing delay. That is a concern as the packet delay should be less than 250 ms to achieve good audio quality in real-time video delivery. Therefore, we suggest that the robots that possess video capturing capability should be capable of performing a decent video compression. Also, the corporate network infrastructure should support making a copy of the compressed video stream to the computing cluster 24 if needed. That allows the computing cluster 24 to intercept the compressed video stream, decompress it, and perform other useful analysis and decisions.
In yet another embodiment, the robots may communicate the computing cluster 24 through an intermediate server. For example, the video captured by the robots may go through a video server on the Internet, e.g., a Google GTalk server, before reaching the computing cluster 24 and the tele-operators. That embodiment has an advantage that the computing cluster 24 does not need to be bothered with video stream distribution and may focus on the robot system software stack.
It is understandable to those skilled in the art that all or part of the preceding embodiments can be implemented by hardware instructed by a program. The program may be stored in a computer-readable storage medium and be executed by a processor. The storage medium may be a magnetic disk, compact disk, Read-Only Memory (ROM), Random Access Memory (RAM), and so on.
The embodiments described above are illustrative examples and it should not be construed that the present invention is limited to these particular embodiments. Thus, various changes and modifications may be effected by one skilled in the art without departing from the spirit or scope of the invention as defined in the appended claims.
| Number | Date | Country | |
|---|---|---|---|
| 61542808 | Oct 2011 | US |
