Patent Application
20150296324
This invention relates generally to interacting with equipment using a mobile computing device, and more particularly to interacting using computer vision and augmented reality.
When a user wishes to interact with equipment using a mobile device, the capabilities and functions of that device may not be known ahead of time by the mobile device, which would limit the ability for the mobile device, and equipment to functionally interact. On the other hand, if the mobile device does know the capabilities of the equipment, the mobile device can readily generate the right commands and interpret the data received from the equipment. For the operations to be fast, the data needs to reside locally on the mobile device. However, it is unlikely that the data always exist on the device, given the innumerable varieties of equipment. In which case, the data need to be acquired from a source.
One possible source is a database server connected to the mobile device by a network. However, for some potential applications, such networks are not always available. This means that the user needs to know in advance every piece of equipment that the user intends to interact with in the future, and download the data ahead of time, assuming that the network is available and the mobile device is permitted to access the network, which, for security reasons, is not always the case.
In some applications, it would be useful to facilitate interactions between a mobile device and equipment by using computer vision (CV) techniques to overlay augmented reality (AR) content, such as graphics, on the device. In this application it is necessary that the device can recognize and/or segment the equipment in acquired images of the equipment based on sensory inputs from, e.g., a two-dimensional camera, a three-dimensional depth scanner, and the like. For the device to be able to perform such computer vision operations, certain data that uniquely identifies the equipment or its parts must exist. Often, the data are highly equipment specific and not easily decipherable by the user. Data to facilitate recognition by CV techniques and subsequent interaction can be achieved as described above, but this may be problematic for the described reasons.
Another potential source for the data required to enable successful CV/AR interaction is for the user to use the device to generate the data. This is also problematic as it is often difficult to acquire the data correctly using sensors of the device so that the interaction can be performed in a reliable manner, which can significantly increase cost and time. In fact, it is often necessary for an expert to perform the data generation. Furthermore, in that case, each device would contain a different copy of the data, which may lead to each devices behaving differently when performing the interaction.
Yet another method, is to place tags, e.g., quick response (QR) codes, on the equipment and its parts. Typically such tags only identify the equipment associated with the tags, which means that the tag is missing specific information about operational characteristics of the equipment. Entering the information manually into the device is time consuming. In addition, such tags can only be viewed accurately from certain angles and are prone to becoming torn or dirty so that the tags become unreadable.
Modern facilities, such as factories, often contain many pieces of large advanced manufacturing equipment; NC milling machines, laser cutters, and robots, for example, are commonplace in today's factories. Maintenance engineers are required to ensure that the factory achieves as much up-time as possible, and their job would greatly benefit from the ability to interact with the equipment in an easy and intuitive manner; they may, for instance, wish to receive detailed machine diagnostic information, or manipulate the machine's actuators to a safe position.
One possible solution to enable such interaction is to supply each piece of industrial equipment with its own interface (i.e., display and input), but this significantly adds cost to each piece of equipment sold; furthermore, some pieces of equipment may be too small or hidden from direct view (e.g., programmable logic controllers). Now that mobile devices, such as tablets, smart phones, and augmented reality glasses, are ubiquitous, the engineer may be supplied with a generic mobile device of such a type that can interact with all pieces of equipment that they might service. However, there remains the problem of how the generic mobile device is able to interact with such a wide variety of equipment.
A tablet is a mobile computer with a display, circuitry, memory, and a battery in a single unit. The tablet can be equipped with sensors, including a camera, microphone, accelerometer and touch sensitive screen, as well as a wireless transceiver to communicate with the equipment and to access networks.
In order for successful interaction to take place, the mobile device may require specific knowledge of one or more of the following: machine functions and returnable, data, called and interpreted by an application programming interface (API); a displayable user-interface that allows the operator easily manipulate the machine or request specific data; descriptive data of the equipment that will allow the operator's device to identify the equipment from incoming sensor data. It is assumed that the equipment is within visible range al the user of the tablet and the camera to make the interaction effective.
One particularly useful application is to provide an interface or diagnostic data directly on top of a live image of the equipment. When a user wishes to interact with a piece of equipment or a machine using a mobile device, via computer vision (CV) and an augmented reality (AR) content, the mobile device needs a way to recognize the equipment, or parts of the equipment, and to extract a relative pose of the device with respect to the equipment. To do so, certain data need to exist on the device to enable the mobile device to interact with the equipment in a reliable manner.
Data to support CV and AR could be determined locally, but this can significantly increase the equipment setup time. In addition, the data can be inaccurate due to scanning techniques, changes in equipment settings, the environment, and many other factors. Alternatively, the data can be predetermined in a conventional manner with enhanced generality to adapt to environmental differences. The predetermined data, such as image descriptors for a set of poses and conditions, can come from a networked database, but this is not always possible, as wireless networks have security and reliability problems in industrial environments.
The embodiments of the invention overcome the limitations of prior art methods by storing the predetermined data on the equipment, and communicating the data to the mobile device via short-range communication technologies, such as near field communication (NFC), Bluetooth® or Fi™. Then, the user of the mobile device can interact with the equipment to transfer the predetermined data to the device. The mobile device can also include software that enables the user to interact with the equipment via the mobile device.
Such a system has many advantages. First, a variety of mobile devices can be used to interact with different equipment by specifying to the equipment the configuration and capabilities of the mobile device, to which the equipment responds with the correct adapted data for performing functional interaction with the equipment. Interaction in such a case can include operating various features of the machine, e.g., switching the machine on/off, manipulating an actuator, changing machine parameters, etc, or the interaction can involve receiving various important pieces of operating data from the equipment and displaying these to the user so that the user can monitor the activity of the equipment, e.g., progress in completing a specific task.
Yet another possibility for interaction is that the user connects the mobile device wirelessly to multiple pieces of equipment, retrieves interaction data from the multiple pieces of equipment, and uses the mobile device to coordinate the interaction between the device and the pieces of equipment, and between the pieces of equipment themselves, where the specific functions that specify interaction between the equipment conic from one or multiple pieces.
In some embodiments, the CV data can be generated by an expert, once for each sensor modality, which means minimal setup time and maximal accuracy and reliability for the user. As an advantage, the user does not need to know ahead of time what pieces of equipment the user will interact with. As another advantage, there is no need for additional equipment, other than the mobile device.
The interaction can be made secure by only storing the data received from the equipment in volatile memory and restricting the device to only receive data supplied by the equipment for a particular interaction.
As shown in
The mobile device 130 also includes a transceiver 131, a processor 132, a touch sensitive display 133, memory 134 (e.g., the SD memory card) and sensors 135, e.g., a camera, a microphone, and an accelerometer. In order to interact with equipment effectively and efficiently, the equipment is within a visible range 140 of the user of the tables, and the camera.
In one embodiment, a manufacturer of equipment can supply a single “master” application (MAPP) for each potential mobile device. This MAPP contains all the necessary functionality to search for equipment built by the manufacturer with this type of capability, establish a connection, and request all the necessary data. It should be noted that the request of data may be implicit in the connection between the device and equipment. This data can comprise of many different pieces that are used to facilitate the interaction between the device and equipment, including CV data (e.g., image descriptors or 3D CAD models), AR overlays, application programmer interfaces (APIs), among others.
Then, the mobile device can be used operate 250 the equipment. Some of the data can be used for a generic controlling application. Other data can be equipment specific.
As shown in
As shown in
One potential scenario in which this type of application might be useful is for a CNC milling machine to signal, via the mobile device, to a mobile robot that the milling process is completed, and that the mobile robot, using CV and location data supplied via the mobile device, can locate and retrieve the finished workpiece.
Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention. Therefore, it is the object of the appended claims to rover all such variations and modifications as come within the true spirit and scope of the invention.
