The invention relates to a method of configuring a visualization device for a machine zone in which at least one sensor is arranged and to a template having an object marker for the method.
A plurality of sensors are used in a machine zone, for example a robot cell, to avoid accidents and to monitor and support the work procedures. It is exceedingly helpful in the setting up and servicing of the robot cell and the sensors to obtain information on the sensors that is not directly recognizable with the naked eye. It is known in principle to visualize such information on a mobile end device such as a smartphone. A superposition of a camera image with the additional information (augmented reality) is particularly descriptive.
The requirement for a correct display of the sensor information is a knowledge of the exact position of the sensor with respect to the camera of the mobile end device. There are approaches in the prior art to recognize and localize physical objects on which information is to be superposed using a 3D model. Not only 3D models of the physical objects are required for this purpose; the comparison between 3D models is additionally complex and prone to error, above all when the object is partly hidden, as is frequently the case.
Another possibility comprises attaching a respective marker to the object. Such a marker must be prepared individually for every object and has to be replaced together with the object or prepared again in the event of a defect. Accessibility during the attachment of the marker is already not always ensured with sensors that are installed in the system and that, for example, have an additional protective housing. The marker then also has to be readable for the visualization and for this purpose has to be recognizable completely and at a sufficient size from the location of the mobile end device despite aging phenomena such as fading or damage. Finally, the correct localization on the basis of such an optical marker represents a challenge that has not yet been satisfactorily mastered.
It is therefore the object of the invention to improve the visualization of a machine zone.
This object is satisfied by a method of configuring a visualization device in accordance with the respective independent claim. The visualization device is preferably a mobile end device such as a tablet, a smartphone, or VR glasses. Information on sensors in a machine zone should be presented by them in operation after a completed configuration. The machine zone is initially thought of as a robot cell or an integrated network on an automated guided vehicle (AGV) or an automated guided container (AGC), but the term can also be understood more broadly and designates a zone in which dynamics by a machine are to be expected at least at times, that is, for example, also a conveyor belt or a grade crossing. There is consequently at least one sensor in the machine zone that, for example, secures a machine and in the event of a risk of an accident provides for its switching into a safe state in good time or supports the machine in its work, for instance monitors work procedures or recognizes workpieces or tools or checks work results. Optoelectronic sensors are particularly suitable for such work. The method in accordance with the invention is equally suitable for visualizing information on different objects than sensors.
The invention starts from the basic idea of localizing the at least one sensor and possibly further objects on which information is to be visualized while using markers that are recorded, detected, or scanned by a detection device. The detection device is preferably likewise a mobile end device corresponding to the visualization device and can thus be identical, but does not have to be. In accordance with the invention reference markers, on the one hand, and object markers, on the other hand, are provided that are attached in the machine zone or to the sensor and are then detected and put into relationship with one another.
Reference markers are attached in a fixed position with respect to the reference system of the visualization device, for instance on the hall floor or on the frame of a vehicle, and their positions thereby become a reference location. There is initially no restriction as to what can be a reference locations; the fitter decides this by attaching the reference marker. The fitter admittedly manages with fewer reference markers that are easily visible from the key points for the configuration of the machine zone by a skillful choice, but this is not essential to the invention.
Object markers are in contrast attached to the sensor or to each of the sensors if there are a plurality of sensors. They are not called sensor markers because there can still be further objects, for instance controllers or machine parts on which information is also to be visualized and which can be integrated in the visualization by means of object markers in the same way as sensors.
At least two markers are now respectively detected by the detection device. At this point, marker is the umbrella term for references markers or object markers, that is at least two reference markers, two object markers or one reference marker and object marker each are detected. These markers or the reference locations and sensors or objects represented by them are linked to one another, and indeed abstractly and/or geometrically. Abstractly means that only a simple relationship between the markers is provided that is called a proximity relationship. If two markers are neighbors in accordance with the relationship, they are, however not necessarily also neighbors in the sensor of critical geometry, for instance in the relationship “was measured with respect to”. The geometrical link preferably represents a spatial relationship in three dimensions, that is, for example, a transformation rule or a part from the one marker to the other marker. A link structure is thus produced with which the sensors and objects are localized with respect to the reference location.
The invention has the advantage that data and information on the sensors and objects can be visualized everywhere in the machine zone and also in its environment. The correct relationship is provided here; the data and information are reliably and easily recognizably associated with the sensors and the objects. A mobile end device without any special requirements such as a tablet or a smartphone is sufficient for the measurement and also for the later visualization. The putting into operation, servicing, and diagnosis of the machine zone or of the machines and sensors therein is thereby substantially simplified. Time and costs are saved and some errors are avoided.
The fitter is preferably automatically guided through the configuration by the detection device that respectively prompts to detect markers and automatically carries out their linking.
The detection of the markers preferably begins with a reference marker. All the abstract and geometrical links can then be linked at its reference location. A coordinate origin can naturally later be displaced with respect to this reference location. A coordinate system is preferably defined relative to the machine zone and not absolutely, in particular in the case of a moving machine zone such as with a vehicle All the sensors and other objects of interest are preferably provided with object markers and have reference markers attached so that at least one reference marker is easily visible from all relative observation locations or where they are possibly needed to bridge larger distances.
Markers are preferably detected and linked to one another pair-wise until the markers attached in the machine zone have been detected. This simplifies the handling and the insertion into the link structure. Long pairs of markers are thus detected until all the existing markers have been considered at least once. This is as a rule the responsibility of the fitter. If a marker is forgotten, the later visualization can have gaps and can then be corrected. In general, an automatic check whether all the markers have been detected would also be conceivable by recording an overview image of the machine zone and by counting the markers or by communication with the sensor network in the machine zone.
A check is preferably made whether one of the two markers respectively detected pair-wise had already been previously detected. The desired behavior of the fitter would be to respectively detect an already known marker and then a new marker. All the already detected markers are at least indirectly linked with one another in this manner and the link structure is successively expanded by a respective one marker. If both newly detected markers are still unknown, the processing can be rejected and the configuration can be continued after the detection of at least one other marker. Alternatively, a further link structure is produced from the two still unknown markers. If the same marker then later appears in a plurality of link structures, they can be connected to one another.
The detection device preferably prompts to first detect the one marker and then the other marker and subsequently shows the generated link between the two markers to have it acknowledged. The geometrical link and not the abstract link is preferably indicated, for example in an image of both markers with the calculated connection line therebetween. The fitter can recognize if the geometrical link is faulty. The calculation can then be repeated with another process or the fitter can be prompted to scan a different marker or to attach a new reference marker.
The abstract linking of the markers preferably takes place in the form of a graph. The nodes of the graph are the already detected and linked markers or the reference locations and objects or sensors represented by them. The edges are per se already the abstract proximity relationship; in addition to the edges, the geometrical transformation or a geometrical path from the one node or marker to the other node or marker can be stored.
The graph is preferably arranged or rearranged such that adjacent nodes in the graph are also geometrically adjacent. This can already be done on the pair-wise reading of markers or subsequent thereto. It is not ensured that the fitter respectively reads two markers adjacent in the geometrical sense. A discrepancy that thereby arises between the order in the graph and the geometrical order is resolved by a correction of the arrangement or rearrangement.
The geometrical link of the markers preferably takes place by evaluating a value and/or a format of the detected markers. This is a relatively simple method to determine geometrical relationships. With markers of a known shape and size, a conclusion can be drawn from the detected shape and size on the distance and the perspective of the detection and thus of the markers relative to the detection device. This is, however, only an example for a localization method, a large number of other localization methods are known, in particular a distance measurement by a 3D process or time of light measurement.
The geometrical linking of the markers preferably takes place in that the movement of the detection device between the detections of different markers is monitored, in that at least two markers are detected at the same time, or in that detections are evaluated during the alignment of the detection apparatus from the one marker to the other marker. If the fitter detects two markers after one another and these two markers are then localized relative to the detection device, an error can result due to interim movements and a rotation of the detection device on the conclusion of the relative arrangement of the two markers to one another. This can be eliminated by calculation in that the movement of the detection apparatus is detected, for example using an inertial measurement unit (IMU). A conclusion can also be drawn on the movement by intermediate detections during the alignment of the one marker with the other marker, for instance using a method of optical flow or in that intermediate images are joined to one another. The situation is comparatively simple when both markers are detected in the same image; the discussed error then does not arise. The fitter can also take care to move the detection device as little as possible between the two markers and to thereby keep said error small so that it does not have to be corrected.
A reference location is preferably associated with a detected reference marker and/or a detected object marker has the sensor represented by it associated with it. The reference location can be a geometrical position, for instance in the form of coordinates, but also only a point in the link structure, in particular a node in the graph. Instead of or in addition to the object marker, the sensor represented by it can be integrated abstractly and/or geometrically into the link structure. Additional configuration steps are also conceivable by the fitter here, for instance the input of a speaking name for the later visualization such as “laser scanner for protected field at material lock”.
An object marker is preferably arranged on a template with a mount for attachment to the sensor. Object markers are here not arranged as conventionally directly on the object or sensor, but rather at a preferably reusable template that is then attached to the sensor with the aid of the mount. This does not only provide a particularly simple possibility of reliably attaching object markers and removing them again. The template positions the object marker on the sensor in a defined manner, which facilitates the further processing.
Information on the position of the sensor relative to the object marker is preferably encoded in the object marker. Thanks to the template, the relative position of the object marker with respect to the actual sensor is fixed and known in advance, for example its optical center. The offset between the object marker and the actual sensor position can thus be reliably and simply corrected. The corresponding information can be encoded directly as a transformation or as relative coordinates or this is added subsequently from an identity of the object marker.
In a preferred further development, a method of visualizing a machine zone using a visualization device configured in accordance with the invention is provided in which a reference marker is first detected and visual sensor information from the environment of the reference marker is then presented. The user stands at a position in the machine zone or in its proximity and aligns the visualization device on a reference marker with reference to which the visualization device is oriented and determines which sensors are located in the environment. Virtual sensor information on these sensors is then presented at the correct location. The object markers have preferably been removed again at this point in time at which the configuration has ended, they are at least no longer needed.
Only sensor information from sensors that are neighbors of the detected reference marker in accordance with the abstract link is preferably presented. This is a possibility of selecting the presumably relevant and visible sensors from the current position of the visualization device. Direct neighbors are preferably selected, but 2-neighbors and the like as well as a restriction to a maximum distance are also conceivable.
The sensor information is preferably presented as a superposition with a live image. The virtual information is thus superposed on the real image and a particularly intuitive presentation is thus achieved (augmented reality).
The sensor information to be presented is preferably read by the sensor, by a controller connected to the sensor, and/or from a database for sensors. Some static information can already be detected via the object marker or a configuration input on its reading. The sensor itself or a higher ranking system to which the sensor is connected can deliver further information. A database with sensor data is conceivable as a further source.
The sensor information preferably comprises at least one of the following pieces of information: name of the sensor, address of the sensor, type of the sensor, a graphical model of the sensor, an alignment and/or a detection zone of the sensor, in particular a scanning plane, a field of view (FOV), a protected field or a region of interest, a sensor parameter such as its temperature or configuration, and/or measurement data of the sensor in any desired preparation as raw data, numerals images, point clouds, grid models of the objects detected by the sensor, and the like.
A template in accordance with the invention has an object marker for an embodiment of the configuration method in accordance with the invention, a mount suitable for a sensor, and a piece of information encoded in the object marker with which a location of the object marker is converted into a location of the sensor. The object can be attached to the sensor very easily in a defined manner via the template. It is no longer necessary to generate individual object markers per object. A template having an object marker suitable for the sensor type can rather be used multiple times.
The invention will be explained in more detail in the following also with respect to further features and advantages by way of example with reference to embodiments and to the enclosed drawing. The Figures of the drawing show in:
The following sensors 18 are installed in the machine zone 10 of
The invention does not look into the design of a robot cell or more generally of a machine zone 10 and the selection and mounting of the required sensors 10. It should rather be of assistance in the configuration of the sensors 18, in particular as part of the putting into operation, diagnosis or servicing, and should provide a visualization of the machine zone 10 together with additional information on the sensors 18 or of the sensors 18 for this. This naturally does not preclude the fitter determining the need for additional sensor 18 or a different arrangement of the sensors 18 with reference to the visualization.
A mount 26 is furthermore provided at the template 22 that is adapted to a specific sensor type. The template 22 can be attached to a sensor 18 of the matching sensor type in a well-defined and reliable manner with the aid of the mount 26, independently of the accessibility and size of the sensor 18. The object marker 24 is then located in a known relative position to the sensor 18 thanks to the template 22. The transformation from the location of the object marker 24 to the location of the sensor 18 is encoded in the object marker 24, either directly, for example in the form of relative coordinates, or indirectly in that a piece of identity information of the object marker 24 in a database or the like is linked to the associated relative coordinates. Thanks to the template 22 and the now known offset between the object marker 24 and the sensor 18 caused by the template 22, the sensor 18 is shown at the correct location and not at that of the object marker 24, for instance, later in the visualization. The template 22 is only required during the configuration of the visualization and can therefore be used multiple times.
One object marker 24 is attached per object to be localized, that is per sensor 18, but also per machine part, controller 20, or the like. This is preferably done via templates 22 and alternatively directly on the sensor 18 or another object. Object markers 24 are preferably not prepared individually for a sensor 18, but for a sensor type. In the example of
At least one reference marker 28 is attached in the machine zone 10 in addition to the object markers 24. The reference markers 28 can be positioned as desired by the fitter. They contain a unique code, for example a 32 digit identification number (universally unique identification, UUID) to preclude confusion with other markers in the machine zone 10. The reference markers 28 serve as reference points. It is later determined in the visualization with reference to a reference marker 28 read from the proximity where the origin of the visualization is and which sensors 18 are in the environment.
In a step S1, a first marker 24, 28 is read, the fitter is therefore prompted to direct the detection device to a marker 24, 28 to be read and to trigger an image recording of a camera, for example. It is of advantage at the start of the configuration for a reference marker 28 to be read first. This then forms the reference point or point of origin. Alternatively, however, the anchoring can take place at a later point in time after a reference marker 28 has been detected.
In a step S2, a second marker 24, 28 is read. A pair of two markers has thus then been read, and indeed by choice of the fitter a pair of two object markers 24, of an object marker 24 and a reference marker 28, or of two reference markers 28 As already stated with respect to step S1, the fitter can be prompted at the first pair to choose at least one reference marker 28 so that there is a point of origin from the start. In later iterations, during the reading of further pairs, the detection device can require that a respective one of the read markers 24, 28 is already known to successively expand the link structure of the markers 24, 28 read during the configuration. Alternatively, two or even more initially separate link structures are generated that can then be joined together as soon as they overlap one another in at least one marker 24, 28 that has become known.
In a step S3, a relationship between the two read markers 24, 28 is automatically determined. There is already an abstract relationship in that the two markers 24, 28 are read together and are now automatically referenced to one another. A graph can, for example, be produced with this relationship that will be explained below with reference to
In an optional step S4, the geometrical relationship is shown to have it acknowledged by the fitter. This is illustrated in
A check is made in a step S5 whether one of the read markers 24, 28 is an object marker 24. A sensor 18 or object is then associated with it in a step S6. In this process, user inputs can also optionally take place with which, for example, the sensor 18 or the object is provided with a name of its own. A reference location, for example a coordinate in a coordinate system, can be associated with reference markers 28. The first read reference marker 28 per link structure preferably fixes the coordinate system, with an origin being able to be displaced as desired. If a plurality of link structures are combined with one another once a marker 24, 28 has appeared in both, the coordinate systems are also aligned.
In a step S7, the configuration ends if all the markers 24, 28 have been detected once. Otherwise, at step S1, a new pair of markers 24, 28 is detected and processed in a further iteration. It is preferably part of the responsibility of the fitter to take all the markers 24, 28 into account. It is, however, also conceivable that the detection device has knowledge of the total number of markers 24, 28, for example via a specification, from an overview image of the machine zone 10 with all the markers 24, 28, or by communication with the sensors 18 or with the controller 20.
At the end of this configuration, the geometrical relationships between all the markers 24, 28 are known and thus all the sensors 18 and other objects such as the controller 20 with object markers 24 are localized. The object markers 24 or the templates 22 can now be removed.
After a completed configuration, sensor data of the sensors 18 can now be visualized. A mobile end device that can, but does not have to, correspond to the detection device of the configuration, for example a smartphone, a tablet, or VR glasses, in turn serves as the visualization device. The user scans a reference marker 28 in his proximity. Sensors 18 and a possible further object such as the controller 20 are localized in the environment of the scanned reference marker 28 using the graph, in particular the direct or indirect neighbors of the scanned reference marker 28 in the graph. The required geometrical transformations are stored from the configuration in the edges of the graph. The sensor information or object information can thus be visualized at the correct location. This preferably has a camera image superposed (augmented reality).
A large variety of visualized information is conceivable. In addition to the name and type of a sensor 18, its configuration can be illustrated, for example a protected field of a laser scanner can be displayed, an operating parameter such as the temperature of the sensor 18 can be displayed, or measurement data of the sensor 18 are visualized. In the case of other objects such as the controller 20, the data are translated into a generic description and are provided with an associated visualization. It can be loaded in dependence on the kind of the visualization.
The invention has up to now been described for the example of a robot cell as the machine zone 10. The concept can be transferred to a vehicle, preferably an automated vehicle. Reference markers 28 and sensors 18 are located on the vehicle and thus in a fixed geometrical relationship to one another in the reference system of the vehicle so that the movement of the vehicle with respect to the external environment does not play a role and the configuration in accordance with the invention and the visualization remains comparable with a stationary machine zone 10. A machine zone 10 is, however, not restricted either to a robot cell or to a vehicle, but rather describes a zone in which at least one sensor 18 is located and in which interventions by a machine take place at least at times for which there are innumerable further examples such as a conveyor belt or also a railroad crossing.
In an expansion, a link to CAD information of the machine zone 10 can take place that as a rule anyway exists in the case of a robot cell or of a vehicle. Markers 24, 28 can thus be localized even more exactly or an optimum number and position of reference markers 28 can also be planned. 3D models can additionally be used to localize the sensors 18 themselves in addition to the markers 24, 28.
Number | Date | Country | Kind |
---|---|---|---|
20198357.4 | Sep 2020 | EP | regional |