Patent Application
20170169291
1. Field of the Invention
The invention relates to a method for automatically creating a data set characterizing a technical drawing having symbols and lines connecting the symbols, from the drawing.
2. Description of the Related Art
The description of complex associations and processes in technical systems and equipment can be represented in many cases and uses by technical drawings or diagrams. Drawings of this type are to be found, inter alia, in function charts for the description and creation of automation functions, such as are described in WO 2013/092654 A1. Typical properties of such drawings are that there are particular symbols which are linked to one another by lines.
Due to the scope of the documents to be processed and the error-proneness of manual processing, significant time and cost savings can be expected from an automated, digitalized method for the processing of drawings. Today, scanning and vectorization of diagrams in a sufficiently high quality, including text recognition (OCR) is possible without difficulty. For this purpose, the diagrams are scanned so that they are present as raster graphics and are converted using modern software tools into vector information and text.
Starting from this vector information (following optional pre-processing) symbol candidates can be filtered out from all the lines (e.g., using rules or searches for rectangles; see e.g. Y. Yu, A. Samal, S. C. Seth: A System for Recognizing a Large Class of Engineering Drawings. IEEE Trans on PAMI19:8, 868-890 (1997) and S. Adam, J. M. Ogier, C. Cariou, R. Mullot, J. Labiche, J. Gardes: Symbol and character recognition: application to engineering drawings. IJDAR 3, 89-101 (2000)) and connecting lines between the symbol candidates can be identified. Subsequently, in this way, a representation is obtained as a graph (in the mathematical sense) which describes the symbols and their connections to one another, specifically for each processed page. This graph, the node of which describes the symbols, forms a data set that can be used for further processing. The lines between the symbols are stored in the data set as connections with end points, where the end points are associated with the respective nodes that are connected by the line.
Using the previously created symbol library, the symbol candidates are then classified, i.e., associated with symbol types stored in the library. For this purpose, corresponding symbol data, in particular graphical identification features, of each symbol are stored in the symbol library. At this point, the recognition and interpretation are essentially concluded. It is further known to also perform a manual post-processing that serves for the correction of errors in the classification, i.e., the association of symbol types with the respective nodes.
A disadvantage in the conventional methods, however, is that although the problem of the computer-internal representation of the documents is solved, associations between documents and meta-information remain unconsidered. Particularly in the use described in WO 2013/092654 A1, it is the recognition and replacement of particular partial structures that is concerned. This requires not only the mere recognition of symbols and connections for the purpose of digitization, but also their semantic interpretation. If such information is required in a particular application, conventionally it must be input manually, which is very time-consuming and also fault-prone.
It is therefore an object of the invention to provide a method with which semantic information and relationships can be extracted automatically from vectorized data of a technical drawing and is thus available for further processing.
This and other objects and advantages are achieved in accordance with the invention by a method of a) scanning the technical drawing, b) identifying the symbols in the technical drawing and storing nodes representing each symbol in the data set, c) identifying lines, each connecting at least two symbols in the technical drawing and storing connections representing each line in the data set, at least two end points being associated with each connection and one of the nodes which represent the symbols connected by the respective line being associated with each end point, d) providing a symbol library having a plurality of symbol types and symbol data associated with each symbol type, and e) associating exactly one symbol type from the symbol library with each node at least on the basis of the symbol data and storing the associated symbol type for the respective node in the data set, where step d) comprises storing a plurality of junction points with junction point data associated with each junction point for a number of the symbol types in the symbol library, and step e) comprises associating precisely one respective junction point with a number of the end points at least based on the junction point data and the storing the respective associated junction point for the respective end point in the data set.
The invention proceeds from the consideration that, although the problem of digitally recording existing documents that describe particular processes and associations in diagram form is already solved with known approaches, many actually semantically available and interpretable items of information remain unconsidered by the methods described in the prior art. In particular, although the connections or their arrangement between the individual symbols are detected, their semantic significance is not. A start should therefore be made precisely with the classification and the tracing of connections. This can be achieved, in particular, in that as early as during the creation of the symbol library, additional information is given, i.e., not only the information as to which symbol is concerned and how it can be recognized, but also further information items that relate to the possible junction points of the symbols or the components, method steps or similar, which are represented by the symbol. For each symbol, therefore, junction points are defined for which, for example, identifying junction point data are stored. The association of the located connections or their end points to the symbols is then further specified as follows: the end points are associated not only with the respective node, but with a particular junction point of the respective node or symbol. In this way, with the storage of corresponding information items in the junction point data in the symbol library, it can also be determined from the data set which connections carry which information or meaning. The data set thus also contains semantic information items that can be used during the subsequent further processing. Furthermore, the junction point data and the junction points form further information items that can be used during the determination of the correct symbol type for a symbol and thereby improve the accuracy of the recognition.
In an advantageous embodiment, the junction point data comprises a junction point type. Depending on the use, it can be indicated here whether a junction point is an entry or an exit, i.e., permits an incoming or an outgoing connection. More generally, a junction point type could also be stored which allows only one directed connection. A further junction point type is, for example, a negation. Combinations of junction point types are also herein conceivable, i.e., a plurality of types are assigned to one junction point. Further advantageously stored herein, for each junction point type, is with which junction point types the respective junction point type can form common end points of a connection. For example, for the connection type “entry”, it could be stored that, in a connection with one end point of which it is associated, the other end point must be associated with an exit. These information items can also be used in the recognition of the symbol types and thus enable a better and more accurate association.
For each of the different junction point types, identification properties are also advantageously stored in the symbol library. This serves to associate the respective end points of the found connections to the correct junction points during the classification of the nodes. For this purpose, for each junction point type, if required also specifically for each symbol type in the library, at least one property is defined, on the basis of which the junction point type can be identified. Depending on the use, this can be, for example, the identification based on the position of the respective connection in relation to a reference point of the symbolic representation, for example, relative to the top left corner of the corresponding envelope surface (bounding box) of the symbol. Alternatively or additionally, the junction point type can also be identified based on a text within a rectangle to be defined relative to the junction point position.
In a further advantageous embodiment of the method, the junction point data contains information that indicates whether the respective junction point must be associated with an end point. This information can also be used for the recognition of the symbol types, i.e., in the classification of the nodes in that specifically the symbol type selected from the library is associated with a node when and only when the junction point is found through the aforementioned information and is also connected.
The aforementioned step e), specifically the association of the junction points of the symbols with the end points of the connections, advantageously occurs serially, starting from a connection by tracing further connections adjoining the node of an end point of the connection. This means that the classification occurs successively along the connecting lines, which are continuously further traced. This enables the pre-setting of relevant properties of the junction points for particular application cases. With this, for example, only connections to junction points with these relevant properties can be taken into account, i.e., further traced. In this way, following a pass, a graph is obtained that contains only the desired structures and thus, under certain circumstances, is significantly smaller than the overall graph. In this way, a further processing of the extracted information items is substantially more efficient.
The symbol data herein advantageously comprises information that identifies whether connections extending beyond the respective symbol should be traced. In this way, limits can be also placed on the recognition and classification of the nodes, if particular regions of the technical drawing are not relevant for the desired application.
In a further alternative or additional embodiment of the method, the symbol data comprises a second representation of the symbol type, where the second representation comprises a plurality of nodes with respectively associated symbol types and connections to end points associated with the plurality of nodes. In other words, a symbol type can be identified as an assembled symbol, i.e., it is actually assembled from a plurality of also previously defined elementary symbols and connections under these elementary nodes and thus forms a subgraph which corresponds to the aforementioned second representation. The outwardly directed junction points that are not connected within the subgraph herein correspond to the junction points of the higher-level assembled symbol. If, during the classification, a symbol is encountered that is marked as an assembled symbol, it can immediately be replaced by its elementary symbols and connections, if this is desired. At this point, all the other symbol and junction point properties, for example, the previously defined end of the further tracing of connections, are also taken into account at a correspondingly marked symbol, etc.
Advantageously, the junction point data also comprises information that indicates whether the respective junction point for the use of the respective second representation must be associated with an end point. This means that the replacement of an assembled symbol by the corresponding subgraph is linked to the condition that one or more specified junction points must be connected. This enables an automatic error message to the user if this should not be the case.
In a further advantageous embodiment of the method, the junction point data contains information that indicates whether the respective junction point can be duplicated. Depending on the respective use, it can occur that a plurality of connecting lines converge at a junction point of a symbol. In order to represent this logically, the corresponding port can be duplicated, i.e., in the logical representation in the data set, an identical junction point is stored at the respective node, i.e., the existing junction point is duplicated and a respective end point of each of the converging connections is assigned to each of the junction points. If an information item is stored on whether such a duplication is permitted, the possibility exists here again of an automated error message to the user if the marking is not set and such a situation is found.
The method described also enables connections with open ends to be traced and formed. This refers to connections in the technical drawing that have at least one end that does not adjoin a symbol. Such connections typically occur if links are to be made over, for example, multiple pages of technical drawings. This is indicated, for example, by textual descriptions (example: an indication “A/02” is intended to convey that a connection to the identifier “A” on page 02 should be completed). The described step c), specifically identifying the connections, advantageously comprises for this purpose the identification of open lines proceeding from a symbol in the technical drawing, a connector being associated with the open end point of the representing connection in the data set, which is stored in the data set and represents a connection to a connector of a second technical drawing. Connectors are identified based on particular pre-set properties stored in the symbol library. If a connector of this type is found, it is included as a node in the overall graph in the data set. Found items of text information are associated with the connector.
The data set generated by the method is advantageously combined, based on the linking of the connectors, with the second data set representing the second technical drawing situated on the other page. In other words, connections between compatible connectors are (depending on the application) automatically closed (in the graph this means that the connector node is removed and a suitable connection, i.e., edge is inserted into the graph, where the end points of the new connection correspond to the respective end points of the connections linked at the connector). It is herein important that particular text information items can be passed on by the graph to all the junction points for which the information items are relevant (e.g., signal names across particular components and pages).
As already described, some of the junction point data of stored junction points (duplicatability, necessity of the connection on replacement by substructure) enable the recognition of errors in the library both in the association as well as in the technical drawing itself. In general, the fulfillment capability of conditions stored in the symbol data and/or the junction point data is advantageously tested and, in the case of non-fulfillability, an error is recognized. Herein, if required, either the classification of the recognized symbols with the correct symbol types can be improved, e.g., in that the user is requested to check manually, or (if a classification on the basis of previously specified criteria has been allocated a sufficient reliability) in this way an error in the technical drawing itself can also be identified.
It is also an object of the invention to provide a computer program product that can be loaded directly into the internal memory of a computer advantageously, which comprises software code portions with which the method described is performed when the computer program product executes on the computer.
A computer system advantageously comprises a scanner and an internal memory into which a computer program product of this type is loaded.
The advantages achieved with the invention consist particularly therein that through the determination and allocation not only of symbols and lines of a technical drawing, but through the prior definition of junction points in the library symbols and the allocation of the junction points to individual connections, markedly more semantic, interpretable information items are taken into account, which significantly improve the recognition accuracy and also the usage possibilities of the data set generated. Not only mere connections, but also their directions can herein be recognized, specifically independently of how it is handled in documents provided (if determinations are made only from the junction points and the corresponding junction point properties of the associated library symbols). The method also enables connections to be further traced across different pages and, if necessary, to be closed based on recognized connectors.
During the classification, reactions and actions can be performed depending on the properties defined in the symbol library, e.g., ending of the further tracing at particular symbols, replacement of assembled symbols by previously defined elementary symbols and automatic duplication of multiply-connected ports.
The method also enables improved automatic error recognition via the directed edges of the graph stored in the data set and the knowledge of which junction point types are connected, senseless connections can be found (e.g., connections between two incoming junction points). Whether errors of this type arise due to an erroneous recognition or from already faulty documents plays no part herein. In each case, a manual intervention is advisable. The relevant user can thus be informed herewith, with specification of the symbols involved, the document and also the junction points involved.
Junction point properties defined in the symbol library can be checked during classification and processing of the documents and, if necessary, suitable action can be taken. Examples thereof from WO 2013/092654 A1 described in the introduction are checking whether a junction point must be connected and whether a multiply-bound junction point may be duplicated automatically. In case of error, a suitable notification can be passed to the user with precise details of document, symbol and junction point.
Text information which is associated with connectors can be passed automatically by means of the graph which arises, including across a plurality of pages. This capability can be used in the application described in WO 2013/092654 A1 to pass signal names and further information items to remotely situated junction points.
Other objects and features of the present invention will become apparent from the following detailed description considered in conjunction with the accompanying drawings. It is to be understood, however, that the drawings are designed solely for purposes of illustration and not as a definition of the limits of the invention, for which reference should be made to the appended claims. It should be further understood that the drawings are not necessarily drawn to scale and that, unless otherwise indicated, they are merely intended to conceptually illustrate the structures and procedures described herein.
Exemplary embodiments of the invention will now be described in greater detail making reference to the drawings, in which:
The same components are provided with the same reference numbers in all the figures.
All the steps of the method shown in
In step a), firstly, the technical drawing is scanned. This includes all pages of the technical drawing, i.e., a plurality of pages of technical drawings are digitally recorded. For further processing, the scanned image files are vectorized, i.e., in each raster image generated, simple geometric objects are identified during the vectorization. This can occur with common technical variants known to persons skilled in the art, e.g., by edge detection, regions of the same or similar lightness or color, also known as posterization, are determined. The final result is coordinate data of graphic primitives in the technical drawing, i.e., lines, open or closed curves, points, etc.
From the vector graphic, following on from step b), the symbols in the technical drawing are identified, i.e., their number and position. In the data set to be generated that is to represent the technical drawing, representative nodes are now stored for each symbol. A data subset therefore forms for each symbol on the respective page.
In step c), which can also occur simultaneously with step b), lines connecting, respectively, at least two symbols in the technical drawing are identified and connections representing each line are stored in the data set. Herein, at least two end points are associated with each connection and one of the nodes that represent the symbols connected by the respective line is associated with each end point.
The steps a), b) and c) will now be described by reference to
The symbol s1 has a connection at the underside thereof identified with “OUT”, and the symbol s2 has a connection at the upper side thereof identified with “IN”. The connections are linked by the line 11. The symbol s2 has a further connection at the right-hand side thereof identified with “OUT”. Starting from this connection, a line 12 leads to a connection at the left-hand side of the symbol s3 and a further line 13 leads to a connection at the left-hand side of the symbol s4. The symbol 82 has a further unidentified connection at its left-hand side that is not connected.
It is an object of the method essentially to automatically extract the above-described properties of the technical drawing 2 and possibly further contextual information items from the technical drawing 2 itself.
The information so far extracted does not in any way correspond to the complete information extracted from the technical drawing 2 and is suitable for further use to only a limited extent. Therefore, making reference again to
In step d), a symbol library 8 is prepared, as shown partially and schematically in
However, the method is also to be more error-proof, to detect more information items from the technical drawing 2 and herein possibly also to recognize errors in the technical drawing 2 itself. For this purpose, the symbol library 8 is expanded with a plurality of information items. Initially, for each symbol type st1, st2, etc., junction points a1, a2, a3, etc. are defined. This means that for each symbol type st1, st2, etc., how many junction points a1, a2, a3, etc. it has is stored. For each of the junction points a1, a2, a3, etc., junction point data 12 is stored which contains further information items regarding the respective junction point a1, a2, a3, etc.
Initially, information items for the identification of the respective junction point a1, a2, a3, etc. are stored in the junction point data 12. For example, it can be stored that symbol type st2 has three junction points st2-a1, st2-a2 and st2-a3. st2-a1 is herein centrally arranged on the upper edge of the bounding box. Or, however, st2-a1 is always identified by means of a particular text at a particular separation and in a particular direction starting from its position, such as the text “IN” and “OUT” shown in
It can also be stored in the junction point data 12 whether a junction point a1, a2, a3 may be duplicated if a plurality of connections to it exist or whether this indicates an error. This can also be linked to further conditions. A junction point a1, a2, a3 can also be marked to the effect that it must necessarily be connected, particular partial structures proceeding therefrom are to be recognized or what significance it has in a format to be generated from the known documents.
With the junction point data 12 stored specifically for junction points a1, a2, a3, step e) can now be performed in relation to
The data set formed is shown by way of example in
In the data set 4, it is stored that for the symbol s2, the connection st2-a1 with line 11, the first duplicate of the connection st2-a1 with line 12, the second duplicate of the connection st2-a1 with line 13 and the connection st2-a3 is free. In addition, in the data 14, any desired further information items extracted from the symbol data 10 and the junction point data 12 which can be used during the further processing are stored. Particular parameters can also be pre-set during the evaluation in step d) and e), so that for example, particular junction points a1, a2, a3, etc. are to be ignored if only one portion of the technical drawing 2 is to be detected for a particular application purpose. Herein, it can also be stored in the symbol data 10 that connections beyond particular symbols st1, st2, etc. are no longer to be traced.
In general, the symbol data 10 is also significantly extended in the embodiment of the method 1 shown. For example, for particular symbol types st1, st2, etc., where this is required or desired, a disassembly into subsymbol types can be stored. An example of this is a unit in a circuit diagram that consists, in turn, of a plurality of components connected to one another. For some uses, it might be sufficient to show just the unit itself with its junction points, whereas for other uses, the inner structure of the unit is also relevant. For this purpose, the substructure can be stored in the symbol data 10.
This is shown in
The substructure is in the same form as the main structure described in the symbol data 10 in the symbol library 8, as shown in
This storage in the symbol library 8 makes it possible during the creation of the data set 4 to replace such symbol types, for which a second representation 16 is stored, with the substructure according to a suitable pre-setting or automatically. Criteria for this can also be stored in the junction point data 12, for example, that a replacement of this type by the second representation 16 only occurs if a particular junction point a1, a2, a3 of the higher-level symbol type is connected in the scanned document.
A further advantage of the method 1 described above is also that connections can be traced across a plurality of pages. This takes place in real technical drawings, for example, via textual descriptions (example: an indication “A/02” is intended to convey that a link to the identifier “A” on page 02 should be completed).
For this purpose, in the above-described step c), open lines emerging from a symbol are also identified in the technical drawing.
The same occurs for page “02”. This page comprises a symbol s6 with a connected line 18. At the open end of the line 18, an identifier “A/01” is stored. Just as for page “01”, the corresponding end point of the line 18 is associated with a connector K2 and with the connector K2, the image data of the identifier is stored.
Once both drawings are scanned in and recorded, the connectors K1, K2 can now be allocated. The identifiers are to be associated with one another in the technical drawings 2, which can now be performed automatically in the digital image of the data set 4: the identifiers are herein automatically recognized so that the connectors K1 and K2 can be associated with one another. With the association of K1 to K2, the lines 17, 18 can be bound into one line and the respective other end points of the lines 17, 18 form the end points of the newly arising line and its associations. Thus, a data set 4 forms which comprises both drawings 2.
The tracing of the open connections, as described, can be implemented efficiently as a stack structure. During the vectorization or text recognition, a type of table of contents is herein created, from which it is apparent which (unambiguous) identifiers belong to which page. In this way, when tracing connectors, the relevant page is opened directly and investigated. In the event that an identifier is not or is falsely recognized, all the pages are naturally investigated nonetheless.
The system further enables an automatic recognition of errors in the technical drawings 2. During the method 1 described, during the association, it is always checked whether a junction point must be connected and whether a multiple-bound junction point may be duplicated automatically. It is further checked whether the rules regarding the linking are adhered to, i.e., it is checked whether, for example, two junction points marked as “incoming” are falsely linked to one another. In case of error, a suitable notification can be output with precise details of document, symbol and junction point to the user who carries out a manual check and correction.
The method 1 has so far been described making reference to a simplified schematic exemplary embodiment.
The input screen 18 shows in the lower region the graphical information 6, i.e., an actual image of the symbol type together with its junction points and their position, which are used as described for allocating the symbol types to symbols.
In the upper region, the input screen 18 shows possibilities for the input of symbol data 10. The list shown enables the input of a name for the symbol type and the automatic allocation to a named API (automation function instance) which represents a particular function block in the SPPA-T3000. Furthermore, particular identification markings can be stored and an automatic replacement by a substructure can be instigated (“Explode macro”).
Finally, in the middle region are possibilities for storing junction point data which takes place in the form of a table. There are switching surfaces for adding or removing junction points, each representing a row of the table. Stored in the columns are the different types of junction point data, firstly a junction point name and a junction point type (here: “IN”=“incoming” or “OUT”=“outgoing”).
Also, the type of the identification is stored. In the example of
With the input screen 18 shown, the creation of a symbol library 8 for use in the SPPA-T3000 system is possible relatively easily, by which the above-described method 1 for the automated detection of technical drawings 2 can be applied rapidly.
While there have been shown, described and pointed out fundamental novel features of the invention as applied to a preferred embodiment thereof, it will be understood that various omissions and substitutions and changes in the form and details of the methods described and the devices illustrated, and in their operation, may be made by those skilled in the art without departing from the spirit of the invention. For example, it is expressly intended that all combinations of those elements and/or method steps which perform substantially the same function in substantially the same way to achieve the same results are within the scope of the invention. Moreover, it should be recognized that structures and/or elements and/or method steps shown and/or described in connection with any disclosed form or embodiment of the invention may be incorporated in any other disclosed or described or suggested form or embodiment as a general matter of design choice. It is the intention, therefore, to be limited only as indicated by the scope of the claims appended hereto.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10 2014 207 874.1 | Apr 2014 | DE | national |
This is a U.S. national stage of application No. PCT/EP2015/058560 filed 21 Apr. 2015. Priority is claimed on German Application No. 10 2014 207 874.1 filed 25 Apr. 2014, the content of which is incorporated herein by reference in its entirety.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2015/058560 | 4/21/2015 | WO | 00 |
