Patent Application
20220358126
Graph databases store entities and explicit connections between entities to thereby reduce the latency in tracing connections between entities. In particular, graph databases are efficient at returning all entities that are connected to a single entity even when those entities are connected through different relationship types. In addition, graph databases are efficient at identifying entities that are connected to an entity through one or more intervening entities.
In a graph database, an origin node representing one entity is connected to a destination node representing another entity through a relationship having a relationship type. Properties of the entity can be stored within the entity's node using a key:value representation. However, adding properties to nodes impacts the scalability of the graph database. In particular, as more properties are added to entity nodes, the size of the graph database increases. In addition, including properties within the nodes increases the amount of time needed to search for nodes having certain properties since the graph database must be traversed and each node examined to determine if the node contains the property. To avoid this, some implementations of the graph database represent certain properties as nodes in the graph and establish connections between those property-based nodes and the entity nodes that contain those properties. Such implementations exponentially increase the size of the graph database and negatively impact the speed at which the graph database can be traversed.
The discussion above is merely provided for general background information and is not intended to be used as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all disadvantages noted in the background.
A computer-implemented method includes receiving a message comprising an origin, a destination and a relationship type for a relationship between the origin and the destination. The message further includes a payload. A first node is created in a graph database for the origin and a second node is created in the graph database for the destination. A relationship is set between the first node and the second node in the graph database based on the relationship type. A node is created in the graph database for the message while preventing the payload from being stored in the graph database. A relationship is created between the first node and the node for the message. The message, including the payload, is stored in a searchable datastore separate from the graph database.
In accordance with a further embodiment, a computer-implemented method includes receiving a message comprising a relationship and a payload, the relationship having an origin and a destination and the payload providing data related to one of a document and an event. A graph database is modified as necessary such that the graph database includes a node for the origin and a node for the destination. The graph database is modified as necessary so that the graph database includes a node for the document or event of the payload and so that the graph database includes a relationship between the node for the document or event and the node for the origin such that the node document or event can be found from the origin node.
In accordance with a still further embodiment, a system includes a lightweight graph database having nodes and relationships between nodes. The nodes include a node representing a document described in a message, the message having at least one element that provides a unique identifier for the document and a payload. The node representing the document is identified by the unique identifier and the payload of the message is excluded from the lightweight graph database. The nodes also include a node representing a destination in the message. The lightweight graph database includes a relationship between the node representing the document and the node representing the destination. The system further includes a datastore containing the message including the payload.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
Embodiments described below provided an improvement to computers and an improvement to database technology. In particular, the embodiments receive incoming messages that each define at least one relationship and have a data payload describing a document or event. The relationships defined in the message are used to create a lightweight graph database consisting of nodes connected by edges. In addition, a node is created for the document or event described by the payload and an edge is created to connect the node for the document/event to a node defined in the relationship section of the message. The data payload is stored in a searchable datastore that can be searched using an index search engine. The number of nodes added to the graph database and the number of properties for each node are limited to keep the graph database as small as possible.
By providing both a lightweight graph database and a searchable datastore, the invention prevents the graph database from becoming excessively large thereby allowing faster searches of relationships between entities while also allowing the full data payload to be searchable through the datastore. In addition, by including a node for the document/event in the graph database, the payload that provided a particular node in the graph can be quickly identified and retrieved from the datastore. As a result, users are able to use the lightweight graph database to identify entities of interest based on their connection to one or more known entities while also allowing the user to obtain details of each of the entities of interest after they have been identified without impacting the speed at which the graph database can be traversed. This is an improvement to the operation of the computer itself by making the computer return graph database results faster and allowing for faster visualization of the connections between entities and is an improvement to database technology by splitting data into two separate types of databases thereby allowing the graph database to scale better to larger sets of data.
Document messages and event messages also include a relationship section 316/416 that provides an array of relationships with each relationship including a destination 318/418 and a type of relationship 320/420. In addition, an optional origin 322/422 can be provided in a relationship. If an origin is not provided, the origin is assumed to be the current message. Destinations 318/418 and origins 322/422 include a unique identifier or a collection of values that can be combined to construct a unique identifier, for the destination and origin, respectively. For example, namespace 324, class 326 and ID 328 of message 300 are combined to form a unique identifier for destination 318 and name space 424, class 426 and ID 428 of message 400 are combined to form a unique identifier for destination 418. Similarly, namespace 330, class 332 and ID 334 of message 300 are combined to form a unique identifier for origin 322 and namespace 430, class 432 and ID 434 of message 400 are combined to form a unique identifier for origin 422. Those skilled in the art will recognize that other techniques can be used to provide a unique identifier for each destination and origin in a message.
At step 202, ingest engine 110 requests a next message from either message broker 104 or message broker 108.
At step 204, ingest engine 110 determines the type of message that has been received using type 312/412.
At step 206, ingest engine 110 searches graph database 112 for a node that has the same unique identifier for the document or event as the message. A node for the document or event will already exist in graph database 112 if a previous message for the document or event has been received by ingest engine 110. If a matching node cannot be found, ingest engine 110 creates a new node for the document or event. Thus, either a document node or an event node is created depending on the type of message. A unique identifier for the node is set to the unique identifier of the document or event. As part of creating the node for the document or event, the display name property for the node is set to display name 310/410 of the message.
When creating the node for the document or event, the payload of the message is not stored within the node but instead is prevented from being added to graph database 112. This reduces the amount of information stored in graph database 112 making it possible for graph database 112 to handle larger numbers of messages and to provide faster search results for visualizing the relationships stored in graph database 112.
At step 208, ingest engine 110 processes relationships in the message that are found in relationship section 316/416 of the message.
At step 508, ingest engine 110 searches lightweight graph database 112 to determine if a node exists that has the unique identifier of the origin. If the node already exists, a node does not need to be created for the origin in graph database 112. However, if the unique identifier of the origin is not in the lightweight graph database 112 at step 508, a value node is created in graph database 112 for the origin at step 510. The unique identifier of the created node is set to the unique identifier of the origin.
After step 504 and step 510 and if a node did not need to be created for the origin at step 508, ingest engine 110 searches lightweight graph database 112 for the unique identifier of the destination at step 514. If a node is found in lightweight graph database 112 that has the unique identifier of the destination, a new node does not need to be created for the destination. However, if the unique identifier is not found in graph database 112, a node for the destination is created using the unique identifier of the destination at step 516. If the destination node already existed at step 514 or after it was created at step 516, a relationship between the origin node and the destination node is updated at step 518.
In accordance with one embodiment, different messages can provide different types of relationships between any given pair of nodes. In addition, different messages can provide the same type of relationship between two nodes. Further, document messages can be used to convey that a specified type of relationship provided by a previous version of the document message should be removed. For example, if a first version of a document is received that provides a relationship of a specified type between two nodes, a second version of the document (having the same unique identifier as the first version) can indicate that the relationship with that specified type is no longer active. Note that this is only effective to remove a type of relationship set by a document having the same unique identifier. It does not affect relationships set by messages having different unique identifiers. For example, if a document message with a first unique identifier sets a relationship of a specified type between two nodes and a second message with a second unique identifier sets a relationship of the same specified type between the two nodes, a later document message having the first unique identifier can only deactivate the relationship of the specified type set by the earlier document message with the first unique identifier. The relationship of the specified type set by the message with the second unique identifier would continue to be active.
When a relationship of a specified type is created between two nodes in graph 112, a property is created for the relationship to keep track of which messages have set the specified type of relationship between the two nodes. In accordance with one embodiment, this property is an array or list of document or event unique identifiers corresponding to documents or events that set the specified type of relationship between the two nodes. When a message is received that sets the specified type of relationship between the two nodes, the unique identifier of the document or event of the message is added to the list if it is not already in the list. When a message is received that indicates that the relationship is no longer active, the unique identifier for the document or event of the message is removed from the list. These actions are shown as steps 606, 608 and 610 below.
Specifically, at step 606, ingest engine 112 determines whether the message is conveying that the relationship between the origin and destination is active or inactive. If the message is a document message with an “inactive” state 314, the message is conveying that, for the purposes of this document message, the relationship between the origin and destination is no longer active. If the current message is an event message or if the current message is a document message with an “active” state 314, the message is conveying that the relationship between the origin and the destination set in the message is currently active.
If the message is an event message or a document message with an “active” state 314 at step 606, the unique identifier of the document or event is added to the list for the specified type of relationship at step 608. Note that the unique identifier is only entered once in the list. Thus, if a message with the same unique identifier for a document or event is received multiple times, it is only added to the list for the specified type of relationship once.
If the message is a document message and state 314 is set to “inactive” at step 606, the relationship between the origin and the destination set in the message is considered to no longer be active for the purposes of the document. Based on this, the unique identifier for the document is removed from the list of unique identifiers for the specified type of relationship at step 610.
After the unique identifier of the document has been removed, the list of unique identifiers for the specified type of relationship is checked at step 612 to determine if it is empty. If the list is empty, the specified type of relationship is no longer considered active between the two nodes, and the specified type of relationship is removed from between the origin and destination nodes at step 614. By removing the specified type of relationship when no active documents provide the specified type of relationship, steps 612 and 614 reduce the size of the graph database thereby improving the performance of the computer system. If the list of messages is not empty at step 612 or after the specified type of relationship has been removed from between the origin and destination nodes at step 614, the process of
When the origin node is the node for the document or event, the creation of the relationship between the document/event node and the destination node allows the document/event node to be located from the destination node. As a result, when the destination node is located during a traversal of the graph, it is possible to follow the relationship from the destination node to the document/event node. Typically, this will be a “describes” relationship that indicates that the document/event node represents a document or event that describes the entity represented by the destination node. Using the unique identifier of the document/event node, a search of datastore 116 can then be performed to retrieve the payload of the corresponding message and thereby retrieve additional information about the entity represented by the destination node.
Returning to
Returning to
After the ingest engine has stored the message in datastore 116, an inference engine 700 of
One or more value nodes may identify a particular person, place or thing, referred to generically as an entity. In step 800 of
Using the unique identifier for the entity node, inference engine 700 searches lightweight graph database 112 to determine if the entity node is already in the graph at step 806. If the entity node is not in the graph at step 806, the entity node is created within lightweight graph database 112 at step 808 and is assigned the unique identifier. A display name property of the entity node is also set to the display name determined from the message. This results in an entity node 730 in
The data stored in lightweight graph database 112 and datastore 116 can be explored using a data exploration engine 900 as shown in
Upon receiving the search criteria in boxes 1002 and 1004, search module 902 provides the search criteria to search engine 752, which then performs a search of datastore 116 for all matching messages. Data exploration engine 900 uses the returned messages to locate the corresponding message nodes in lightweight graph database 112 and the corresponding value nodes linked to those message nodes by a relationship. Data exploration engine 900 then finds the entity nodes linked to those value nodes. The display names for the matching entities and the label for the matching entities are returned by search 902 so as to be displayed in a search result list 1010 of user interface 906. For example, in
Upon selection of one of the returned items in list 1010, data exploration engine 900 provides a user interface 1100 of
User interface 1100 also includes tabs 1106, 1108 and 1110. Initially, tab 1106 is selected to show documents that define properties for the selected item as shown in
When tab 1110 is selected, visualization module 904 of data exploration engine 900 generates a visualization of the relationships between the entities in list 1202 and the selected entity.
Ingest engine 110, index builder 114, lightweight graph database 112, datastore 116, inference engine 700, search engine 752, data exploration engine 900, graph search 908 and user interface 906 are implemented on one or more computing devices, an example of which is shown in
Computing device 10 further includes an optional hard disc drive 24, an optional external memory device 28, and an optional optical disc drive 30. External memory device 28 can include an external disc drive or solid-state memory that may be attached to computing device 10 through an interface such as Universal Serial Bus interface 34, which is connected to system bus 16. Optical disc drive 30 can illustratively be utilized for reading data from (or writing data to) optical media, such as a CD-ROM disc 32. Hard disc drive 24 and optical disc drive 30 are connected to the system bus 16 by a hard disc drive interface 32 and an optical disc drive interface 36, respectively. The drives and external memory devices and their associated computer-readable media provide nonvolatile storage media for the computing device 10 on which computer-executable instructions and computer-readable data structures may be stored. Other types of media that are readable by a computer may also be used in the exemplary operation environment.
A number of program modules may be stored in the drives and RAM 20, including an operating system 38, one or more application programs 40, other program modules 42 and program data 44. In particular, application programs 40 can include programs for implementing any one of the applications discussed above. Program data 44 may include any data used by the systems and methods discussed above.
Processing unit 12, also referred to as a processor, executes programs in system memory 14 and solid-state memory 25 to perform the methods described above.
Input devices including a keyboard 63 and a mouse 65 are optionally connected to system bus 16 through an Input/Output interface 46 that is coupled to system bus 16. Monitor or display 48 is connected to the system bus 16 through a video adapter 50 and provides graphical images to users. Other peripheral output devices (e.g., speakers or printers) could also be included but have not been illustrated. In accordance with some embodiments, monitor 48 comprises a touch screen that both displays input and provides locations on the screen where the user is contacting the screen.
The computing device 10 may operate in a network environment utilizing connections to one or more remote computers, such as a remote computer 52. The remote computer 52 may be a server, a router, a peer device, or other common network node. Remote computer 52 may include many or all of the features and elements described in relation to computing device 10, although only a memory storage device 54 has been illustrated in
The computing device 10 is connected to the LAN 56 through a network interface 60. The computing device 10 is also connected to WAN 58 and includes a modem 62 for establishing communications over the WAN 58. The modem 62, which may be internal or external, is connected to the system bus 16 via the I/O interface 46.
In a networked environment, program modules depicted relative to the computing device 10, or portions thereof, may be stored in the remote memory storage device 54. For example, application programs may be stored utilizing memory storage device 54. In addition, data associated with an application program may illustratively be stored within memory storage device 54. It will be appreciated that the network connections shown in
Although elements have been shown or described as separate embodiments above, portions of each embodiment may be combined with all or part of other embodiments described above.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms for implementing the claims.
