The present disclosure relates to data processing systems, and datastores to such systems. In particular, the present disclosure relates to data node types related to an interlocking trees datastore.
Data structures facilitate the organization and referencing of data. Many different types of data structures are known in the art, including linked lists, stacks, trees, arrays and others. The tree is a widely-used hierarchical data structure of linked nodes. The conventional tree is an acyclic connected graph where each node has a set of zero or more child nodes and at most one parent node. A tree data structure, unlike its natural namesake, grows down instead of up, so that by convention, a child node is typically referred to as existing “below” its parent. A node that has a child is called the child's parent node (or ancestor node, or superior node). In a conventional tree, a node has at most one parent. The topmost node in a tree is called the root node. A conventional tree has at most one topmost root node. Being the topmost node, the root node does not have a parent. Operations performed on the tree commonly begin at the root node. All other nodes in the tree can be reached from the root node by following links between the nodes. Nodes at the bottommost level of the tree are called leaf nodes or terminal nodes. As a leaf node is at the bottommost level, a leaf node does not have any children.
The KStore or K is a datastore made up of a forest of interconnected, highly unconventional trees of one or more levels. Each node in the KStore can have many parent nodes. The KStore is capable of handling very large amounts of highly accessible data without indexing or creation of tables. Aspects of KStore are the subject of a number of patents including U.S. Pat. Nos. 6,961,733, 7,158,975, 7,213,041, 7,340,471, 7,348,980, 7,389,301, 7,409,389, 7,418,455 and 7,424,480, which are hereby incorporated by reference in their entirety.
Nodes in a KStore are typically comprised of at least four fields, including a pointer to an asCase node, a pointer to an asResult node, a pointer to an asCaseList and a pointer to an asResultList. Because either an asCaseList or an asResultList but not both will exist in any particular node, two new node types can be created, one node type comprising a node with an asCase pointer, an asResult pointer and a pointer to an asCaseList and one node type comprising a node with an asCase pointer, an asResult pointer and a pointer to an asResultList. The asCaseList and asResultList fields are pointers to pointer lists and thus are the same size or larger than the asCase or asResult fields, thus the eliminated field can reduce the size of the node by about 25%. As a KStore is composed essentially entirely of nodes, eliminating the pointer to pointer list filed can reduces the size of the KStore itself by about 25%. As KStores on the order of 50 GigaBytes are not uncommon, large amounts of space can be saved by elimination of the pointer to a pointer list field. Alternatively, a single node structure may exist with a pointer to a list in one field and an indicator stored in another field that indicates whether the list pointed to is an asCaseList or an asResultList.
In the drawings:
A KStore or K is a datastore made up of a forest of interconnected trees.
The interlocking trees datastore comprises a first tree that depends from a first root node (a primary root node) and may include a plurality of branches. Each of the branches of the first tree ends in a leaf node called an end product node. The first root node may represent a concept, such as but not limited to a level begin indicator (e.g., BOT or Beginning of Thought). For example, referring to
A second root (e.g., root node 114) of the same level of the same trees-based datastore is linked to each leaf node of the first tree (e.g., to nodes 104, 106, 108, 110 and 112) and is called an EOT (End Of Thought) node. Leaf nodes of a KStore are also called end product nodes. End product nodes include a count that reflects the number of times the sequence of nodes from BOT to EOT has occurred for the unique sequence of nodes that end with that particular end product node. For example, node 106 with a count of 1 reflects the counts associated with the path connecting nodes 102, 124, 134, 138, 140 and 142. The second root (e.g., root node 114) is a root to an inverted order of the first tree or to an inverted order of some subset of the first tree, but does not duplicate the first tree. Node 134 is a node that is shared by the KStore path that ends with end product node 106 and by the KStore path that ends with end product node 108. Thus the count of node 134 (4) is the combination of the count of node 106 (1) and the count of node 108 (3).
Finally, the trees-based datastore comprises a plurality of trees of a third type in which the root node of each of these trees can be described as an end product node of an immediately adjacent lower level or as an elemental root node and may include or point to data such as a dataset element or a representation of a dataset element. The root nodes 116, 118, 120 and 122 are end product nodes of the immediately adjacent lower level of the KStore. It will be appreciated that not all of the root nodes of KStore 100 are illustrated in
Branches of the first tree are called asCase branches or asCase paths. AsCase paths are linked via asCase links denoted by solid lines in the Figures. Together, all the asCase paths of a KStore form the asCase tree of that level. The asCase tree depends from a first root (the primary root, e.g., node 102 in
The interlocking trees datastore may capture information about relationships between dataset elements encountered in an input file by combining a node that represents a level begin indicator (e.g., BOT) with a node that represents a dataset element to form a node representing a subcomponent. A subcomponent node may be combined with a node representing a dataset element to generate another subcomponent node in an iterative sub-process. Combining a subcomponent node with a node representing a level end indicator may create a level end product node. The process of combining a level begin node with a dataset element node to create a subcomponent and combining a subcomponent with a dataset element node and so on may itself be iterated to generate multiple asCase branches in a level. AsResult trees may also be linked or connected to nodes in the asCase tree, such as, for example, by a root node of an asResult tree pointing to one or more nodes in the asCase tree.
As nodes are created, asCase and asResult links may be simultaneously generated at each level and asCaseLists and asResultLists may be generated and updated. As described above, an asCase link represents a link to the first of the two nodes from which a node is created. For example, referring to
where Tom and Bill are salesmen, 100 and 40 are product numbers and PA and NJ are states in which the salesmen sold their products. The asCase tree generated from this input may comprise a view of the data in the context of “state information with the context of salesman” context.
An asResult link represents a link to the second of the two nodes from which a node is created. For example, the asResult link of node 124 points to node 116 (Bill). The generation of the asResult links creates a series of interlocking trees where each of the asResult trees depend from a root comprising a dataset element. This has the result of recording all encountered relationships between the root nodes and the nodes of the asCase trees in the KStore. That is, the asResult trees capture all the possible contexts of the nodes of the interlocking trees. If, for example, the input to the interlocking trees datastore generator comprises a universe of sales data including salesman name, day of the week, product number and state, the resulting asResult links of the generated interlocking trees datastore could be used to extract information such as: “What salesmen sell in state X”, “How many items were sold on Monday?” “How many items did Salesman Bill sell on Monday and Tuesday?” and the like, all from the same interlocking trees datastore, without creating multiple copies of the datastore, and without creating indexes or tables.
It will be appreciated that this information is determinable from the structure of the interlocking trees datastore itself rather than from information explicitly stored in the nodes of the structure. Paths can be traversed backwards towards the root node to determine if the subcomponent or end product belongs to a particular category or class of data. Links between nodes may be bidirectional. For example, a root node for the dataset element “Monday” (e.g. root node 118) may include a pointer to a subcomponent BOT-Bill-Monday (e.g., node 134) in node 118's asResultList while the node BOT-Bill-Monday, node 134 may include a pointer to the node Monday, node 118, as its asResult pointer and so on. Furthermore, by following asCase links of the nodes containing a desired dataset element, other subcomponents and end products containing the desired dataset element can be found along the branch of the asCase tree. It will be appreciated that the described features cause the datastore to be self-organizing.
As described above, a generalized node in a KStore can be used to instantiate any node in a KStore. However, subcomponent nodes and BOT nodes have entries only in the pointer to asCaseList field because there are no subcomponent nodes which are the asResult node of other nodes. Moreover, elemental root nodes and end product nodes will not have a pointer to an asCaseList in the pointer to asCaseList field because an elemental root node or end product node will not be the Case node for another node. The one exception is the BOT node or primary root node, which will have a pointer to an asCaseList in its asCaseList field. To create a more memory efficient KStore, in accordance with aspects of the subject matter disclosed herein, the unused pointer list pointer is omitted from the nodes of the KStore. Two different node structures can be defined, one having the field for the asCaseList pointer and no field for the asResultList pointer and one having the field for the asResultList pointer and no field for the asCaseList pointer. Alternatively, one node structure can be defined, the node structure comprising a field for a pointer to the asCase node, a field for a pointer to the asResult node, a field for a pointer to a list of pointers and a field for an indicator for the type of the list of pointers the pointer points to or a field for an indicator for the type of node.
300 illustrates a first structure of two different node structures of nodes of a KStore. Node 300, which can be a BOT or primary root node or subcomponent node, has a field for a pointer to an asCase node, field 706, a field for a pointer to an asResult node, field 708, a field for a pointer to an asCaseList, field 710. Node 300 may also include one or more additional fields, field 714 and a field for a value or a pointer to a value, field 716 but does not have a field for a pointer to an asResultList.
400 illustrates a second structure of two different node structures of nodes of a KStore. Node 400, which can be an end product node or an elemental root node except for the BOT node. Node 400 has a field for a pointer to an asCase node, field 706, a field for a pointer to an asResult node, field 708, a field for a pointer to an asResultList, field 712. Node 300 may also include one or more additional fields, field 714 and a field for a value or a pointer to a value, field 716 but does not have a field for a pointer to an asCaseList.
500 illustrates a node structure of a node of a KStore. Node 500 can represent an end product node, an elemental root node, a BOT node, an EOT node, or a subcomponent node. Node 500 has a field for a pointer to an asCase node, field 706, a field for a pointer to an asResult node, field 708, a field for a pointer to a list, field 718 and a field 720 that includes an indicator for the type of node it is. Node 500 may also include one or more additional fields, field 714 and a field for a value or a pointer to a value, field 716. The field 718 of the node represented by node 500 may have either a pointer to an asCaseList or a pointer to an asResultList but cannot have both pointers to an asCaseList and an asResultList.
The Learn Engine 26 may receive data from many types of input data sources and may transform the received data to particles suitable to the task to which the KStore being built will perform. For example, if the data being sent to the KStore is information from a field/record type database, particular field names may be kept, changed, or discarded, depending on the overall design of the KStore the user is creating. After breaking down the input into appropriate particles, the Learn Engine 26 may make appropriate calls to the K Engine 14 and pass the data in particle form in a way that enables the K Engine 14 to put it into the KStore structure,
API utilities such as API utility 23 receive inquiries and transform the received inquiries into calls to the K Engine, to access the KStore directly or to update associated memory. In the event that a query is not to be recorded in the structure of a KStore a LEARN SWITCH may be turned off. In the event that a query is to be recorded in the structure of the KStore, (as in Artificial Intelligence applications, for example) the LEARN SWITCH may be turned on. API utilities may get information from the KStore using predefined pointers that are set up when the KStore is built (rather than by transforming the input into particles and sending the particles to the KEngine). For instance, a field may point to the Record End of Thought (EOT) node, the Field EOT node, the Column EOT node and the Beginning Of Thought (BOT) node. This field may be associated with the K Engine, and may allow the K Engine to traverse the KStore using the pointers in the field without requiring the API Utility to track this pointer information.
Within the KStore computing environment information may flow bi-directionally between the KStore or KStores, a data source 30 and an application 34 by way of a K Engine 14. The transmission of information between the data source 30 and the K Engine 14 may be by way of a learn engine 26, and the transmission of information between the application 34 and the K Engine 14 may be by way of an API or API utility engine 23. Data source 30 and application 34 may be provided with graphical user interfaces 36, 38 to permit a user to communicate with the system.
Objects or other types of system components such as learn engine 26 and the API utility engine 23 may be provided to service learn and query threads so that applications and interfaces of any kind can address, build and use the KStore(s). Learn engine 26 may provide an ability to receive or get data in various forms from various sources on the same computer or on different computers connected via a network and to turn it into input particles that the K Engine 14 can use. The API Utility engine may provide for appropriate processing of inquiries received by application software of any kind. The API utility engine 23 and the learn engine 26 get information from and/or put information into a KStore. It will be understood by those of skill in the computer arts that software objects can be constructed that will configure the computer system to run in a manner so as to implement the attributes of the objects. It is also understood that the components described above may be created in hardware as well as software.
The various techniques described herein may be implemented in connection with hardware or software or, where appropriate, with a combination of both. Thus, the methods and apparatus described herein, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing aspects of the subject matter disclosed herein. In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. One or more programs that may utilize the creation and/or implementation of domain-specific programming models aspects, e.g., through the use of a data processing API or the like, may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language, and combined with hardware implementations.
While the subject matter disclosed herein has been described in connection with the figures, it is to be understood that modifications may be made to perform the same functions in different ways. While innumerable uses for this invention may be found, and significant variability in the form and manner of operation of this invention are described and will occur to those of skill in these arts, the invention is not limited in scope further than as set forth in the following claims.
This application claims the benefit of U.S. patent application Ser. No. 11/084,996, filed Mar. 18, 2005, entitled “SYSTEM AND METHOD FOR STORING AND ACCESSING DATA IN AN INTERLOCKING TREES DATASTORE” by MAZZAGATTI et al. which application is a Continuation of U.S. patent application Ser. No. 10/385,421, filed Mar. 10, 2003 and U.S. patent application Ser. No. 11/185,620, filed Jul. 20, 2005, entitled “METHOD FOR PROCESSING NEW SEQUENCES BEING RECORDED INTO AN INTERLOCKING TREES DATASTORE,” by MAZZAGATTI. These applications are incorporated in their entirety herein.