Patent Application
20080222176
The present invention relates to XPath evaluation, and more particularly to the streaming evaluation of XPath expressions with predicates for data processing or network data routing.
XML databases and XML content-based routing are well known in the art. For XML databases, XPath is a language for accessing XML documents in the database. Efficient evaluation of XPath is of particular interest because evaluation of XPATH queries may greatly affect the performance and scalability of XML databases. Typically, XML documents are stored according to a tree data model, such as XQuery data model or Document Object Model (DOM). The nodes of the data tree are streamed and scanned. The XPath is then evaluated and a result which satisfies the XPath query is returned. For XML content-based routing, XML documents are parsed and XPath queries are evaluated. Data are sent based on the query results. High-performance of XPath evaluation is extremely important also.
One of ordinary skill in the art will recognize that conventional approaches of processing XPath queries on XML data streams, such as automata or transducer-based approaches, may suffer from various problems. Conventional approaches explicitly express all the possible matching paths for an input XML node in their state machines or working buffers. However, the number of matching paths can be very large in some situation. In particular, the number of combinations being tracked may grow near exponentially. Thus, this conventional approach may be very inefficient. Moreover, conventional approaches to XPath streaming also passively process every input node or event and do not or cannot skip uninterested XML sub-trees.
Moreover, predicates should be accounted for. Predicates add complexity to XPath evaluation because a predicate may refer to a value that may only be available at the end of the node with which it is associated. Thus, both candidate result nodes and data for the predicate evaluation may need to be buffered. Conventional methods do not buffer candidate result nodes efficiently enough.
Accordingly, there exists a need for an improved method for evaluating XPath with predicates. The method is preferably capable of processing XPath expressions more efficiently and requiring one scan of an XML document. The present invention addresses such a need.
The present invention provides a method and system for evaluating XPath queries with predicates. The method and system comprise providing a query tree including a plurality of query nodes. At least one of the query nodes corresponds to at least one predicate and has at least one level. The predicate is evaluated for at least one previous query node. The method and system comprise scanning a plurality of data nodes of a document and determining if the plurality of data nodes matches the plurality of query nodes. The method and system also comprise placing data related to the data node in match stacks corresponding to matched query nodes. The data for the at least one query node includes at least one attribute (or variable) corresponding to the at least one predicate. The method and system further comprise propagating a matching of the at least one query node backward to a matching of the at least one previous query node.
The present invention provides improved method for streaming evaluation of XPath with predicates. The following description is presented to enable one of ordinary skill in the art to make and use the invention and is provided in the context of a patent application and its requirements. Various modifications to the preferred embodiments and the generic principles and features described herein will be readily apparent to those skilled in the art. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features described herein.
Although the embodiments below are described in the context of XML documents and XPath, any hierarchical data and query language with similar characteristics to XPath can be used without departing from the spirit and scope of the present invention.
The present application is related co-pending U.S. patent application Ser. No. 10/990,834 entitled “Streaming XPath Algorithm for XPath Value Index Key Generation” Filed on Nov. 16, 2004 and assigned to the assignee of the present application. Applicant hereby incorporates by reference the above-identified co-pending patent application.
The present invention provides a method and system for evaluating queries, such as XPath queries. The method and system comprise providing a query tree including a plurality of query nodes. At least one of the query nodes corresponds to at least one predicate and is at a level. The predicate is evaluated for at least one previous query node. The method and system comprise scanning a plurality of data nodes of a document and determining if the plurality of data nodes matches the plurality of query nodes. The method and system also comprise placing data related to the data node in match stacks corresponding to matched query nodes. The data for the at least one query node includes at least one attribute corresponding to the at least one predicate. The method and system further comprise propagating at least one value for the at least one predicate backward from the at least one query node to the at least one previous query node.
A query that is preferably an XPath query is received for processing, via step 102. The query is then compiled and a query tree built, via step 104. The query tree is thus based upon the XPath query. The compiled query tree is provided, via step 106. Consequently, using steps 102, 104, and 106, an XPath query tree may be provided. The query tree is discussed below. XML data, for example in the form of a tree or stream, is received, via step 108. This XML data is scanned, via step 110. In step 110, the XML data is preferably scanned in order of the data nodes, with the node kind, the name, level (depth), node ID, and value being read. In a preferred embodiment, step 110 is performed using a single scan. In one embodiment, a portion of a data tree including data nodes and a query tree are available after step 110. The data nodes are matched against the query nodes, via step 112. Stated differently, it is determined whether the data nodes match the query nodes in step 112. Also in step 112, if data and query nodes do match, then the matching data nodes are placed in match stacks corresponding to the matched query nodes and predicates are accounted for by evaluating and propagating any variable values. Depending on whether the query node itself or other values associated with the query node are needed, the node and other values may be extracted and placed in the match stacks. Thus, the document is processed and matches found in step 112. Once the processing completes, the result may be outputted, via step 114.
To represent matches found in step 112, a logical stack or list of matching units is associated with each query node. Each matching unit contains a data node that matches with the query node, and the data nodes of the matching units in a stack have AD (i.e. ancestor-descendant) relationships among themselves.
The information contained in a matching unit includes:
The matching units are preferably stored in a match stack table. A stack top table preferably stores the addresses (or indexes) of the top matching units of logical stacks in the match stack table for each query node. If an XPath expression contains PC (i.e. parent-child) relationships only, then the stacks contain at most one entry each. Multiple entries in a stack occur for a query node that is at or below an AD step. For some matching units across the neighboring stacks, there are also relationships that are either PC or AD corresponding to the query steps. The matching units may thus form a matching grid. In turn, a matching grid can be represented by one combined stack or an array of stacks, one for each query node. Using an array may eliminate the cost of maintaining multiple stacks and improve locality during processing.
In a preferred embodiment, a data node matches a query node if the following three conditions hold: (1) if the query node is not the root step (the root step matches the document root), then there is a match for the query node in the previous step of the query; (2) the data node matches the query node of the current step (i.e., the node names match); and (3) the edges of the data and query nodes match. If the relationship between the query node of the current step and the query node of its previous step is a PC relationship, then condition (3) is satisfied if the level of the data node is the same as the level of the matching unit in the previous step plus one. If the relationship is an AD relationship, then condition (3) is satisfied if the level of the data node is greater than the level of the data node of the matching unit in the previous step.
When the query tree is large, it may be inefficient to test the three conditions for all the internal query nodes when a data node arrives. To improve the matching process for such a case, the active states for the queries may be maintained. A query node is “active” if it can potentially match the next data node in the XML stream. The set of active query nodes is called active states (AS). A query node is “direct” if the edge to its previous step is a solid line (PC relationship). Otherwise, it is “indirect” (AD relationship, or also called in-transitive). The active states are divided into two sets. The direct active states, once matched, may become inactive, while the indirect active states will continue to be active after their matchings. Note that initially only the root step is active.
Table 1 below depicts an embodiment of the rules of maintaining the direct active states. The next direct states are the union of direct query nodes of the current matched query nodes. When adding a data node, the set of matches (M) is first calculated. Then, the union of the direct nodes of the query nodes in M is obtained to get the resulting direct states. When removing matches from the stack, the most recent matches, i.e., the matches at the top of the stack with the largest level, can be obtained from the match stack table 504.
Table 2 shows examples of rules for maintaining the indirect active states. An indirect query node is active only if the query node of the previous step has some matches. Thus, the stack of a query node is checked. If it is empty before adding a match, then its indirect nodes are activated. If it is empty after removing a match, then its indirect nodes are deactivated.
Because the rules for maintaining direct and indirect active states are different, two hash tables (or other associative memory) are preferably used to keep track of the active states: a direct AS hash table for maintaining the direct active query nodes, and an indirect AS hash table for maintaining the indirect active query nodes.
The query tree provided using the method 120 includes query nodes and links. The link between two query nodes represents the relationship between the query nodes. For example parent-child (PC) or ancestor-descendant (AD) relationship. Thus, a query tree Q(V, E) is defined as follows. V is a set of query nodes. Each query node, q, corresponds to a step and may be labeled with a QName for the name test, and contains attribute definitions needed to evaluate the path expression, including the predicates associated with the step. E is a set of edges, or links, connecting two query nodes. Each edge represents a child or descendant relationship from one step to the previous step in correspondence to the axis of the step. Note that relative path expressions in predicates are merged into the query tree, and in predicates, relative path expressions are replaced with references to relevant attribute variables. Graphically single line is used to represent a child axis and double line a descendant axis. A dotted line is used to represent a segment of a query of less interest. The terms previous step and next step refer to a parent query node and child query node in a query tree, respectively.
If a node matches a non-leaf query node, it may need to evaluate some aggregate attributes, such as the candidate result sequence (CRS), which contains nodes that matches the output query node but not yet fully filtered by the predicates, or the predicate truth value. Some of the aggregate functions are listed in Table 4 below. For example, five attributes will be involved in predicate a<b, which is equivalent to min(a)<max(b), and attribute value(a) and value(b) associated with query nodes labeled a and b are kept, and aggregate attributes min(a) and max(b) associated with their previous steps, and propagated to an ancestor step where min(a)<max(b) is calculated as another attribute and consumed.
To evaluate an aggregate attribute for a matching from the values of matchings beneath it, the following framework is followed: 1. Init function, which initializes the attribute(s); 2. Extract function, which applies to each matched child or descendant node; 3. Accumulate function, which evaluates new value(s) based on the current cumulative value(s) and new extracted value(s); and 4. Final function, which is the final evaluation of an attribute that may be based on a set of attributes at the matching node.
For example, if a step (node) has with predicate a[b=10], there will be an attribute p for the predicate, with the following functions: (1) init: to false; (2) extract: e:=if number(b)=10 then true else false; (3) accumulate: p=p or e; (4) final: none. If the predicate is [b=10 and c>“ABC”], then the final could be to evaluate the “and”. Notice that the above framework is essentially the same as evaluating an aggregate function in general, such as average, where multiple attributes are defined to get the final result.
In addition to performing the matching in steps 160, 162, and 164 at the start of a node, when finishing traversal of the descendants of a node and at the end of a node, the values for a data node may be used to account for predicates at a previous query node and for the query result, via step 166. In a preferred embodiment, performing step 166 includes following rules related to the predicate and propagating the value to the previous query node. In addition, the matching unit is preferably popped of the match stack in step 166.
After step 162, 164, or 166, the active query states (or nodes) are maintained, via step 168. Thus, the queries states which are active are tracked in step 168 updated with query nodes that become active or inactive. In one embodiment, it can be determined whether a query node is active by checking stack emptiness of a previous step. This may be achieved through the matching process of step 162 without using separate data structures. To reduce the number of query nodes to check, a name index to query nodes may be maintained. Consequently, only query nodes that match with the current node name may be checked. In another embodiment, active query nodes may be tracked by analyzing the query tree. The rules are described from paragraph [033] to [036]. In addition, early finish also impacts the state of a query node when a positional predicate turns to true. Early finish suppresses an active query node if the finish condition is true. In addition, the matching order preferably follows the breadth-first order of the query tree for the propagation scheme.
Thus, using the method 150, predicates can be accounted for. Moreover, the method 150 stores the attributes and utilizes stacks corresponding to the query nodes. This may eliminate the cost of maintaining multiple stacks and improve locality during processing. Moreover, the method 150 may only traverse query nodes in the tree for which matches are found. Consequently, the method 150 has improved efficiency.
In one embodiment, in order to propagate a value in the method 100 or 170, attributes of the query nodes are considered. In particular, an attribute that indicates the relationship between query nodes, such as a sequence-valued attribute, may be utilized. The sequence-valued attribute is for the sequence of child nodes or descendant nodes. The rules to calculate such an attribute depend on the axis of a step, as shown in Table 5, where “U” means union of two sequences, which results in a new sequence with unique nodes from two sequences in document order.
Thus, as can be seen in Table 5, an attribute for a sequence of children is not transitive, while an attribute for a sequence of descendant-or-self is transitive, called p-transitive. When an attribute is p-transitive, its value may be propagated sideways at the end as well as upward if there is an upward link. In the last case of Table 5, duplicate propagation may be avoided if: for b: propagate upward if there is an upward link or else propagate sideways, and for a: propagate sideways and accumulate for b descendants of a. As a result, there may be no duplicates and document order may be guaranteed for b descendants of a using simple concatenation.
If there is a predicate associated with a particular query node, then duplicate propagation may be avoided using another mechanism. If there is a predicate p for query node b, for all the cases in Table 5 (paths: . . . a/b[p], . . . a//b[p]), upward propagation is allowed only if the predicate p is true. If the predicate p is false, the matching unit is dropped, but it will allow sideways propagation to pass through. When a predicate p is associated with a in . . . a[p]//b, the sideways propagation of sequence of b descendants of a between a matching units is not affected by predicate p.
In addition, it may be desirable to propagate a possible match, or CRS, for a path expression beyond two steps, or two successive query nodes in the query tree. Whether or not a CRS attribute is transitive depends on a step. For a pair of steps p and q, if PC(p, q), m(p, d1), m(q, d2), and PC(d1, d2) then s(d2), CRS at d2, can be propagated upward to d1, but not to an ancestor of d1, such as do, where m(p, d0) and AD(d0, d1), because PC(d0, d2) is not true. Thus, a CRS attribute is not p-transitive on a previous step of a child axis. Similarly, a CRS attribute is p-transitive on a previous step of a descendant axis. This property is independent of whether the result query node has child axis or descendant axis. For example, for a query/a[u]/b[v]/c[w]//d, a sequence of d descendants can be propagated sideways at step c, but as a CRS, it cannot be propagated sideways at step b or a. If we have query //a[u]//b[v]/c[w]//d/e, a CRS can be propagated at step c, a, or root, but not b.
In general, when there is no predicate, the following simple propagation rules will guarantee no duplicates in a CRS:
Simple concatenation for accumulation can guarantee uniqueness and document order of a CRS. However, when predicates are present, the simple propagation rules become problematic. The following propagation rules apply when there are predicates:
The rationale is that we keep propagating a CRS along the matching path as long as the predicate is true. However, if a false predicate is encountered, a different matching path may be used. Some examples will make this clear.
The methods 100, 150, and 170 and description above may be further understood using an exemplary expression, such as the expression /book//section[.//title =“XML” and figure/@width>300].
Consequently, the predicate may be accounted for using the method 170 and query trees 180, 180, and 180″. At the same time, documents can be efficiently scanned and matched to queries.
A method and system for evaluating hierarchical path queries that accounts for predicates are disclosed. The present invention has been described in accordance with the embodiments shown, and one of ordinary skill in the art will readily recognize that there could be variations to the embodiments, and any variations would be within the spirit and scope of the present invention, such as in a computer network having linked electronic devices and applications generally of any combination of programming and/or hardware capable of distributing an application over the network, where such often includes a computer with a processor, storage medium, bus, and program code operable thereon. The invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For instance, in one implementation, the present invention is directed to a computer program product including a computer-readable medium having instructions stored thereon for processing data information, such that the instructions, when carried out by a processing device, enable the processing device to perform the steps of streaming evaluation of a XPath path query, the path query corresponding to a query tree including a plurality of query nodes, at least one query node of the plurality of query nodes corresponding to at least one predicate and having a level, the at least one predicate being for at least one previous query node, the program including instructions for: singly scanning a plurality of data nodes of an XML document in order of the data nodes with node kind, name, level, node ID and value being read to provide a data tree; determining if the plurality of data nodes match the plurality of query nodes. For the purposes of this description, a computer-usable or computer readable medium can be any medium that can contain, store, or maintain programs and data for use by or in connection with the instruction execution system. The computer readable medium can comprise any one of many physical media such as, for example, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor media. More specific examples of a suitable computer-readable medium would include, but are not limited to, a portable magnetic computer diskette such as a floppy diskette or hard drive, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory, or a portable compact disc.
Under 35 USC §120, this application is a continuation application and claims the benefit of priority to co-pending U.S. patent application Ser. No. 11/356,366 filed entitled “Streaming XPath Algorithm for XPath Expression With Predicates”, filed on Feb. 16, 2006, which is related to U.S. patent application Ser. No. 10/990,834, filed on Nov. 16, 2004, entitled “Streaming XPath Algorithm for XPath Value Index Key Generation”, all of which is herein incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 11356366 | Feb 2006 | US |
| Child | 12122963 | US |
