The present invention generally relates to computer implemented database management systems, and particularly to systems and methods for more efficient XPath/XQuery processing.
The extensible markup language (XML) is a flexible tag-based markup language suitably used to store data for posting on the Internet or local intranets, wide area networks, or the like. XML is increasingly being used as a native language for data storage in database management systems. In these and other XML applications, it is advantageous to have a flexible XML query language for creating and modifying XML documents, for efficiently and selectively retrieving data from XML documents or collections of XML documents, for sorting data, for inputting data into XML documents, and for otherwise manipulating XML items and data. Various XML query languages have been developed, including XML-QL, XQL, Quilt, and XQuery.
XML employs markup tags to classify, group, interrelate, or otherwise provide metadata regarding data stored in XML documents. An XML query can be viewed as producing streams of sequences of items. In a tabular notation using one column, each separately processed sequence comprising an XML item or a concatenation of XML items is suitably viewed as a row, while each XML item within a row is suitably viewed as an XML item, fragment, or row element. Such terminology is used herein to provide a convenient tabular visualization of the data; however, the use of such terminology does not imply that the data is stored or processed in tabular format.
In typical query processing, an XML query is constructed by a user, a software module, or the like, and is converted from a textual format to a data flow model. At the data flow model level, query rewrites of identified inefficient data flow structures are performed to optimize the data flow model. A query rewrite is a query transformation that produces a more efficient query without changing the query output. The optimized data flow model is compiled into executable instructions. Optionally, query rewriting is performed at a lower level than the data flow model, such as at the executable instructions level. It will be appreciated that the executable instructions substantially correspond to a data flow model written in a specific executable instruction set. Similarly, the query text substantially corresponds to a data flow model written in a high level text-based language, and so optionally query rewrites are performed at the text query level. Typically, however, the XML query is converted into an intermediate data flow model which formats the XML query in a manner well-suited for efficient XML query rewrite processing.
Regardless of the processing level at which query rewrites are performed, the query rewrites perform various optimization tasks such as reducing XML item sequence construction, reducing usage of memory for temporary data storage, promoting data flow pipelining, improving index usage and improving I/O behavior. For XQuery, heuristics may be applied, such as “try to express the whole query with as few FLOWR expressions as possible” or “apply filters and extractions early during data processing” in the form of rewrite rules (or rewrites). However, there are problems which arise when queries are rewritten in order to evaluate XPath expressions as early as possible in order to filter out unwanted items in a more efficient way.
Some of these problems are associated with a query rewrite technique known as extraction push down of Xpath in which an XPath extraction is moved/pushed into a lower (closer to base table) query block without changing the semantics of the query, but possibly changing the result of the lower query block. One example is known as the multiple consumer problem, which occurs when an extraction pushdown cannot be performed since other parts of the same query refer to the result of the lower query block that it is being pushed into.
Accordingly, there is a need for systems and methods for increasing the efficiency of the processing of XML queries. There is also a need for systems and methods for rewriting queries to filter out unwanted items more efficiently while avoiding issues such as the multiple-consumer problem.
To overcome the limitations in the prior art briefly described above, the present invention provides a method, computer program product, and system for querying an XML document.
In one embodiment of the present invention an XML query rewrite method comprises identifying an XPath within an XML query; and transforming the XPath into a pre-filter of the XML query.
In another embodiment of the present invention, an XML query rewrite method for processing an XML query comprises: identifying an XPath within the XML query; transforming the XPath into a pre-filter; executing the pre-filter; and executing the XML query after the execution of the pre-filter.
In a further embodiment of the present invention an XML compilation processor comprises: an execution compiler transforming an XML query into an executable XML query plan; a query rewrite processor performing query transformation on the XML query, the query transformations including transforming an XPath within the XML Query into a pre-filter; and transforming the XML query into a transformed XML query including the pre-filter.
In an additional embodiment of the present invention comprises an article of manufacture for use in a computer system tangibly embodying computer instructions executable by the computer system to perform process steps for querying an XML document, the process steps comprising: identifying an XPath within the XML query; transforming the XPath into a pre-filter; executing the pre-filter; and executing the XML query after the execution of the pre-filter.
Various advantages and features of novelty, which characterize the present invention, are pointed out with particularity in the claims annexed hereto and form a part hereof. However, for a better understanding of the invention and its advantages, reference should be made to the accompanying descriptive matter together with the corresponding drawings which form a further part hereof, in which there is described and illustrated specific examples in accordance with the present invention.
The present invention is described in conjunction with the appended drawings, where like reference numbers denote the same element throughout the set of drawings:
The present invention overcomes the problems associated with the prior art by teaching a system, computer program product, and method for efficient processing of XML queries using pre-filters. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. Those skilled in the art will recognize, however, that the teachings contained herein may be applied to other embodiments and that the present invention may be practiced apart from these specific details. Accordingly, the present invention should not be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features described and claimed herein. The following description is presented to enable one of ordinary skill in the art to make and use the present invention and is provided in the context of a patent application and its requirements.
The present invention provides a system and method of using pre-filters in an XQuery processor. The invention addresses problems which arise when queries are rewritten in order to evaluate XPath expressions as early as possible in order to filter out unwanted items in a more efficient way. The invention does this by executing a pre-filter before the execution of the original XPath expression. The execution of the pre-filter removes unwanted data early on. As a result, other costly operations are performed on a reduced set of data and run-time performance may be significantly improved. Also, the techniques of the present invention may improve index usage.
The textual XML query 14 is preferably constructed in accordance with XQuery, which is a commonly used XML query language. However, other XML query constructions can be employed in which XML documents are processed as data streams of sequences of XML fragments or XML items. In XQuery, for example, data streams of XML items and sequences of items are constructed using FLWOR expressions in which FOR clauses produce data streams of individual XML items, LET clauses produce concatenated sequences of XML items, combination of FOR and LET clauses cause a Cartesian product of the produced streams, WHERE clauses provide filtering based on predicate expressions, and the RETURN clause causes the computation of the result that is flowed to a subsequent operation. The FLWOR expression is used as an example herein; other XQuery expressions also can create LET and FOR data flow, henceforth called LET and FOR quantifiers. Example data flow stream transformations are described herein with reference to the preferred XQuery syntax; however, these exemplary data flow stream transformations are readily adapted to other XML query languages that may employ different syntax, clause names, or the like for producing data streams of single XML items, concatenated sequences of XML items, or combinations thereof.
The textual XML query 14 is processed by a compilation processor 20 to produce an executable XML query plan 22, which, when executed, accesses the Internet service 11 to receive, sort, search, transmit, or otherwise process data content from or to the Internet service 11, accesses the XML documents 12 to store, read, sort, search, or otherwise process data content of the XML documents 12, or otherwise interacts with a source or repository of XML items.
In a read operation, an XML items collection engine 24 formats a data stream of XML items extracted from the Internet service 11, the XML documents 12, or other source of XML items into a query data model or other suitable format that is received by the executing XML query plan 22. In a write operation, the XML items collection engine 24 formats a stream of XML items output from the executing XML query plan 22 into a format suitable for receipt by the Internet service 11, for storage in the XML documents 12, or for storage or receipt by another repository of XML items. Rather than storing or transmitting the output of the executing XML query plan 22, the output can be displayed on a display 26 or otherwise communicated to a user.
More specifically, the compilation processor 20 includes various components for transforming and optimizing the textual XML query 14 to produce the executable XML query plan 22. In the exemplary illustrated embodiment, a textual XML query parser 30 parses the textual XML query 14 to produce a parsed XML query from which a data flow modeler 32 produces an equivalent data flow model. The data flow model format is convenient for performing query rewrite and optimization of the XML query prior to conversion to executable code.
The data flow model is analyzed by a query rewrite processor 34 which employs heuristic, cost-based, or other methods to identify query rewrites that improve or are likely to improve efficiency of the XML query. The query rewrite processor 34 performs identified query rewrites if it can determine that the rewrite is safe, that is, if it can determine the making the rewrite will not alter the output of the query.
In accordance with embodiments of the invention, a pre-filter 35 determines an expression that will act as a pre-filter for the XML Query. The function of the pre-filter is based on the consideration that, given almost any XPath inside of XQuery, one can determine an expression which will act as a pre-filter. By annotating the original query with this pre-filter, the semantic remains unchanged. Given some of the unique properties of the pre-filter 35, the query rewrite processor 34 can apply certain rewrites in a much more powerful way. This enables better index usage and better execution behavior.
The concept of the pre-filter 25 begins with an observation in XPath:
$i[e1]/e2==$i[e1][e2]/e2 (1)
where e1 is a filter and e2 is a deterministic XQuery expression. Note that e1 is not required to be present leading to $i/e2==$i[e2]/e2).
Now we can express $i[e1]/e2 as:
$i[e1]/e2===>
1. for $imp in $i[e1][e2]return $imp/e2, if $i is provably singleton or
2. let $return :=$i[e1]/e2 where fn:exists($i[e1][e2])], otherwise. (2)
Hence, this gives us a pre-filter $i[e1][e2] which can be optionally applied. Also, note that
fn:exists($i[e1]/e2)===>fn:exists($i[e1][e2]) (3)
is a correct rewrite using just the pre-filter and dropping the original expression. The significance of expressions I, 2 and 3 is that the pre-filter is an XPath of the form $i[filter] which enables a far more aggressive push down behavior since the pre-filter will only cause $i to be discarded. In other words, it does not change the items being flowed.
Consider the following example (omitting other rewrites for simplicity):
Note that we cannot push $i/b[c=10]nor $i/d[e>5] into the nested FLOWR since both of the two would change the data flow the other expression is dependent upon. However, using a pre-filter as described above one can rewrite the query into (with subsequent other rewrites applied):
thus enabling potential index usage. Also, other rewrites might also benefit from this.
Expression 3 above allows us to rewrite fn:exists queries and XMLEXISTS queries into the form of fn:exists($i[fn:exists([e1]/e2)]) which allows a much better early-out behavior since we do not have to construct any items dictated by e2.
In another example consider the XML Query:
$i/person/phone[ac=408]==
$i[person/phone[ac=408]]/person/phone[ac=408] (4)
In expression (4) by using the pre-filter
person/phone[ac=408]
all phone numbers not having 408 as the area code are filtered out. This may be done using an index. Without the pre-filter all area codes would have to be navigated. By removing the unwanted area codes the performance is significantly improved. The performance improvement can be significant because, between the execution of the pre-filter and the original expression, there may be other costly operations which benefit from a reduction of data being flowed early on. In addition, this is accomplished without raising the multiple consumer problem.
Referring again to
With continuing reference to
Once the query rewrite processor 34 determines that further query rewrite operations are not indicated, the data flow model, as modified by processing performed by the query rewrite processor 34, is input to a query optimizer 40. The query optimizer 40 performs optimizations based on a costing analysis of physical operators of the data flow model to provide greater XML query efficiency. An execution compiler 42 compiles the data flow model after optimization into the executable XML query plan 22 which is suitable for execution.
The components of the XML query compilation processor 20 are exemplary only. Those skilled in the art can readily modify the described XML query compilation processor 20 to suit specific XML query syntaxes or languages, specific applications, specific database management system configurations, and the like. For example, if the XML query is provided to the XML query compilation processor in a non-textual format rather than as the textual XML query 14, then the text parser 30 is suitably omitted.
In another contemplated embodiment, the data flow modeler 32 is omitted, in which case the query rewrite processing is suitably performed on the parsed XML query or on the executable XML query plan. Alternatively, a plurality of data flow modeling processors can be employed to implement a plurality of data flow model formats. For example, another data flow modeler is optionally inserted between the query rewrite processor 34 and the query optimizer 40 to optimize the data flow model format for processing by the query optimizer 40.
Still further, although the XML query rewrite processor 34 is shown in
The compilation processor 20 is typically implemented using one or more computer programs, each of which executes under the control of an operating system, such as OS/2, Windows, DOS, AIX, UNIX, MVS, or so forth, and causes a computer (not shown) to perform the desired XML query compilation processing as described herein. Thus, using the present specification, the compilation processor 20 may be implemented as a machine, process, or article of manufacture by using standard programming and/or engineering techniques to produce software, firmware, hardware or any combination thereof.
The invention can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both hardware and software elements. In a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.
Furthermore, the invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Examples of a computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk. Current examples of optical disks include compact disk-read only memory (CD-ROM), compact disk-read/write (CD-R/W) and DVD.
A data processing system suitable for storing and/or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.
Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers.
Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.
The computer system can include a display interface 64 that forwards graphics, text, and other data from the communication infrastructure 62 (or from a frame buffer not shown) for display on the display unit 66. The computer system also includes a main memory 68, preferably random access memory (RAM), and may also include a secondary memory 70. The secondary memory 70 may include, for example, a hard disk drive 72 and/or a removable storage drive 74, representing a floppy disk drive, a magnetic tape drive, an optical disk drive, etc. The removable storage drive 74 reads from and/or writes to a removable storage unit 76 in a manner well known to those having ordinary skill in the art. Removable storage unit 76, represents a floppy disk, a compact disc, magnetic tape, optical disk, etc. which is read by and written to by removable storage drive 74. As will be appreciated, the removable storage unit 76 includes a computer readable medium having stored therein computer software and/or data.
In alternative embodiments, the secondary memory 70 may include other similar means for allowing computer programs or other instructions to be loaded into the computer system. Such means may include, for example, a removable storage unit 78 and an interface 80. Examples of such may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM, or PROM) and associated socket, and other removable storage units 78 and interfaces 80 which allow software and data to be transferred from the removable storage unit 722 to the computer system.
The computer system may also include a communications interface 82. Communications interface 82 allows software and data to be transferred between the computer system and external devices. Examples of communications interface 82 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, etc. Software and data transferred via communications interface 82 are in the form of signals which may be, for example, electronic, electromagnetic, optical, or other signals capable of being received by communications interface 82. These signals are provided to communications interface 82 via a communications path (i.e., channel) 84. This channel 84 carries signals and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, and/or other communications channels.
In this document, the terms “computer program medium,” “computer usable medium,” and “computer readable medium” are used to generally refer to media such as main memory 68 and secondary memory 70, removable storage drive 74 and a hard disk installed in hard disk drive 72. These computer program products are means for providing software to the computer system. The computer readable medium allows the computer system to read data, instructions, messages or message packets, and other computer readable information from the computer readable medium. The computer readable medium, for example, may include non-volatile memory, such as a floppy disk, ROM, flash memory, disk drive memory, a CD-ROM, and other permanent storage. It is useful, for example, for transporting information, such as data and computer instructions, between computer systems. Furthermore, the computer readable medium may comprise computer readable information in a transitory state medium such as a network link and/or a network interface, including a wired network or a wireless network, which allows a computer to read such computer readable information.
Computer programs (also called computer control logic) are stored in main memory 68 and/or secondary memory 70. Computer programs may also be received via communications interface 82. Such computer programs, when executed, enable the computer system to perform the features of the present invention as discussed herein. In particular, the computer programs, when executed, enable the processor 60 to perform the features of the computer system. Accordingly, such computer programs represent controllers of the computer system.
In accordance with the present invention, we have disclosed systems and methods for creating pre-filters to achieve more efficient Query processing. Those of ordinary skill in the art will appreciate that the teachings contained herein can be implemented using many kinds of software and operating systems, including, but not limited to, XML-enabled database systems such as relational databases and native XML databases as well as middleware such as query gateways, federation, and information integration. References in the claims to an element in the singular is not intended to mean “one and only” unless explicitly so stated, but rather “one or more.” All structural and functional equivalents to the elements of the above-described exemplary embodiment that are currently known or later come to be known to those of ordinary skill in the art are intended to be encompassed by the present claims. No claim element herein is to be construed under the provisions of 35 U.S.C. section 112, sixth paragraph, unless the element is expressly recited using the phrase “means for” or “step for.”
While the preferred embodiments of the present invention have been described in detail, it will be understood that modifications and adaptations to the embodiments shown may occur to one of ordinary skill in the art without departing from the scope of the present invention as set forth in the following claims. Thus, the scope of this invention is to be construed according to the appended claims and not limited by the specific details disclosed in the exemplary embodiments.
Number | Name | Date | Kind |
---|---|---|---|
7096224 | Murthy et al. | Aug 2006 | B2 |
7162485 | Gottlob et al. | Jan 2007 | B2 |
7165063 | Beyer et al. | Jan 2007 | B2 |
7406478 | Agrawal et al. | Jul 2008 | B2 |
7574652 | Lennon et al. | Aug 2009 | B2 |
7698341 | Rubendall | Apr 2010 | B2 |
7752212 | Heifets et al. | Jul 2010 | B2 |
7809734 | Itoh et al. | Oct 2010 | B2 |
7827177 | Murthy | Nov 2010 | B2 |
20020107973 | Lennon et al. | Aug 2002 | A1 |
20020152267 | Lennon | Oct 2002 | A1 |
20030018607 | Lennon et al. | Jan 2003 | A1 |
20030115065 | Kakivaya et al. | Jun 2003 | A1 |
20040010752 | Chan et al. | Jan 2004 | A1 |
20040015783 | Lennon et al. | Jan 2004 | A1 |
20040044659 | Judd et al. | Mar 2004 | A1 |
20040068487 | Barton et al. | Apr 2004 | A1 |
20040148278 | Milo et al. | Jul 2004 | A1 |
20040181543 | Wu et al. | Sep 2004 | A1 |
20040267760 | Brundage et al. | Dec 2004 | A1 |
20050022115 | Baumgartner et al. | Jan 2005 | A1 |
20050050066 | Hughes | Mar 2005 | A1 |
20050060647 | Doan et al. | Mar 2005 | A1 |
20050097084 | Balmin et al. | May 2005 | A1 |
20050198065 | Eppley et al. | Sep 2005 | A1 |
20050203957 | Wang et al. | Sep 2005 | A1 |
20050228768 | Thusoo et al. | Oct 2005 | A1 |
20050229158 | Thusoo et al. | Oct 2005 | A1 |
20050257201 | Rose et al. | Nov 2005 | A1 |
20050273772 | Matsakis et al. | Dec 2005 | A1 |
20050289125 | Liu et al. | Dec 2005 | A1 |
20060026168 | Bosworth et al. | Feb 2006 | A1 |
20060030292 | Bosworth et al. | Feb 2006 | A1 |
20060031228 | Bosworth et al. | Feb 2006 | A1 |
20060031256 | Bosworth et al. | Feb 2006 | A1 |
20060031264 | Bosworth et al. | Feb 2006 | A1 |
20060053368 | Bosworth et al. | Mar 2006 | A1 |
20060101058 | Chidlovskii | May 2006 | A1 |
20060117073 | Bosworth et al. | Jun 2006 | A1 |
20060167928 | Chakraborty et al. | Jul 2006 | A1 |
20060173985 | Moore | Aug 2006 | A1 |
20060218194 | Yalamanchi | Sep 2006 | A1 |
20060224627 | Manikutty et al. | Oct 2006 | A1 |
20060235839 | Krishnaprasad et al. | Oct 2006 | A1 |
20060242115 | Baras et al. | Oct 2006 | A1 |
20060265489 | Moore | Nov 2006 | A1 |
20060265689 | Kuznetsov et al. | Nov 2006 | A1 |
20070016604 | Murthy et al. | Jan 2007 | A1 |
20070034763 | Kitazawa | Feb 2007 | A1 |
20070038985 | Meijer et al. | Feb 2007 | A1 |
20070055658 | Hsiao et al. | Mar 2007 | A1 |
20070061266 | Moore et al. | Mar 2007 | A1 |
20070061393 | Moore | Mar 2007 | A1 |
20070061487 | Moore et al. | Mar 2007 | A1 |
20070083807 | Shaudys et al. | Apr 2007 | A1 |
20070106536 | Moore | May 2007 | A1 |
20070106537 | Moore | May 2007 | A1 |
20070106750 | Moore | May 2007 | A1 |
20070106751 | Moore | May 2007 | A1 |
20070106752 | Moore | May 2007 | A1 |
20070106753 | Moore | May 2007 | A1 |
20070106754 | Moore | May 2007 | A1 |
20070116036 | Moore | May 2007 | A1 |
20070116037 | Moore | May 2007 | A1 |
20070168461 | Moore | Jul 2007 | A1 |
20080005086 | Moore | Jan 2008 | A1 |
20080040151 | Moore | Feb 2008 | A1 |
20080040657 | Kuznetsov et al. | Feb 2008 | A1 |
20080065590 | Castro et al. | Mar 2008 | A1 |
20080065674 | Liu et al. | Mar 2008 | A1 |
20080072306 | Krasnoiarov et al. | Mar 2008 | A1 |
20080097959 | Chen et al. | Apr 2008 | A1 |
20080097974 | Chen et al. | Apr 2008 | A1 |
20080120321 | Liu et al. | May 2008 | A1 |
20080133465 | Lee et al. | Jun 2008 | A1 |
20080140645 | Denoual | Jun 2008 | A1 |
20080154868 | Ozcan et al. | Jun 2008 | A1 |
20080183674 | Bush et al. | Jul 2008 | A1 |
20080189303 | Bush et al. | Aug 2008 | A1 |
20080196016 | Todd | Aug 2008 | A1 |
20080222087 | Balmin et al. | Sep 2008 | A1 |
20080222187 | Beyer et al. | Sep 2008 | A1 |
20080294604 | Goswami | Nov 2008 | A1 |
20080320382 | Kuznetsov et al. | Dec 2008 | A1 |
20090006314 | Balmin et al. | Jan 2009 | A1 |
20090006316 | Fan et al. | Jan 2009 | A1 |
20090006447 | Balmin et al. | Jan 2009 | A1 |
20090037379 | Bou-Diab et al. | Feb 2009 | A1 |
20090055432 | Smith et al. | Feb 2009 | A1 |
20090063401 | Harbarth et al. | Mar 2009 | A1 |
20090063533 | Ting | Mar 2009 | A1 |
20090083239 | Colgrave et al. | Mar 2009 | A1 |
20090210631 | Bosworth et al. | Aug 2009 | A1 |
20090300656 | Bosworth et al. | Dec 2009 | A1 |
20100070448 | Omoigui | Mar 2010 | A1 |
Number | Date | Country |
---|---|---|
2007034763 | Feb 2007 | JP |
Number | Date | Country | |
---|---|---|---|
20090210383 A1 | Aug 2009 | US |