Patent Grant
8572125
This application is related to application Ser. No. 11/161,908, entitled “XML Sub-Document Versioning Method in XML Databases Using Record Storages,” filed Aug. 22, 2005, and application Ser. No. 11/209,997, entitled “Packing Nodes Into Records to Store XML XQuery Data Model and Other Hierarchically Structured Data,” filed Aug. 22, 2005, both of which are assigned to the assignee of the present application.
The present invention relates to the storage of XML data in relational tables, and more particularly to the storage of native XML column data in relational tables.
When eXtensible Mark-up Language (XML) is stored in a relational table, it becomes a column type. How to store the XML data to achieve scalability is a challenging problem to solve. The conventional approach is to store the XML data within the base table as a VARCHAR (variable-length character) or CLOB (Character Large Object) column. The textual XML format stored in these column types does not work well for both XML queries and update, since textual XML needs to be parsed every time, an expensive operation during querying, and only whole document replacement can be used for update in general.
Another conventional stored XML format is based on object relational data model by decomposing the XML into a relational or object model, and store nodes and edges as rows or objects. Due to the large number of entries generated for documents and joins needed for XML queries, this approach is not scalable. In addition, there may be difficulties in supporting proper concurrency control with the decomposed data, such as locking a subtree.
Yet another approach to store XML data is storing XML-specific binary data, such as token stream or hierarchical data model, into a BLOB (Binary Large Object). This format can speed up the queries, but not update due to LOB model restriction. Unfortunately, extending LOB operation model to support flexible partial update without affecting references in indexes is not easy since XML indexes will reference the position in the LOBs.
Accordingly, there exists a need for a scalable storage scheme for native XML column data of relational tables. The storage scheme should store XML data such that scalability is increased and queries are more efficient. The present invention addresses such a need.
A method and system for providing a scalable storage scheme for hierarchically structured data (such as XML data) for relational tables have been disclosed. The scheme includes a base table with a DocID column shared by all hierarchically structured data columns, indicator columns for each hierarchically structured data with information pertaining to the data that is stored, data tables for storing the actual hierarchically structured data, and node identifier indexes corresponding to the data tables for mapping between the indicator columns and the hierarchically structured data in the data tables. In an embodiment, the actual corresponding to each indicator column is stored in a separate data table, and each data table has a separate node identifier index. The node identifier index is searched with a key containing the document identifier and a logical node identifier, and a record identifier of a record in the data table containing the node assigned the logical node identifier is retrieved. Row clustering, versioning, table partitioning, and locking can be reliably provided with this scheme as well.
The present invention provides a scalable storage scheme for native XML column data of relational tables. The following description is presented to enable one of ordinary skill in the art to make and use the invention and is provided in the context of a patent application and its requirements. Various modifications to the preferred embodiment will be readily apparent to those skilled in the art and the generic principles herein may be applied to other embodiments. Thus, the present invention is not intended to be limited to the embodiment shown but is to be accorded the widest scope consistent with the principles and features described herein.
To more particularly describe the features of the present invention, please refer to
Unlike conventional approaches, the XML columns 104-105 do not contain the XML data itself. Instead, internal XML tables 102, 111 are provided for storing the nodes for the XML data for the XML columns 104-105 in the base table 101, via step 202. In this embodiment, one internal XML table 102, 111 is created for each XML column 104-105, respectively. An XML column is represented by an indicator column used to keep a null flag and other state information, such as version number for the XML data if versioning is used (described further below). An internal XML table in accordance with the present invention is a segmented table space, which provides efficient storage management. It stores XML data as a table with a DocID column 106, 113, a minimum NodeID (minNodelD) column 107, 114, and an XML data column 108, 115. The XML data column 108, 115 contains the representations of the nodes of the XML document tree, stored as described in co-pending U.S. patent application, titled “Packing Nodes into Records to Store XML XQuery Data Model and Other Hierarchically Structured Data”, filed on Aug. 22, 2005, and assigned to the assignee of the present application. Applicant hereby incorporates this patent application by reference. The DocID and MinNodelD together provide clustering for the rows in the internal XML table 102 to increase search efficiency, and also provide document ordering for the rows.
To link the base table 101 to the XML data in the internal XML tables 102, 111, NodeID Indexes 110, 112 are provided for mapping between the XML columns 104-105 and the XML data 108, 115 in the internal XML tables 102, 111, via step 203. A NodeID index 110, 112 is created for each internal XML table 102, 111. The NodeID indexes 110, 112 provide mappings between logical node ID's, assigned to each node in the XML document, to a RID of a record in the internal XML table 102, 111 that contains the node. To get to the exact node, traversal of the nodes in the XML data 108, 115 column is performed.
Internal XML Table Partitioning
If the base table 101 is a partitioned table, several possible partitioning mechanisms can be used for the internal XML tables 102, 111. In a first partitioning mechanism, there is an internal XML table partition for each base table partition. Here, there is no simple logic to determine which partition a document is in a given DocID. Thus, the NodeID index 110, 112 is a non-partitioned secondary index on the internal XML table 102, 111. The advantage of this mechanism is that when the base table partitions are changed, the internal XML table partitions can follow without searching for individual XML documents corresponding to the base table partitions.
In a second partitioning mechanism, the internal XML table partitioning is independent of the base table partitioning. If range partition is used, then the sequence value for the DocID column 106, 113 is scrambled to achieve randomness. Alternatively, hash partitioning based on the DocID can be used. Under these partitioning schemes, the partition to which an XML document belongs, given a DocID, can be determined by the partitioning logic (either range or hash). The NodeID index 110, 112 can be a data partitioned secondary index or non-partitioned index. However, if there are value indexes, then they are non-partitioned secondary index. Partitioning can even be based on the DocID and NodeID together to partition large XML documents. When the NodeID is involved in partitioning, range partitioning is preferred since the XML document order is important for XPath and XQuery evaluation. XPath and XQuery evaluations are known in the art and will not be described here.
In a third partitioning mechanism, if the base table is partitioned by range, then the internal XML table will be partitioned by range also. However, the range will be determined by the base table row. For example, if the base table row is in partition 2 based on the portioning key, then the XML document will be inserted in partition 2 also.
If the base table is partitioned by growth then the internal XML table will be partitioned by growth as well. In this case, the partition of the base table row is unrelated to the partition of the XML document rows. Since the nodeID index is an NPI, the document can be found based on the nodeID index key. In this case the base table and the XML internal table partitions overflow to new partitions independently.
Storing XML Data In-line with Base Table
For small XML documents, it may be inefficient to store the XML data separately from the base table. In this situation, the XML data can be stored in-line in the base table 101. The node record format used in this embodiment, described in the co-pending patent application referenced above, is in-line ready. Thus, in-lining the XML data in the base table 101 can be easily achieved, when the XML columns are treated as VARBINARY data. In complete in-lining, all of the XML data are in the base table columns, and there is no need for the DocID index 109 or the NodeID indexes 110, 112. However, there is a size limit for in-lined columns due to page size limit, and the largest size will be the largest page size minus the overhead and size of other columns.
In an automatic in-lining scheme, an XML document is in-lined if its size is within the limit, or otherwise will be stored in the internal XML table. To locate an XML document, especially from a value index, the table space information is kept in the NodeID index. At least a flag is needed to tell whether an XML document is in-lined in the base table space, or in the internal XML table space for the RID. This flag can be part of an extended RID for XML value indexes. Alternatively, for XML value indexes, only the DocID and the NodeID are kept. The DocID/NodeID conversion to RID is then performed by the NodeID index. For automatic in-lining based on size, the internal XML table space and the base table space are not processed independently by utilities, but are rather processed as one unit.
Locking
In the locking mechanism for XML data in this embodiment, the XML table space level locks will be acquired when XML data is accessed. XML column level locks will be acquired when specific XML data columns are accessed, and the individual XML data rows and pages containing the XML data are locked to take advantage of existing DB2 space management scheme. XML column locks are acquired when the base table page/row locks are acquired. Access to the XML pages will be serialized by the normal page latch mechanism. The freshness of the XML page will be controlled by the normal page buffer management.
Uncommitted readers will also lock XML columns for the duration for which the data is being serialized out. If the result set is placed in a work file, the XML lock will be held until the work file is deleted, otherwise, the XML lock will be release as soon as the position is moved to the next row. ISO RR and RS readers will not acquire XML locks unless the result set is placed in a workfile. Lock avoiders will get conditional xml locks to ensure that XML documents are completely inserted.
Versioning
In many applications, XML documents are read-only or document update is very infrequent. Also, the pessimistic locking approach for concurrency control is sometimes too expensive in case of small XML documents. In these situations, multiple versioning is more efficient.
In multi-versioning concurrency control, when a new document replaces the old document, the old document is not deleted. Rather, a new version is created. Any references to an old version will be guaranteed to be valid as the old document is available. To support multi-versioning concurrency control, the XML indicator columns, XML table rows, and indexes all include a version number. Any deferred fetch of XML data will contain a version number in addition to the DocIDs.
Two scenarios require deferred access to the XML data. One scenario is the materialization of base table columns due to sort for GROUP BY or ORDER BY, where only references to the XML column data are materialized to a work file. The references include the DocID and a version number. In another scenario, value indexes are used for predicate evaluation on an XML column. The result is a DocID and a version number that makes the predicate true. To get the XML document with the DocID and a given version number, the NodeID index is enhanced as described below.
Without versioning, a NodeID index contains the key (DocID, NodeID), which is mapped to a RID of the internal XML table. The search is for a given (DocID, NodeID) pair, to find the first index entry that is greater than or equal to this pair. With a version number, an entry with an equal version number needs to be found. Since index searches uses a greater-than-or-equal comparison, the version number is the first key component with a reverse (descending) order (with version number bits flipped), i.e., the key will be (VerNum, DocID, NodeID). This way, if a ‘0’ is supplied as the version number, the latest version will be found. Here, a version number must be an existing one. Otherwise, using the NodeID index will find the beginning of the latest version that is older than the given version number. A continuation search is needed to locate a record with a given NodeID in the middle of an XML document using the correct version number.
Here, the base table and XML value indexes only contain the latest version. The NodeID index contains old versions as well as the latest version. Removal of old versions can be achieved by a background task. The background task can be triggered based on the ratio of the number of old version rows versus the total number of rows in the internal XML table since the last cleanup cycle. An alternative is to keep the latest versions only with the NodeID index, which may not contain a version number as a key, and move old versions to a temporary table with its own NodeID index. If a version cannot be found in the normal XML table, the temporary table is searched for an old version.
A method and system for providing a scalable storage scheme for native hierarchically structured data of relational tables have been disclosed. The scheme includes a base table with indicator columns with information pertaining to hierarchically structured data of a document, data tables for storing the hierarchically structured data corresponding to the indicator columns, and node identifier indexes corresponding to the data tables for mapping between logical node IDs and physical record IDs, and also the indicator columns and the hierarchically structured data in the data tables. In an embodiment, actual data corresponding to each indicator column is stored in a separate data table, and each data table has a separate node identifier index. The node identifier index is searched with a key containing the document identifier and a logical node identifier, and a record identifier of a record in the data table containing the node assigned the logical node identifier is retrieved. Row clustering, versioning, table partitioning, and locking can be reliably provided with this scheme as well.
Although the present invention has been described in accordance with the embodiments shown, one of ordinary skill in the art will readily recognize that there could be variations to the embodiments and those variations would be within the spirit and scope of the present invention. Accordingly, many modifications may be made by one of ordinary skill in the art without departing from the spirit and scope of the appended claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4794528 | Hirose et al. | Dec 1988 | A |
| 5151697 | Bunton | Sep 1992 | A |
| 5608904 | Chaudhuri et al. | Mar 1997 | A |
| 5883589 | Takishima et al. | Mar 1999 | A |
| 6058397 | Barrus et al. | May 2000 | A |
| 6263332 | Nasr et al. | Jul 2001 | B1 |
| 6295526 | Kreiner et al. | Sep 2001 | B1 |
| 6313766 | Langendorf et al. | Nov 2001 | B1 |
| 6353820 | Edwards et al. | Mar 2002 | B1 |
| 6381605 | Kothuri et al. | Apr 2002 | B1 |
| 6510434 | Anderson et al. | Jan 2003 | B1 |
| 6539369 | Brown | Mar 2003 | B2 |
| 6539396 | Bowman-Amuah | Mar 2003 | B1 |
| 6563441 | Gold | May 2003 | B1 |
| 6587057 | Sheuermann | Jul 2003 | B2 |
| 6610104 | Lin et al. | Aug 2003 | B1 |
| 6647391 | Smith et al. | Nov 2003 | B1 |
| 6810414 | Brittain | Oct 2004 | B1 |
| 6889226 | O'Neil et al. | May 2005 | B2 |
| 6915304 | Krupa | Jul 2005 | B2 |
| 6985898 | Ripley et al. | Jan 2006 | B1 |
| 7072904 | Najork et al. | Jul 2006 | B2 |
| 7080065 | Kothuri et al. | Jul 2006 | B1 |
| 7133865 | Pedersen et al. | Nov 2006 | B1 |
| 7246138 | McCauley et al. | Jul 2007 | B2 |
| 7274671 | Hu | Sep 2007 | B2 |
| 7293005 | Fontoura et al. | Nov 2007 | B2 |
| 7293028 | Cha et al. | Nov 2007 | B2 |
| 7333982 | Bakalash et al. | Feb 2008 | B2 |
| 7636739 | Stefani et al. | Dec 2009 | B2 |
| 7716253 | Netz et al. | May 2010 | B2 |
| 7792866 | Van Der Linden et al. | Sep 2010 | B2 |
| 20020099715 | Jahnke et al. | Jul 2002 | A1 |
| 20020120679 | Hayton et al. | Aug 2002 | A1 |
| 20020145545 | Brown | Oct 2002 | A1 |
| 20030014397 | Chau et al. | Jan 2003 | A1 |
| 20030023528 | Wilce et al. | Jan 2003 | A1 |
| 20030088639 | Lentini et al. | May 2003 | A1 |
| 20040002939 | Arora et al. | Jan 2004 | A1 |
| 20040044959 | Shanmugasundaram et al. | Mar 2004 | A1 |
| 20040111672 | Bowman et al. | Jun 2004 | A1 |
| 20040128296 | Krishnamurthy et al. | Jul 2004 | A1 |
| 20040167864 | Wang et al. | Aug 2004 | A1 |
| 20040167915 | Sundararajan et al. | Aug 2004 | A1 |
| 20040205638 | Thomas et al. | Oct 2004 | A1 |
| 20050055336 | Hui et al. | Mar 2005 | A1 |
| 20050125431 | Emmick et al. | Jun 2005 | A1 |
| 20050192955 | Farrell | Sep 2005 | A1 |
| 20050210052 | Aldridge | Sep 2005 | A1 |
| 20060004792 | Lyle et al. | Jan 2006 | A1 |
| Number | Date | Country |
|---|---|---|
| 08190543 | Jul 1996 | JP |
| 2001034619 | Feb 2001 | JP |
| 2002269139 | Sep 2002 | JP |
| 2004178084 | Jun 2004 | JP |
| WO 0203245 | Jan 2002 | WO |
| Entry |
|---|
| Elias, “Interval and Recency Rank Souce Coding Two On-Line Adaptive Variable Length Schemes,” IEEE Transactions on Information Theory, V IT-33, N1, Jan. 1987, pp. 3-10. |
| Tadayon et al., “Grouping Algorithm for Lossless Data Compression,” IEEE Computer Society, Data Compression Conference, Mar. 30-Apr. 1, 1998, 15pgs. |
| Grust, “Accelerating XPath Location Steps,” ACM SIGMOD 2002, Jun. 4-6, 2002, Madison, WI, pp. 109-120. |
| Bremer et al., “An Efficient XML Node Identification and Indexing Scheme,” Technical Report CSE-2003-04, Dept. of Computer Science, University of California at Davis, 2003, 14pgs. |
| Bruno et al., “Holistic Twig Joins: Optimal XML Pattern Matching,” ACM SIGMOD 2002, Jun. 4-6, 2002, Madison, WI, pp. 310-321. |
| Li et al., “Indexing and Querying XML Data for Regular Path Expressions,” Proceedings of the 27th International Conference on VLDB, Sep. 11-14, 2001, Rome, Italy, pp. 361-370. |
| Cohen et al., “Labeling Dynamic XML Trees,” Proceedings of the 21st ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, Jun. 4-6, 2002, Madison, WI, pp. 271-281. |
| Jagadish et al., “On Effective Multi-Dimensional Indexing for Strings,” ACM SIGMOD 2000, May 14-19, 2000, Dallas, TX, pp. 403-414. |
| “Technique for Performing Generalized Prefix Matches,” IBM Technical Disclosure Bulletin, V40, N3, Mar. 1997, pp. 189-200. |
| Wang et al. ,“ViST: A Dynamic Index Method for Querying XML Data by Tree Structures,” ACM SIGMOD 2003, Jun. 9-12, 2003, San Diego, CA, pp. 110-121. |
| Guo et al., “XRANK: Ranked Keyword Search over XML Documents,” ACM SIGMOD 2003, Jun. 9-12, 2003, San Diego, CA, pp. 16-27. |
| Garofalakis et al., “XTRACT: A System for Extracting Document Type Descriptors from XML Documents,” ACM SIGMOD 2000, Dallas, TX, V29, N2, pp. 165-176. |
| Fiebig et al., “Anatomy of a Native XML Base Management System,” VLDB Journal, Springer Verlag, Berlin, Germany, V11, 2002, pp. 292-314. |
| Amer-Yahia et al., “Logical and Physical Support for Heterogeneous Data,” CIKM 2002 ACM, McLean, VA, Nov. 4-9, 2002, pp. 270-281. |
| Kanne, Carl-Christian et al., “Efficient Storage of XML Data,” Universität Mannheim, Germany, Jun. 16, 1999, pp. 1-20. |
| Zhang, Guogen (Gene), “XML Data Model Storage,” IBM Confidential, Mar. 17, 2004, pp. 1-10. |
| Zhang, Guogen (Gene), “XML Data Model Storage Advanced Features,” IBM Confidential, Apr. 25, 2004, pp. 1-4. |
| Glance, Natalie S. et al., “Generalized Process Structure Grammars (GPSG) for Flexible Representations of Work,” Proceedings of the 1996 ACM Conference on Computer Supported Cooperative Work, pp. 180-189. |
| Number | Date | Country | |
|---|---|---|---|
| 20070043751 A1 | Feb 2007 | US |
