A relational database management system (RDBMS) stores data in tables, in which the tables are a collection of rows all having the same columns. Each of the columns in the tables maintains a particular type of data for the data that comprise the rows. The Structured Query Language (SQL) is often used to query, access, and manipulate the data contained in the tables. The SQL language includes the set operators Union, Union All, Intersect, Intersect All, Except, and Except All. Complex SQL queries use various combinations of the set operators Union, Intersect, and Except.
Features of the present disclosure are illustrated by way of example and not limited in the following figure(s), in which like numerals indicate like elements, in which:
For simplicity and illustrative purposes, the present disclosure is described by referring mainly to an example thereof. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be readily apparent however, that the present disclosure may be practiced without limitation to these specific details. In other instances, some methods and structures have not been described in detail so as not to unnecessarily obscure the present disclosure.
As used throughout the present disclosure, the term “includes” means includes but not limited to, the term “including” means including but not limited to. The term “based on” means based at least in part on. In addition, the terms “a” and “an” are intended to denote at least one of a particular element.
Disclosed herein are example methods for executing a query having multiple set operators, in which the query may include a mix of set operators. Also disclosed herein are example apparatuses for implementing the example methods and example non-transitory computer readable mediums on which are stored machine readable instructions that implement the example methods. According to an example, the method for executing the query is implemented or invoked in a Vertica™ column-stored database.
An example method for executing a query disclosed herein comprises a set of operations that efficiently eliminate duplicates for set operators without requiring that multiple Group By operations be performed in determining an output to the query. In other words, through implementation of the method disclosed herein, the result to a query may be determined through creation of at most one hash table through a single Group By operation regardless of the number or type of set operators included in the query. By way of example, a Group By operation operates on various groups of rows and returns one row per each of the various groups, for instance, to eliminate duplicate rows from a result table. The set operators include Union, Intersect, and Except.
An example method disclosed herein may, however, also be extended to implement Intersect All and Except All without requiring significant changes to the method. Particularly, the example method may merely be changed to include a change in the number of times each tuple is shown in an output. By way of particular example, in the query select * from TableA Intersect All select * from TableB, the predicate will be the same as the Intersect query, i.e., count(A)>0 AND count(B)>0. However, each tuple that satisfies the above condition has to be outputted min(count(a), count(B)) times. All of the remaining operations will remain the same. In an example in which the set operator is Except All, the predicate will be the same as that of Except, but each tuple that satisfies the above condition has to be outputted max(count(a)−count(B),0) times.
In contrast, a conventional technique for executing a query containing a group of set operators is to perform a Join operation, which only accepts two inputs at a time. In this conventional technique, for n different set operators, n−1 hash tables are required to be built, which may not be possible when the memory into which the hash tables are being built is too small for any of the n−1 hash tables to be stored at a given time. As such, the query may perform poorly if a hash table has to be written to disk multiple times because of insufficient memory to hold any of the hash tables. In addition, this conventional technique depends upon the use of the cost model to select the inner and outer tables. Thus, if the cost model incorrectly selects the inner and outer tables, the resulting Join operation may result in a relatively large hash table, which results in a relatively large consumption of memory resources. The relatively large hash table resulting from this conventional technique may also result in poor performance, for instance, in the case of the Except operator because in case of Except, the hash table may only be created from the right input as a result, if the size of the right input is very large as compared with the left input then the query may perform poorly. Another possible disadvantage of this conventional technique is that in order to eliminate a duplicate input, another hash table is required to be built for a Group By operation apart from the one that was built during the Join operation.
Another conventional technique comprises a technique implemented by Postgres. Under this conventional technique, only two inputs are allowed per operation and thus, a query involving N set operators may require building of N−1 hash tables to determine the output. This technique is bound by the precedence of the set operator query. For queries involving multiple set operators, this conventional technique applies multiple Group By operations, which may be costly. Furthermore, this conventional technique depends on the cost model for selecting the inner and outer tables in the case of Intersect and always selects the left table in the case of Except.
In comparison with conventional query processing techniques, therefore, aspects of the present disclosure enable the result to the query to be determined in a relatively efficient manner and without reliance upon a relatively large memory.
With reference first to
The computing environment 100 is depicted as including a machine 110, a data store 120, a network 130, and a plurality of client devices 140a-140n, in which “n” represents an integer greater than or equal to one. The machine 110, which may comprise a computer, a server, etc., is also depicted as including a database manager 112, a database 114 containing tables 116, and a memory 122 containing query sets 124. The database manager 112 is to manage the database 114, for instance, access and modify data contained in the tables 116 of database 114, execute queries on the data contained in the tables 116, etc. For instance, the database manager 112 is to execute SQL operations involving Union, Intersect, Except, Union All, Intersect All, and Except All set operators. As also shown in
The actual data for the tables 116 may be stored in the data store 120, which comprises volatile and/or non-volatile memory, such as an optical or magnetic media on a hard disk, DRAM, EEPROM, MRAM, phase change RAM (PCRAM), Memristor, flash memory, or the like. Additional tables, for instance, those resulting from set operators, such as Union, Intersect, Except, etc., may also be stored in the data store 120. According to an example, the database 114 comprises a Vertica™ column-stored database. In addition, although the data store 120 has been depicted as being separate from the machine 110, the data store 120 may be integrated with the machine 110 without departing from a scope of the disclosure contained herein.
According to an example, the machine 110 is to receive queries for information contained in the data store 120 from the client devices 140a-140n and to communicate results of query operations to the client devices 140a-140n over the network 130. The client devices 140a-140n comprise one or more of desktop computers, laptop computers, tablet computers, personal digital assistants, smartphones, cellular telephones, etc. In addition, the network 130 comprises an IP network, such as the Internet, a cellular network, a local area network, a wide area network, etc. In addition, or alternatively, queries may be inputted directly through the machine 110.
Turning now to
The machine 110 is depicted as including all of the same components as those in the machine 110 depicted in
The database manager 112 is depicted as including an input/output module 210, a data scanning module 212, a column adding module 214, a Union All operation performing module 216, A Group By operation performing module 218, an aggregation module 220, and a predicate applying module 222. The processor 202 is to invoke or implement the modules 210-222 in performing various operations as discussed in greater detail herein below.
According to an example, the database manager 112 comprises machine readable instructions stored, for instance, in a volatile or non-volatile memory, such as DRAM, EEPROM, MRAM, flash memory, floppy disk, a CD-ROM, a DVD-ROM, or other optical or magnetic media, and the like. In this example, the modules 210-222 comprise modules of machine readable instructions stored in the memory, which are executable by the processor 202. According to another example, the database manager 112 comprises a hardware device, such as a circuit or multiple circuits arranged on a board. In this example, the modules 210-222 comprise circuit components or individual circuits, which the processor 202 is to control. According to a further example, the database manager 112 comprises a combination of modules with machine readable instructions and hardware modules.
Various manners in which the modules 210-222 of the database manager 112 may operate are discussed with respect to the methods 300 and 500 respectively depicted in
With reference first to
According to an example, prior to implementation of the method 300, an input query may be received, for instance, by the input/output module 210. For example, the input/output module 210 receives an input query communicated over the network 130 by a client device 140a. The input query may be transformed to identify the set operators to be applied to a plurality of tables 116 in order to determine an output to the query. In some instances, the transformation of the input query results in a query requiring a mix of set operators to be applied on the plurality of tables. The set operators comprise, for instance, Union, Intersect, and Except. As discussed in greater detail below with respect to
The Union operator produces a result set that combines the tuples of two input result tables (also referred to as relations) of an input query, such that each of the tuples in a result set is in either or both input result tables. The result set may comprise a result of a query on the input result tables and may also comprise a table or a relation. The result set may also be stored on the memory 122 instead of the data store 120, as shown in
In addition, the data contained in a plurality of result tables 116 of an input query are scanned, for instance, by the data scanning module 212. The result tables 116 may be scanned to determine which of the result tables 116 contain data that is relevant to the received input query. In addition, the result tables 116 may be scanned to determine the data contained in the result tables 116. By way of particular example, and with reference to
At block 302, an additional column is added in each input result table of an input query, for instance, by the column adding module 214. The additional column includes an identifier of the input result table in which the data is contained to thus enable identification of the input result table from which particular data has been extracted. The results of a query performed on the tables 116 contained in the database 114, i.e., the input result tables, and the additional columns may be stored in the memory 122. By way of example, as shown in
At block 304, a Union All operation is performed on the data contained in the input result tables to generate an intermediate result set 410 (
At block 306, a Group By operation is performed on the data contained in the intermediate result set, for instance, by the Group By operation performing module 218. Generally speaking, the Group By operation operates on various groups of rows and returns one row per each of the various groups, for instance, to eliminate duplicate rows from a result table. By way of example, the Group By operation returns one value for one group of rows and another value for another group of rows.
At block 308, aggregation columns containing counts of the number of times that each tuple exists in each of the plurality of input result tables are added in the grouped result set, for instance, by the aggregation module 220. That is, the aggregation columns are added to the grouped result set to identify the count of the tuples existing in each of the input result tables. According to an example, the count of the tuples existing in each of the input result tables may be determined from the identifiers listed in the intermediate result set 410. The grouped result set may comprise a table or a relation and may be stored in the memory 122.
At block 310, a predicate is applied on the grouped result set to execute the query, for instance, by the predicate applying module 222. The predicate may be generated through implementation of an operation that is performed to generate the predicate, which is discussed in greater detail herein below. Generally speaking, however, the predicate may be included in a Having clause and is used to filter rows resulting from the Group By clause. By way of particular example, and with reference to
Although the method 300 may end following performance of block 310, the results of the executed query may also be communicated to the client device 140a that requested the query to be performed, for instance, by the input/output module 210.
Turning now to
Generally speaking, implementation of the method 500 results in generation of a predicate to be applied on the grouped result set discussed above with respect to block 310 in
At block 502, a current node in a set operator parse tree is accessed as an input. By way of example, the current node being accessed may comprise the root node labeled “Union”, the node labeled “Intersect”, or the node labeled “Except” in
At block 504, a determination is made as to whether the current node being accessed in the set operator parse tree is a leaf node. That is, a determination is made as to whether the current node has a child node. If the current node is a leaf node, i.e., contains no child nodes, the current node is marked as having been visited, as indicated at block 506. In addition, at block 508, a predicate having a count (A)>0 is created, where A is the column containing the counter for each tuple. (e.g., the aggregation column). In other words, a predicate is created out of the query defined in the current node, in which the count of the query is greater than zero. Following creation of the predicate for the current node at block 508, the parent node of the current node is visited as indicated at block 510. By way of example in which the current node comprises the node “Query1” in
In addition, a determination is made as to whether the parent node comprises a leaf node at block 504. In response to a determination that the parent node is not a leaf node, a determination is made as to whether the left child node of the parent node has been visited, as indicated at block 512. In response to a determination that the left child node has not been visited, at block 514, the left child node is visited. In addition, blocks 504-514 are repeated until a predicate for a left child leaf node is created at block 508. By way of example with respect to the diagram 600 in
Following creation of the predicate for the left child node, the parent node is again accessed at block 510 and blocks 504 and 512 are repeated. In response to a determination that the left child node has been visited at block 512, a determination is made as to whether the right child node of the parent node has been visited at block 518. In response to a determination that the right child node has not been visited, at block 514, the right child node is visited. In addition, blocks 504-518 are repeated until a predicate for a right child leaf node is created at block 508. By way of example with respect to the diagram 600 in
Following access back to the parent node at block 510, and determinations that the left and right child nodes have been visited at blocks 512 and 518, a predicate for the parent node may be created depending upon the type of the parent node. Although the following operations have been depicted as occurring in a particular order, it should clearly be understood that the following operations may be performed in any order without departing from a scope of the present disclosure.
At block 522, a determination may be made as to whether the parent node (i.e., the currently accessed node) is of the type “Union”. In response to a determination that the parent node is of the type “Union”, at block 524, a predicate is created that includes the left child node's predicate OR the right child node's predicate. However, in response to a determination that the parent node is not of the type “Union,” a determination is made as to whether the parent node is of the type “Intersect” at block 526. In response to a determination that the parent node is of the type “Intersect”, at block 528, a predicate is created that includes the left child node's predicate AND the right child node's predicate. However, in response to a determination that the parent node is not of the type “Intersect,” a determination is made as to whether the parent node is of the type “Except” at block 530. In response to a determination that the parent node is of the type “Except”, at block 532, a predicate is created that includes the left child node's predicate AND NOT the right child node's predicate. However, in response to a determination that the parent node is not of the type “Except,” an error out message is reported at block 534. An error out message may be reported, for instance, because an error condition exists if the parent node does not comprise one of the types of nodes discussed above.
Following any of the blocks 524, 528, and 532, a determination is made as to whether the current node comprises a root node of the set operator parse tree at block 536. In response to a determination that the currently accessed node does not comprise a root node, the parent node of the current node is visited at block 538. In addition, blocks 504-538 may be repeated on the parent node. In response to a determination that the current node is a root node, i.e., does not have a parent node, and/or if the error out condition is reached at block 534, the method 500 may end as indicated at block 540.
Following implementation of the method 500 on the leaf nodes and the non-leaf nodes of the set operator parse tree, a number of predicates may have been created at blocks 524, 528, and 532. In this regard, the predicates may be combined according to a higher level parent node, in which the predicates created at blocks 524, 528, and 532 become the predicates of the left and right child nodes of the parent nodes in further iterations of the method 500. Thus, the final predicate is one that is created according to the type of the root node. By way of example with respect to the diagram 600 in
(count(Query1)>0 AND count (Query2)>0) OR (count(Query3)>0 AND NOT (count(Query4)>0), in which count(QueryN) indicates the number of times the tuple occurs in an output for QueryN.
Some or all of the operations set forth in the methods 300 and 500 may be contained as utilities, program, or subprograms, in any desired computer accessible medium. In addition, the methods 300 and 500 may be embodied by or stored as machine readable instructions, which may exist in a variety of forms both active and inactive. For example, they may exist as source code, object code, executable code or other formats. Any of the above may be embodied or stored on a non-transitory computer readable storage medium. Examples of non-transitory computer readable storage media include conventional computer system RAM, ROM, EPROM, EEPROM, and magnetic or optical disks or tapes. It is therefore to be understood that any electronic device capable of executing the above-described functions may perform those functions enumerated above.
Turning now to
The computer readable medium 710 comprises any suitable medium that participates in providing instructions to the processor 702 for execution. For example, the computer readable medium 710 may be non-volatile media, such as memory. The computer-readable medium 710 may also store an operating system 714, such as but not limited to Mac OS, MS Windows, Unix, or Linux; network applications 716; and a database managing application 718. The operating system 714 may be multi-user, multiprocessing, multitasking, multithreading, real-time and the like. The operating system 714 may also perform basic tasks, such as but not limited to recognizing input from input devices, such as but not limited to a keyboard or a keypad; sending output to the display 704; keeping track of files and directories on medium 710; controlling peripheral devices, such as but not limited to disk drives, printers, image capture device; and managing traffic on the bus 712. The network applications 716 include various components for establishing and maintaining network connections, such as but not limited to machine readable instructions for implementing communication protocols including TCP/IP, HTTP, Ethernet, USB, and FireWire.
The database managing application 718 provides various components for executing a query as discussed above with respect to the methods 300 and 500 in
In certain examples, some or all of the processes performed by the application 718 may be integrated into the operating system 714. In certain examples, the processes may be at least partially implemented in digital electronic circuitry, or in computer hardware, machine readable instructions (including firmware and software), or in any combination thereof, as also discussed above.
What has been described and illustrated herein are examples of the disclosure along with some variations. The terms, descriptions and figures used herein are set forth by way of illustration only and are not meant as limitations. Many variations are possible within the scope of the disclosure, which is intended to be defined by the following claims—and their equivalents—in which all terms are meant in their broadest reasonable sense unless otherwise indicated.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/US12/62750 | 10/31/2012 | WO | 00 |