Patent Application
20200201680
The present disclosure relates to optimization of database queries to improve performance of a database system, and is particularly directed to a system, method, and computer-readable medium for iteratively dynamically assigning data objects to a plurality of processing modules of a database system.
A database is a collection of stored data that is logically related and that is accessible by one or more users or applications. A popular type of database system is the relational database management system in which information is represented in the form of relational tables, also referred to as relations, made up of rows and columns (also referred to as tuples and attributes). Each row represents an occurrence of an entity defined by a table, with an entity being a person, place, thing, or other object about which the table contains information.
Another popular type of database system is the object database management system in which information is represented in the form of objects as used in object-oriented programming. Stored data objects can have a wide range of varying attributes, such as object byte size, data compression effectiveness, and data ingest rates. Object database systems are object-oriented, whereas relational database systems are table-oriented.
One of the goals of a database management system is to optimize the performance of database queries for access and manipulation of data stored in the database system. Given a target environment, an optimal query plan is selected, with the optimal query plan being the one with the lowest cost (e.g., response time) as determined by an optimizer. The response time is the amount of time it takes to complete the execution of a database query on a given database system.
In known database systems that can support object storage subsystems, data objects stored in data object servers are pre-assigned to a plurality of processing modules when a database query is received. The plurality of processing modules is connected in parallel to provide a parallel query engine. Each processing module processes its pre-assigned data objects, and has no control over organization or demographics of the data objects.
Since each processing module has no control over organization or demographics of the data objects assigned to it, performance issues (e.g., data skew, selection skew, join skew, and central processing unit skew) are magnified for the parallel query engine. Moreover, since data objects can have a wide range of varying attributes, data sharing complexities are created for the parallel query engine, especially when the parallel query engine is required to shard data objects at query time or metadata definition time. It would be desirable to overcome performance drawbacks of known parallel query engines of database systems.
Disclosed embodiments provide a system, method, and computer-readable medium for iteratively dynamically assigning data objects to a plurality of processing modules of a database system to improve performance of the database system. A database query is received from a client computer system. Data objects are iteratively dynamically assigned from a plurality of data object servers to each processing module based upon a select one of (i) processing activity of the processing module, and (ii) a characteristic of the data objects to be dynamically assigned to the processing module.
Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying figures, in which:
It is to be understood that the following disclosure provides many different embodiments or examples for implementing different features of various embodiments. Specific examples of components and arrangements are described below to simplify the present disclosure. These are, of course, merely examples and are not intended to be limiting.
As shown, the database system 100 includes one or more processing modules 1051 . . . Y that manage the storage and retrieval of data in data-storage facilities 1101 . . . Y. Each of the processing modules 1051 . . . Y may be one or more physical processors or each may be a virtual processor, such as an Access Module Processor (AMP), with one or more virtual processors running on one or more physical processors. For the case in which one or more virtual processors are running on a single physical processor, the single physical processor swaps between the set of virtual processors. For the case in which N virtual processors are running on an M-processor node, the node's operating system schedules the N virtual processors to run on its set of M physical processors. Each of the processing modules 1051 . . . Y manages a portion of a database that is stored in corresponding data-storage facilities 1101 . . . Y. Each of the data-storage facilities 1101 . . . Y includes one or more disk drives or other storage medium.
The database system 100 stores data in one or more tables in the data-storage facilities 1101 . . . Y. The rows 1151 . . . Z of the tables are stored across multiple data-storage facilities 1101 . . . Y to ensure that the system workload is distributed evenly across the processing modules 1051 . . . Y. A parsing engine 120 organizes the storage of data and the distribution of table rows 1151 . . . Z among the processing modules 1051 . . . Y and accesses processing modules 1051 . . . Y via an interconnect 130. The parsing engine 120 also coordinates the retrieval of data from the data-storage facilities 1101 . . . Y in response to queries received from a user, such as one using a client computer system 135 connected to the database system 100 through a communications network connection 125. The database system 100 typically receives queries in a standard format, such as the Structured Query Language (SQL) put forth by the American National Standards Institute (ANSI).
Each of the processing modules 1051 . . . Y is also connected through a communications network connection 145 to an object storage subsystem that includes a number of data object servers 1401 . . . X. Each of the data object servers 1401 . . . X includes one or more disk drives or other storage medium. Each of the data object servers 1401 . . . X stores one or more data objects (i.e., an object store).
The parsing engine 120, on receiving an incoming database query, applies an optimizer component 122 to the query to assess the best plan for execution of the query. Selecting the optimal query-execution plan includes, among other things, identifying which of the processing modules 1051 . . . Y are involved in executing the query and which database tables are involved in the query, as well as choosing which data-manipulation techniques will serve best in satisfying the conditions of the query. Database statistics are used in making these assessments during construction of the query-execution plan. For example, database statistics may be used by the optimizer to determine data demographics, such as attribute minimum and maximum values and data ranges of the database.
In one example system, the parsing engine 120 is made up of three components: a session control 200, a parser 205, and a dispatcher 210, as shown in
In accordance with an aspect of the present disclosure, a control task module 150 is connected through the communications network connection 145 to the data object servers 1401 . . . X. Although the control task module 150 is shown in
When a database query is received from the client computer system 135, the control task module 150 dynamically assigns data objects to the processing modules 1051 . . . Y. More specifically, the control task module 150 is arranged to iteratively dynamically assign data objects from the data object servers 1401 . . . X to each processing module of the plurality of processing modules 1051 . . . Y based upon a select one of (i) processing activity of the processing module, and (ii) a characteristic of the data objects to be dynamically assigned to the processing module.
The iterative dynamic assigning of data objects is performed incrementally, and is embedded in the database query. The result is smaller chunks of unit-of-work being handed out to each of the processing modules 1051 . . . Y during the data pipeline flow of the particular database query. As such, the database query is being enhanced or optimized based upon performance of the control task module 150.
In some embodiments, the processing activity of each processing module during the database query comprises random processing activity associated with the processing module. In some embodiments, the processing activity of each processing module during the database query comprises a number of output rows being generated by the processing module. As an example, if a requesting processing module has an output record count of about 300,000 while each of the other processing modules has an output record count of only about 100,000, then the requesting processing module is placed in a message wait queue until the output record count of each of the other processing modules becomes more equal to the output record count of the requesting processing module.
In some embodiments, the characteristic of the data objects to be dynamically assigned to the processing module comprises size of one or more data objects to be assigned to the processing module. In some embodiments, the characteristic of the data objects to be dynamically assigned to the processing module comprises a collection of small batches of data objects to be dynamically assigned to the processing module.
In some embodiments, the data objects are dynamically assigned from the plurality of data object servers 1401 . . . X to each processing module based upon both processing activity of the processing module and a characteristic of the data objects to be dynamically assigned to the processing module.
In some embodiments, data objects from a processing module that is unable to process its assigned data objects are dynamically reassigned to remaining processing modules so that the remaining processing modules can process the reassigned data objects. In some embodiments, data objects are reassigned to a processing module based upon processing speed of the processing module to which the data objects are reassigned. In some embodiments, data objects are dynamically reassigned to a processing module based upon a determination that the processing module from which data objects are being reassigned is classified as a straggler.
In block 410, a database query is received from a client computer system. Then, in block 420, data objects are iteratively dynamically assigned from a plurality of data object servers to each processing module based upon a select one of (i) processing activity of the processing module, and (ii) a characteristic of the data objects to be dynamically assigned to the processing module. The process then ends.
In some embodiments, a database query involving execution of a join operator is received. As an example, the join operator may involve joining one or more objects stored in data object servers and one or more objects stored in the native database system.
In some embodiments, data objects are iteratively dynamically assigned from a plurality of data object servers to each processing module based upon both processing activity of the processing module and a characteristic of the data objects to be dynamically assigned to the processing module.
In some embodiments, data objects are iteratively dynamically assigned from a plurality of data object servers to each processing module based upon a number of output rows being generated by the processing module.
In some embodiments, data objects are iteratively dynamically assigned from a plurality of data object servers to each processing module based upon size of one or more data objects to be dynamically assigned to the processing module.
In some embodiments, the method further comprises dynamically reassigning data objects from a processing module that is unable to process its assigned data objects to remaining processing modules so that the remaining processing modules can process the reassigned data objects. In some embodiments, data objects are dynamically reassigned to a processing module based upon processing speed of the processing module to which the data objects are reassigned. In some embodiments, data objects are dynamically reassigned to a processing module based upon a determination that the processing module from which data objects are being reassigned is classified as a straggler.
In block 550, a determination is made as to whether data stream associated with the database query has ended. If the determination in block 550 is negative (i.e., the data stream has not ended), the process returns back to block 510 to repeat requesting of data objects from the control task module and providing processing activity information to the control task module until end of data stream associated with the database query is reached. When the determination in block 550 is affirmative (i.e., the data stream has ended), the process ends.
It should be apparent that the above description describes the control task module 150 dynamically monitoring the execution pipeline during a database query and iteratively dynamically adjusting the assignment of data objects to the processing modules 1051 . . . Y based upon a number of different factors. These factors include, but not limited to, data object size, output record counts, processing speed, object store input/output latency, failed processing modules, and stragglers. In the case of stragglers or failed processing modules, there is no need to steal and/or shed shards since data objects were not pre-assigned. Object store input/output latency is related to processing speed, and is usually greater than (and can vary more than) local data storage.
It should be apparent that the control task module 150 iteratively dynamically adapts to all forms of processing skew including, but not limited to, join skew, aggregation/reduce skew, selection/filter skew, and complex analytical functions skew. Moreover, the control task module 150 cooperates with the parsing engine 120 to support filtering shards based upon the particular query format.
It should also be apparent that the above-described control task module 150 supports object storage subsystems in addition to local block-oriented storage. The control task module 150 may be implemented in any type of database system. Moreover, the object can be a specific object or a part of the object that is addressable without reading the whole object. The object can be a full object or a part of the object if the object can be split into multiple addressable parts (e.g., Parquet and row groups). Accordingly, it is conceivable that only part of a data object can be iteratively dynamically assigned by the control task module 150.
It should further be apparent that the above description describes query optimization methods, computer-readable media, and systems that facilitate performance enhancement of a database system. The query optimization methods, media, and systems are applicable to a wide variety of queries, such as a join type of query. Other types of queries which can be optimized are possible.
Each of the above-described flowcharts depicts process serialization to facilitate an understanding of disclosed embodiments and is not necessarily indicative of the serialization of the operations being performed. In various embodiments, the processing steps described in each of the flowcharts above may be performed in varying order, and one or more depicted steps may be performed in parallel with other steps. Additionally, execution of some processing steps of each of the flowcharts above may be excluded without departing from embodiments disclosed herein.
The illustrative block diagrams and flowcharts depict process steps or blocks that may represent modules, segments, or portions of code that include one or more executable instructions for implementing specific logical functions or steps in the process. Although the particular examples illustrate specific process steps or procedures, many alternative implementations are possible and may be made by simple design choice. Some process steps may be executed in different order from the specific description herein based on, for example, considerations of function, purpose, conformance to standard, legacy structure, user interface design, and the like.
Aspects of the disclosed embodiments may be implemented in software, hardware, firmware, or a combination thereof. The various elements of the system, either individually or in combination, may be implemented as a computer program product tangibly embodied in a machine-readable storage device for execution by a processing unit. Various steps of embodiments may be performed by a computer processor executing a program tangibly embodied on a computer-readable medium to perform functions by operating on input and generating output. The computer-readable medium may be, for example, a memory, a transportable medium such as a compact disk, a floppy disk, or a diskette, such that a computer program embodying aspects of the disclosed embodiments can be loaded onto a computer. The computer program is not limited to any particular embodiment, and may, for example, be implemented in an operating system, application program, foreground or background process, or any combination thereof, executing on a single processor or multiple processors. Additionally, various steps of embodiments may provide one or more data structures generated, produced, received, or otherwise implemented on a computer-readable medium, such as a memory.
Although disclosed embodiments have been illustrated in the accompanying drawings and described in the foregoing description, it will be understood that embodiments are not limited to the disclosed examples, but are capable of numerous rearrangements, modifications, and substitutions without departing from the disclosed embodiments as set forth and defined by the following claims. For example, the capabilities of the disclosed embodiments can be performed fully and/or partially by one or more of the blocks, modules, processors or memories. Also, these capabilities may be performed in the current manner or in a distributed manner and on, or via, any device able to provide and/or receive information. Still further, although depicted in a particular manner, a greater or lesser number of modules and connections can be utilized with the present disclosure in order to accomplish embodiments, to provide additional known features to present embodiments, and/or to make disclosed embodiments more efficient. Also, the information sent between various modules can be sent between the modules via at least one of a data network, an Internet Protocol network, a wireless source, and a wired source and via a plurality of protocols.
