Patent Application
20170228422
Distributed processing and storage systems provide redundancy and increased computing power using commodity processing hardware. Such a framework allows for the distributed processing of large data sets across clusters of computers using simple programming models. Each machine or server provides local computation and storage, and is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failure.
Such systems are particularly useful in processing of large datasets given their power to access portions of the dataset in parallel. One such framework is the Hadoop® software library from the Apache Software foundation. The Hadoop® framework includes a distributed storage component known as the Hadoop Distributed File System (HDFS) and a distributed processing component known as MapReduce. Structured query language (SQL) is a programming language designed for managing data held in a database system. In order to utilize, for example, an SQL database on a plurality of processing systems using the Hadoop® framework, a component known as Hive™ was developed. Hive is an open-source Java project which converts a SQL query to a series of MapReduce jobs which run on standard Hadoop® task trackers. This component uses a metastore (itself a database) to store table schemas, partitions, and locations. The Hive™ data warehouse software facilitates querying and managing large datasets that reside in distributed storage. Hive™ provides a mechanism to project structure onto this data and query the data using a SQL-like language. Hive™ is considered a traditional toolset installed on many installations. Queries performed with Hive™ are usually very slow because of the overhead associated with using MapReduce.
Traditional techniques implementing, for example, SQL databases running on Hadoop-based multiple parallel processing (MPP) systems, can be inefficient. For example, implementing SQL on Hadoop® traditionally comprises translation of an SQL query into series of MapReduce jobs (using, for example, the Hadoop® Hive™ infrastructure). This technique has various limitations, such as limited SQL syntax support and slow spawn-up time for short-running queries.
The technology includes a processor-implemented MPP scheduling method. The method includes receiving a query for processing MPP database data, with splits of the MPP database data being stored on a plurality of processing systems and with each processing system including a storage system with at least a split of the MPP database data stored therein. The method also includes determining a total splits number of the MPP database data and if the total splits number is greater than a splits threshold number, streaming partial task maps to the processing system by: creating a partial task map including a task set for the plurality of processing systems, the task set being a predetermined number of tasks which is less than all tasks needed to access the MPP database data to respond to the query, outputting the partial task map to each processing system of the plurality of processing systems upon creation of the partial task map, and repeating said creating and outputting to create and output a sequence of partial task maps. Each partial task map has tasks for different splits of the MPP database data, and the repeating continues until all tasks accessing all MPP database data needed to respond to the query are included in at least one partial task map.
In another embodiment, the technology comprises a non-transitory computer readable medium storing computer instructions to query MPP database data stored in a data structure spanning a plurality of processing devices. The instructions when executed by one or more processors cause the one or more processors to perform the steps of: determining a total splits number of MPP database data in the data structure, each split containing at least a portion of the MPP database data; creating a successive plurality of task maps if the total splits number is greater than a splits threshold number; and outputting each of the successive plurality of task maps to each of the plurality of processing devices upon creation of each of the successive plurality of task maps. Each task map contains a predetermined number of instructions to the plurality of processing devices to access at least one split of MPP database data to respond to a query. The instructions when executed by one or more processors cause the one or more processors to create and output the successive plurality of task maps until instructions to access all MPP database data to respond to the query are included in at least one task map. Each of the successive plurality of task maps has instructions to read different splits of the MPP database data.
In another embodiment, the technology comprises a multiple parallel processing (MPP) system, including: a plurality of MPP processing devices, each MPP processing device including a non-transitory memory storage including instructions and one or more processors in communication with the memory. The one or more processors on one of the MPP processing devices execute the instructions to receive a query for processing MPP database data, portions of the MPP database data being stored on the plurality of processing devices. The one or more processors on one of the MPP processing devices execute the instructions to retrieve a total splits number, each split including a portion of the MPP database data. If the total splits number is greater than a splits threshold number, the one or more processors on the one of the MPP processing devices execute the instructions to create a partial task map including a predetermined number of instructions for the plurality of processing devices, the instructions to access at least one split of MPP database data to respond to the query. The one or more processors on one of the MPP processing devices execute the instructions to output the partial task map to each processing device of the plurality of processing devices upon creation of the partial task map. The one or more processors on one of the MPP processing devices execute the instructions to repeat said creating and outputting to create and output a sequence of partial task maps, with each partial task map having tasks for different splits of the MPP database data, and continue to repeat creating and outputting until instructions for all splits of the MPP database data are included in at least one partial task map.
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
Technology is disclosed to provide improved database query performance in a multiple parallel processing (MPP) system, and in particular a MPP database (MPPDB) system running in a Hadoop® HDFS environment. The MPP system includes processing hardware and an underlying software framework, wherein the MPP system can be implemented on multiple types of processing hardware. An MPPDB system utilizes the MPP system to implement database storage and database processing on the MPP system. The technology in some examples is implemented in an MPP system supporting a Hadoop® MPPDB system on HDFS. The technology in some examples is described with respect to use of an SQL database and an SQL query. However, the technology can be implemented on other systems and for other database protocols or database languages.
Various existing implementations attempt to improve query performance for database queries against database data stored in a Hadoop® HDFS environment. Generally, database data is stored in the form of tables, with table portions or “splits” stored in different storage locations throughout the environment. As used herein, the term “splits” refers to portions of database data that have been divided into smaller files for storage in an MPP system. Traditional implementations of SQL on Hadoop® use some form job scheduling based on a translation of the SQL query into series of MapReduce jobs. MapReduce is known to be slow and inefficient in these implementations. Existing implementations of SQL on Hadoop® all share similar scheduler logic for SQL on Hadoop® task assignments. These scheduler designs do not scale well. When there are a large number of table splits (HDFS files) for a large database data table, the scheduler itself becomes a bottleneck. The scheduling operation takes time, and all MPP data nodes in the environment which store table portions must wait for the scheduler to finish processing all files in the database data before they begin to work on reading the files. In addition, the size of a MapReduce task map is linearly proportional to the size of a database data file, and thus large database data tables generate large task maps. Distributing a number of such large task maps at the same time may cause network congestion. In the context of this disclosure, database data refers to MPP database data, meaning data stored in an MPP environment.
The present technology presents a solution for an MPP database to access database data in a HDFS environment without using the MapReduce framework. The technology uses a flexible, hybrid scheduler to instruct MPP data nodes to process database data table portions efficiently based on data locality and data node workloads.
The scheduler selects between operating in a task “streaming” mode or a “batch” mode based on the size and number of splits of the database data being accessed. In streaming mode, portions of the tasks needed to execute a query of a large amount of database data are captured in partial task maps, and streamed to MPP processors in small batches. This enables the query processing to start even though not all tasks have been created or delivered to processing nodes, and continues streaming these small, partial task maps while still creating other, subsequent partial task maps. This streaming mode generates these partial task maps during query execution, rather than during compile time. In streaming mode, scheduling can be done without affecting task operation and without affecting communications bandwidth. By breaking the scheduling down into smaller batches and sending out small scheduling assignments to various processing nodes, the processing nodes begin processing even as the scheduler continues to generate other small scheduling portions and stream them out to the processing nodes. Thus, the processing nodes can be kept continuously busy without needing to generate a large task schedule and without needing to send out a large task schedule to various processing nodes. This allows processing to begin earlier than in the case when all processing nodes receive the one large task file. In batch mode, used for smaller database data queries, a complete batch file is used and generated at compile time. The technology selects between using each mode based on the size of database data being accessed.
The flexible, hybrid scheduler retrieves database data split information directly from the HDFS NameNode, allowing the scheduler to determine the block location of each table portion directly and to assign the best possible MPP data node to read a database data portion in response to a database query. The flexible, hybrid scheduler evenly distributes work on the splits to available MPP data nodes.
The technology improves the performance of a Hadoop® MPPDB system on HDFS when receiving a database query, where the database data is divided into multiple splits for storage on a plurality of processing systems. The technology determines a total splits number of the database and creates a task instruction file or task map for either the batch mode or for streaming mode, based on the number of splits of the database. If the total splits number is greater than a splits threshold number, a sequence of partial task maps (with each partial task map in the sequence including a tasks set for a subset of splits) is created. The sequence of partial task maps are output to processing systems acting as data nodes upon creation of each partial task map. In this streaming mode, task map creation begins upon receipt of the query and after compiling the database query. Conversely, if the database is small and/or the total splits number is smaller than the splits threshold number, then a single complete task map for all splits in the database can be created for all splits and distributed to the target processing systems. In this latter batch mode, task map creation occurs at compile time.
It should be understood that database data may have many representations in an MPP system. In a Hadoop® system, for example, a standard and commonly used Hadoop® table representation is a Hive™ table. A Hive™ table generally comprises HDFS directories where the directory name is the table name, and HDFS files represent the table contents in a specific file format (such as ORC, CSV, etc.). The data in the table may be sliced into multiple splits before writing to HDFS, and each split is a HDFS file which represents part of the table. The number of table splits is controlled by the tool or application used to load the data. The present technology addresses how a MPPDB system reads those splits out from a Hadoop® distributed file system. Other forms of database data and other types of databases which may be supported on an MPP system may be utilized in accordance with the present technology.
Processing system node 205 houses the name node for the MPP system. In one embodiment where the MPP system is implemented as a Hadoop® cluster, the Hadoop® NameNode 210 keeps a directory tree of all files in the file system, and tracks where, across the cluster, all given file data is kept. A requesting application 200 communicates with the NameNode 210 whenever it needs to locate a file, or when the application needs to manipulate (add/copy/move/delete) a file. The NameNode 210 returns a list of relevant data nodes where the data is stored. In the presently described embodiment, the requesting application 200 is a database application seeking to access data from a database data 100. Hence, a requesting application may run an SQL query against data in the processing system nodes 220, 230, 240 by seeking access to such data via that NameNode 210. Other components of an MPP database system can be included, but are not illustrated in
As illustrated in
In
In one embodiment, the MPP data nodes 342, 343, 340 and MPPDB coordinator 310 are components of any MPP database system. The HDFS NameNode 315, HDFS data nodes 352, 353, 354 and reader components 345, 346 and 347 are all elements of the MPP system. Each HDFS data node 352, 353, and 354 all physically store the database data splits 355, 356, and 357 identified with respect to
Scheduler 350 performs the functions outlined in the flowcharts of
In general, as requests for data from the requesting application 200 are received by the MPPDB coordinator 310, the MPPDB coordinator 310 will request split information from the HDFS NameNode 315. For example, when a query comes to the MPPDB coordinator identifying a particular database table to be read, the MPPDB coordinator 310 will determine the directory path through the metadata stored by the MPPDB coordinator 310, and pass the information to the scheduler 350. The scheduler will then query HDFS NameNode 315 to determine the location where the table data physically resides (for example, the HDFS directory), as well as all the split information inside the directory. The split information includes information about how many replicas the split has, along with the HDFS data node location where each replica resides.
The HDFS NameNode will return file block locations to the MPPDB coordinator 310 and scheduler 350. Scheduler 350 will then create a complete task map or a partial task map 320 (depending on the size of the table), identical copies of which are forwarded to each of the processing system nodes 330, 332, and 334. In a batch mode, when the number of splits is less than a splits threshold number (for example, one-thousand splits,) a single task map along with a compiled SQL query is forwarded to each of the processing system nodes 330, 332, 334, 336. In a streaming mode, when the number of splits is greater than the splits threshold number (once again, for example, one-thousand), the scheduler 350 will create partial task maps and will send identical copies of each partial task map 322 each of the processing system nodes 330, 332, 334. Each partial task map will include for example, a sub-set of all read tasks (a partial read tasks amount) needed to respond to the query. The partial read tasks amount may be set by as a predetermined number of splits selected based on how large each partial task map should be given design considerations of the network connecting the MPP systems. The total number of splits needed to respond to the query is referred to herein as the total splits number. The partial read tasks amount may be less than the total splits number of the database data needed to respond to the query, and may be the same as or different than the splits threshold number utilized to determine when to enter streaming mode or batch mode. Each partial task map received by the processing system nodes 330, 332, 334, is read and executed by a HDFS reader to acquire data from a corresponding HDFS data node to access table data in the table splits, allowing the MPP data nodes to return the results of the SQL query to the MPPDB coordinator 310. The MPPDB coordinator 310 is operable to compile, optimize, and stream the queries to the processing system nodes 330, 332, 334.
At 410, a determination is made as to whether the number of splits (the total splits number) in the database data is greater than a splits threshold number. If so, then the streaming mode (steps 450, 454 and 456) is performed. If total splits number is less than a splits threshold number, then batch mode (steps 425 and 430) is performed.
In batch mode, at 425, the scheduler iterates through the number of splits for the database data and creates a complete task map containing read tasks for all splits needed to answer the query. The task maps contain a list of tasks identifying which MPP data nodes are to read which table splits. At 430, the task map is distributed to all nodes in the MPP system. As explained below, task map creation occurs at compile time and a query plan and task map may thereafter be distributed to all processing devices in the cluster.
In streaming mode, tasks instructing a read of only a partial number (a fraction of the total splits number) are sent in each partial task map. In streaming mode, at 450, the method iterates through the splits up to a set number of splits and when the set number of splits is reached, a partial task map is created and output to the nodes in the MPP system at 454. If there are additional splits in the database data at 456, the method continues iterating and generating partial task maps at 450 until all the splits in the database data are addressed and the method completes at 460.
The scheduling process of
In order to determine the number of splits in database data for a particular query, at 505 the scheduler communicates with the MPPDB coordinator 310 to retrieve the number of splits and the directory path from MPPDB coordinator 310. At 510, the location (for example, a HDFS directory) where the database data physically resides, as well as all the split information inside the directory, is retrieved from the HDFS NameNode 315. As noted above, database data 100 may have many representations in an MPP system. In a Hadoop® embodiment, when a query comes to the MPPDB to read a particular Hadoop® database data table, the coordinator will determine HDFS directory path through the table metadata in its metastore, and pass the information to the scheduler. The scheduler will then query HDFS NameNode at 510 to find out the HDFS directory where the database data table physically resides, as well as all the split information inside the directory. The split information includes the how many replicas the split has, along with the HDFS data node location where each replica resides.
At 515 a determination is made as to whether the number of splits is greater than a splits threshold number. The splits threshold number may be less than the total splits number. In one embodiment, the splits threshold number is set at 1000 splits, for example. The size of the split threshold at 515 is generally set to remain small to avoid having the scheduler turn into a bottleneck if the table to be accessed is very large. (This may be the case in a Hadoop® embodiment where tables on the order of terabytes or petabytes are encountered.) In batch mode, the scheduling computation happens at compile time, and thus all MPPDB data nodes wait till they receive the final complete task map before they can actually read the data.
If the number of splits to be accessed to respond to the query is less than the splits threshold number, then step 425 is performed. Step 425 begins compiling the query at 520. In general, the MPP coordinator will compile and optimize the query for execution on each of the MPP nodes in the system. At 522, a complete task map is created. A method for creating the task map is illustrated in
Once the task map is complete, the task map and compiled query are distributed to the MPP nodes at 524.
Returning to step 515, if the number of splits to be accessed to respond to the query is greater than the splits threshold number, then the streaming mode begins at 450. At 540, the scheduling process begins before or in parallel with compiling of the query at 530. Since, in streaming mode, the scheduler does not wait for the compiling of the query to complete, the scheduler is no longer a bottleneck on large query jobs. Once completed, the compiled query is output at 532 and the MPPDB data nodes will await the task maps to begin executing the query.
At 540, the scheduler iterates through the splits until a predetermined number of splits is reached at 542. Once reached, a partial task map is created at 544. The method of
Steps 552, 554, 555, and 556 illustrate the processing of a partial task map. At 552, when the streaming mode is enabled, the data node will receive the compiled query and await the read task instructions in the partial task maps. The data node will receive a next partial task map at 554 and parse the partial map for tasks assigned to the data node at 555. At 556, the data node will begin reading the split data, executing the query at 558 on each set of splits for that data node in each partial task map. At 558, if there are additional splits in the database data, the process continues at 540 until the query is executed and finishes retrieving all data. The results of the query are returned at 560. The technology thus improves the speed at which the output results of the initial query are returned in an MPP system.
If no data nodes are available at 610, then a determination is made at 642 as to whether a replica data node is available at 642. If no data node or replica data node is available, an error is output.
If data nodes are available at 610, then the method will determine at 612 whether the MPP data nodes are local to the split. In other words, a determination is made as to whether the data resides on a same physical processing system as the split. Step 612 scans through all the MPPDB data nodes and finds all available data nodes which are local to the split (having the same hostname/IP as the split), storing them as the “available candidates” for the next step.
From the “available candidates” obtained from step 612, at 614 the method iterates through local MPP data nodes and at 616 finds the first data node with no task assigned. If such a local node is found at 616, the task is assigned to it at 618. If all candidates have tasks assigned, then step 616 finds the data node with least tasks assigned and assigns the split to it. This process can be further optimized to deal with large amount of available worker nodes by using a data structure similar to a “LRU” cache, which puts the worker node with a lighter load at the front of the queue and repositions the heavy processing load to the rear of the cache each time a split is assigned to a worker processing node.
If none of the MPPDB data nodes are local to the split at 612, then the method will choose one remote MPPDB data node with least tasks assigned and append the split to the task map for that node. The method will iterate through all MPPDB data nodes at 620 to determine which data node has the least work at 622. The method continues at 630 until all tasks are assigned. Assigning non-local tasks is not an optimal situation because a remote read is almost always slower than a local read, but the method has to deal with this situation when MPPDB data nodes are not deployed on every HDFS data nodes.
The output of the method of
The method of
One alternative or additional solution to the issue of schedule bottleneck is for the scheduler to compress the data node-task map before sending it out. This may be used with either the batch mode or the streaming mode. The streaming mode implements so called “lazy” scheduling in which the scheduling computation is delayed from compile time to execution time. The MPPDB coordinator delays calling the scheduler to do the task map creation, but compiles the query and constructs a query plan. Once the query plan is delivered to the MPPDB data nodes, the MPPDB data nodes signal the scheduler that they are ready to read the data. The scheduler will then start to compute the task map, and enter the task streaming stage.
The technology provides a system in which the MPPDB data nodes need not wait until the scheduler finishes all the scheduling work before reading the data. For example, once the scheduler finishes the first 1000 splits, it sends the task map to all MPPDB data nodes to process, while continuing to schedule the rest of splits in parallel, streaming the batches out every 1000 splits, for example. Since the number of MPPDB data nodes is fixed, each batch can be configured as a columnar data structure, which is essentially an N by 1000 2-dimensional array with N being the number of MPPDB data nodes, where each column of the matrix contains the tasks assigned to the MPPDB data node. In MPPDB architecture, the streaming usually happens between data nodes, but in this case, the streaming operation between CN and data nodes can be utilized for the data node task map.
Illustrated in non-volatile storage 730 are functional components which may be implemented by instructions operable to cause processor 710 to implement one or more of the processes described below. While illustrated as part of non-volatile storage 730, such instructions may be operate to cause the processor 710 to perform various processes described herein using any one or more of the hardware components illustrated in
Non-volatile storage 730 may comprise any combination of one or more non-volatile computer readable media. The computer-readable non-transitory media includes all types of computer readable media, including magnetic storage media, optical storage media, and solid state storage media and specifically excludes signals. It should be understood that the software can be installed in and sold with the device. Alternatively the software can be obtained and loaded into the device, including obtaining the software via a disc medium or from any manner of network or distribution system, including, for example, from a server owned by the software creator or from a server not owned but used by the software creator. The software can be stored on a server for distribution over the Internet, for example.
The computer system 702 can include a set of instructions that can be executed to cause computer system 702 to perform any one or more of the methods or computer based functions disclosed herein. Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language conventional procedural programming languages. The program code may execute entirely on the computer system 702, partly on the computer system 702, as a stand-alone software package, partly on the computer system 702 and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider) or in a cloud computing environment or offered as a service.
The processor 710 is configured to execute software instructions in order to perform functions as described in the various embodiments herein. The processor 710 may be a general purpose processor or may be part of an application specific integrated circuit (ASIC). The processor 710 may also be a microprocessor, a microcomputer, a processor chip, a controller, a microcontroller, a digital signal processor (DSP), a state machine, or a programmable logic device. The processor 710 may also be a logical circuit, including a programmable gate array (PGA) such as a field programmable gate array (FPGA), or another type of circuit that includes discrete gate and/or transistor logic. The processor 710 may be a central processing unit (CPU), a graphics processing unit (GPU), or both. Additionally, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors may be included in, or coupled to, a single device or multiple devices.
Illustrated in the non-volatile storage 730 are components for implementing a name node, including a MPP Coordinator code component 732, scheduler code component 734, and MPP Framework component 746. Also illustrated are components for implementing a data node, including FS data node code component 754, MPP data node code component 752, and database data 748. In general, the name node will not be co-located with a data node and, components 732, 734, and 746 will not reside on the same physical node as components 754, 752, and database data 748. They are thus illustrated in
As shown, the computing system 702 may further include a display unit (output device) 750, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, or a cathode ray tube (CRT). Additionally, the imaging processor may include an input device 760, such as a keyboard/virtual keyboard or touch-sensitive input screen or speech input with speech recognition, and which may include a cursor control device, such as a mouse or touch-sensitive input screen or pad.
Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatuses (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable instruction execution apparatus, create a mechanism for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer readable medium that when executed can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions when stored in the computer readable medium produce an article of manufacture including instructions which when executed, cause a computer to implement the function/act specified in the flowchart and/or block diagram block or blocks. The computer program instructions may also be loaded onto a computer, other programmable instruction execution apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatuses or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
The technology advantageously provides a processor-implemented method of responding to a database query, wherein portions of the database are stored on a plurality of processing systems. Each of the processing systems include a storage system with at least a portion of the database stored therein. The method determines a total splits number of the database, each split containing at least a portion of the data stored in the database. If the number of total splits is greater than a threshold number, the method creates creating a partial task map including a set of tasks for the processing systems for a set number of splits. The method outputs the partial task map to the plurality of processing systems upon creation of the partial task map, and repeats creating additional partial task maps to create a sequence of partial task maps. Each partial map has tasks for different splits of the data up to the set number of splits until all splits of the total splits number are included in at least one partial task map. If the number of splits is less than a threshold number, the method creates a complete task map including the total splits number and outputs the complete task map with the total splits number to the plurality of processing systems. A query result is provided by executing the query against each of the splits based on the set of tasks for the processing system.
The technology thus provides a system and method of improving the performance of an MPPDB system when receiving a query for a database where the database itself is split into multiple parts on multiple processing systems. The technology determines a total splits number of the database and creates a read task instruction file or task map using either a complete task map, batch mode or in multiple, partial task map, streaming mode based on the number of splits of the database. If the number of total splits is greater than a threshold number, a partial task map including a set of tasks for a set number of splits is created. The partial task maps are created for the set number of splits sequentially and are output to the processing systems upon creation of each partial task map. In this streaming mode, task map creation begins upon receipt of the query and after compiling the database query. If the database is small and the number of splits is smaller than the threshold number, then a single complete task map for all splits in the database is created and distributed. In this latter batch mode, task map creation occurs at compile time.
In a Hadoop® environment, the technology bypasses the Hadoop® MapReduce framework, directly accessing file metastore information from a Hadoop® name node, and assigns table splits to their local MPPDB data nodes. The technology provides for evenly distributing tasks among MPPDB data nodes per query, thereby improving performance and balancing the processing load. Lazy scheduling and task streaming of partial task batches enables efficiency between the scheduling computation and data reading processes for large scale data sets.
In accordance with the above advantages, the technology includes the MPPDB coordinator 310 for receiving a query for database data, splits of the database are stored on a plurality of processing system nodes 330, 332, 334, with each processing system comprising a storage system with at least a portion of the database stored therein. Also included is a scheduler 350 for determining a total splits number of the database, with each split containing at least a portion of the database data. If the total splits number is greater than a threshold number, the scheduler 350 operates to create a partial task map including tasks for a predetermined number of splits for the processing systems, outputs the partial task map to the plurality of processing systems upon creation of the partial task map, and repeatedly creates a sequence of partial task maps having tasks for different splits of the database data up to the predetermined number of splits until all splits of the total splits number are included in at least one partial task map. If the number of splits is less than a splits threshold number, the scheduler 350 operates to create a complete task map including the total splits number and outputting the complete task map with the total splits number to the plurality of processing systems.
The technology provides a number of advantages over existing solutions for SQL on Hadoop®. The technology provides parallelized scheduling of MPP DN task jobs, reducing the potential bottlenecks which result from task scheduling computation. The scheduler provides load balancing by processing data locally stored on a data node, if possible, and balancing task performance among the data nodes by intelligently choosing the best data node to handle the task
The technology provides MPPDB query speed improvements by using hybrid operational modes, choosing between a batch task assignment mode and streaming mode. In streaming mode, “lazy” scheduling is performed by performing scheduling before compiling the database query. The scheduler accesses database data split information directly from the HDFS name node to provide improved data consistency.
This differs significantly from the prior art which uses task scheduling techniques that do not scale well to large data sets. The scheduling combined with task streaming mechanism in this invention addresses the pain point by breaking the pipeline and parallelizing the scheduling work.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
