SQL structure analyzer

Abstract

A method to addresses structural performance problems of a database query language statement is provided. The method includes receiving a database query language statement at an optimizer, evaluating choices in a search space to generate an execution plan for the statement, and producing annotations to record one or more reasons for selecting each choice while generating the execution plan.

Description

FIELD OF THE INVENTION

This invention is related to the field of database management.

BACKGROUND

Currently, database administrators (DBAS) and application developers spend a large amount of time trying to tune poorly performing and resource intensive SQL statements (which is commonly referred to as bad sql). However, it is often a very challenging task. First, it requires a high level of expertise in several complex areas, such as query optimization and SQL design. Second, it is a time consuming process because each statement is unique and needs to be tuned individually. Third, it requires an intimate knowledge of the database (i.e., view definitions, indexes, table sizes, etc.) as well as the application (e.g. process flow, system load). Finally, the SQL tuning activity is a continuous task because the SQL workload and the database are always changing. As a result, tuning is often done on a trial and error basis, resulting in loss of productivity.

Often a SQL statement can be a high load SQL statement simply because it is badly written. This usually happens when there are different, but not semantically equivalent, ways to write a statement to produce same result. Knowing which of these alternate forms is most efficient in producing the query result is a difficult and daunting task for application developers since it requires both a deep knowledge about the properties of data they are querying as well as a very good understanding of the semantics and performance of SQL constructs.

To help DBAs and application developers overcome these challenges, several software companies have developed diagnostics tools that help identify SQL performance issues and suggest actions to fix them. However, these tools are not integrated with the database compiler, which is the system component that is most responsible for SQL performance. Indeed, these tools interpret the optimization information outside of the database to perform the tuning, so their tuning results are less robust and limited in scope. Moreover, they cannot directly tackle the internal challenges faced in producing an optimal execution plan.

The SQL Structure Analyzer component of the Automatic Tuning Optimizer performs what-if analysis to recognize missed query rewrite opportunities and makes SQL restructuring recommendations for the user to undertake.

SUMMARY

A method to address structural performance problems of a database query language statement is described. The method includes receiving a database query language statement at an optimizer, evaluating choices in a search space to generate an execution plan for the statement, and producing annotations to record one or more reasons for selecting or rejecting each choice while generating the execution plan. The method can further include the examination of the annotations associated with the costly operators in the chosen plan and producing appropriate SQL restructuring recommendations to improve the execution performance of the statement.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of a device that includes the SQL Structure Analyzer.

FIG. 2 shows an example of a method for analyzing the structure of a SQL statement.

FIG. 3 is a block diagram of a computer system suitable for implementing an embodiment of a SQL structure analyzer.

DETAILED DESCRIPTION

Overview

The embodiments of the invention are described using the term “SQL”, however, the invention is not limited to just this exact database query language, and indeed may be used in conjunction with other database query languages and constructs.

A SQL Structure Analyzer is a component of an Automatic Tuning Optimizer that addresses structural performance problems of a statement. This component can be used by programmers, during an application development, to detect poorly written SQL statements, and apply alternative ways of rewriting them to improve the performance. The analyzer can determine whether a SQL statement is a high load statement simply because it is badly written. For example, different, but not necessarily semantically equivalent, ways to write a statement to produce same result can be examined to determine which of these alternate forms is most efficient. Although this is a difficult and daunting task for application developers, since it requires both a deep knowledge about the properties of data they are querying as well as a very good understanding of the semantics and performance of SQL constructs, the optimizer can perform the structural analysis process in an efficient manner.

Performing the structural analysis of a statement within and by the automatic tuning optimizer itself, while generating an execution plan for the statement, allows the procedure to identify and gather information about the statement's structure that will help produce an efficient plan. The method can use this information to compare different, but not necessarily equivalent, ways of writing a statement to produce the same result.

The query optimizer can perform extensive query transformations while preserving the semantics of the original query. Some of the transformations are based on heuristics (i.e. internal rules), but many others are based on cost-based selection. Examples of query transformations include subquery unnesting, materialized view (MV) rewrite, simple and complex view merging, rewrite of grouping sets into UNIONs, and other types of transformations.

The query optimizer may apply a transformation when the query can be rewritten into a semantically equivalent form. Semantic equivalence can be established when certain conditions are met; for example, a particular column in a table has the non-null property. However, these conditions may not exist in the database but can be enforced by the application. The SQL Structure Analyzer performs what-if analysis to recognize missed query rewrite opportunities and makes recommendations for the user to undertake.

When a rewrite is not possible, the optimizer generates diagnostic information in the form of internal annotations to remember the reasons why the particular rewrite was not possible. The annotations can include necessary conditions that were not met, as well as various choices that were available during the plan generation process. After a best plan is generated, the optimizer examines the annotations, and produces appropriate recommendations for improving the execution plan. For example, the recommendations can be suggestions on how to rewrite the statement, as well as suggestions for changing the schema, in order to improve the performance of the statement. In addition, the optimizer can use the an notations to produce rationale and informative messages about potential improvements that can be made to the statement, in order to educate application developers who code the SQL statement.

One possible output from the SQL Structure Analyzer can be a rewritten SQL text that the user can accept as an alternative form of the original statement. If the user accepts the alternate form then he has to pass the rewritten SQL text as input to the query optimizer in place of original SQL text.

The SQL structural analysis is a cost-based process, wherein it considers the annotations associated with costly operators in the annotated execution plan. As a result, the process generates recommendations for costly nodes and operators that, when reconsidered by changing the structure of the query statement, will significantly improve the performance of the execution plan. For example, a costly node can be defined as a node having an individual cost that is greater than a threshold, such as 10% of the total plan cost. The recommendation for the costly operator is then mapped to the corresponding node in the plan tree, as well as to the operator in the SQL statement.

An example of a device that includes the SQL Structure Analyzer is shown in FIG. 1. The Auto-tuning optimizer 110 receives a SQL statement 120. Then, the optimizer generates an execution plan for the statement. During the plan generation process, the SQL Structure Analyzer 130 generates diagnostic information for the statement. For example, when the optimizer 110 evaluates choices for the plan, the Analyzer 130 can produce annotations to the plan to explain a choice made by the optimizer.

The annotations can include alternatives that were considered and rejected. An alternative structure can be rejected because it may cause a change in the query results. An alternative may also be rejected for other reasons as well. For example, when the optimizer explores the possibility of merging a view, it runs its tests to determine if it is logically possible to merge a view. If this is not possible, the analyzer records the reason for not being able to merge the view in the execution plan. If the optimizer can merge the view, but it decides not to merge it, then the analyzer can record the reason for not choosing to merge the view.

The annotated plan is then examined by the optimizer to generate recommendations for rewriting the statement, as well as recommendations on schema changes to improve the performance of the SQL statement. For example, after gathering information about the structure of the statement, the optimizer can identify an expensive operator in the statement. Using the annotations for the expensive node, the optimizer can access a knowledge base or a rule base to retrieve a rule for replacing the expensive operator in the statement with a less expensive operator.

If the expensive operator is, for example, a UNION operator, the optimizer can find a rule in the knowledge base for this operator, such as “replace UNION with UNION ALL.” The optimizer can determine if applying this rule to the query will reduce the cost of the operator. If so, then the optimizer can recommend that the user rewrite this operator of the statement by replacing the UNION operator with the UNION ALL operator. However, with this particular rewrite, the query results may be different, because the UNION ALL operator will not remove duplicates from the results, but the UNION operator will. Thus, the recommendation will include an improved performance benefit resulting from this rewrite, as well as the potential for different query result. If the user decides that the improved performance is worth the trade-off in the results, the user can apply this recommendation to the SQL statement.

In addition to applying rules stored in a knowledge base, the optimizer can accept rules from the user and apply them while considering the annotations. Also, the user can disable certain rules in the knowledge base to prevent the query optimizer from giving recommendations that cannot be implemented by the user.

There are various causes of poor performance, which are related to the structure of a SQL statement, that can be identified and overcome by using the structural analysis process. These causes can be syntax-based, semantics-based, or design issues.

An example of a semantic-based factor that can be analyzed to improve performance is a UNION operator in a SQL statement. The replacement of the UNION operator with the semantically different UNION ALL operator may provide an equivalent result if duplicate rows are not in the result. For example, if the UNION-ALL operator is used for tables that have different data, such as ‘last year's sales’ and ‘this year's sales,’ the UNION-ALL operator in this example can provide the same result as the UNION, because the result of the operation has no duplicate rows, making the duplicate elimination performed by the UNION operator redundant. Thus, an analysis of the structure provides a basis to recommend replacing UNION with UNION-ALL, thus eliminating an expensive duplicate elimination procedure from the execution plan.

Another example is the use of the semantic-based NOT IN subquery. When this semantic-based construct is replaced by a corresponding but not semantically equivalent NOT EXISTS subquery, the result can be a significant performance boost. This replacement can be recommended by the analysis process if NULL values are not present in the related join columns, thus ensuring that same result is produced by either of these operators. Another example is

Syntax-based constructs are generally used to specify predicates in a SQL statement. The corresponding performance attributes of syntax-based constructs are therefore related to the specification of predicates in the SQL statement. For example, if a predicate such as col=:bnd is used with col and :bnd having different types, then such a predicate is unable to be used as an index driver. Similarly, a predicate involving a function or expression (e.g. func(col)=:bnd, col1+col2=:bnd) on an indexed column prevents the query optimizer from using an index as an access path. As a result, this predicate, which involves this function may not be used as an index driver unless there is a functional index on the function itself. Therefore, rewriting the statement by simplifying the complex predicate can enable index access paths leading to a better execution plan.

Design issues related to performance include an accidental use of a Cartesian product, for example, which occurs when one of the tables is not joined to any of the other tables in a SQL statement. This problem is frequent when the query involves a large number of tables. Therefore, rationale and informative messages can be produced to educate programmers who code SQL statements about potential design improvements to the statements.

For example, during the development stage, developers are generally focused on writing SQL statements that produce a desired result, rather than designing the statement to optimize the performance of the statement. The informative messages can help the developers improve performance by identifying design mistakes and offering alternatives. For example, the structural analysis method can identify a mistake that causes a SQL statement to perform poorly, such as a type mismatch between the column and its predicate value, which essentially disables the use of an index even if one is available, then inform the user of the mistake in the design of the statement.

An example of a SQL structure analysis method that can be performed by the automatic tuning optimizer to detect poor SQL constructs falling into one or more categories listed above is shown in FIG. 2. The SQL structure analysis method is fully cost-based.

The automatic tuning optimizer generates internal annotations and diagnostic information, 210 and associates them to the execution plan operators. The annotations are produced while the automatic tuning optimizer is evaluating the various choices during the process of building the execution plan. Each annotation can be quite extensive and can include the reasons for making a decision as well as the alternatives that were considered and the corresponding reasons for rejecting them. For example, when the automatic tuning optimizer explores the possibility of merging a view, it will check necessary conditions to see if it is logically possible to merge the view. If not possible, it can record the reason for not merging the view. If it can merge but it decides not to merge the view, it can record the reason for not doing so.

After the optimal execution plan has been built, the automatic tuning optimizer examines the costly operators in the annotated execution plan. For example, a costly operator can be defined as one whose individual cost is more than 10% of the total plan cost. The automatic tuning optimizer examines the annotations associated with each of the costly operators and produces appropriate recommendations, 220. The Automatic Tuning Optimizer also provides rationale behind each of its recommendations. For example, a rationale can provide an explanation for using a recommended SQL construct in place of the original one to improve the cost and hence the performance of the corresponding execution plan.

The SQL structure recommendations allow a developer or the optimizer to rewrite a problematic SQL statement. Therefore, the SQL structure analysis method can be used to improve SQL statements while they are being developed, before they are deployed into a production system or a packaged application. Another important benefit of the SQL structure recommendations is that they can help educate the developers in writing well-formed SQL statements.

FIG. 3 is a block diagram of a computer system 300 suitable for implementing an embodiment of the SQL structure analyzer and structural analysis method. Computer system 300 includes a bus 302 or other communication mechanism for communicating information, which interconnects subsystems and devices, such as processor 304, system memory 306 (e.g., RAM), static storage device 308 (e.g., ROM), disk drive 310 (e.g., magnetic or optical), communication interface 312 (e.g., modem or ethernet card), display 314 (e.g., CRT or LCD), input device 316 (e.g., keyboard), and cursor control 318 (e.g., mouse or trackball).

According to one embodiment of the invention, computer system 300 performs specific operations by processor 304 executing one or more sequences of one or more instructions contained in system memory 306. Such instructions may be read into system memory 306 from another computer readable medium, such as static storage device 308 or disk drive 310. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention.

The term “computer readable medium” as used herein refers to any medium that participates in providing instructions to processor 304 for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as disk drive 310. Volatile media includes dynamic memory, such as system memory 306. Transmission media includes coaxial cables, copper wire, and fiber optics, including wires that comprise bus 302. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

Common forms of computer readable media includes, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, carrier wave, or any other medium from which a computer can read.

In an embodiment of the invention, execution of the sequences of instructions to practice the invention is performed by a single computer system 300. According to other embodiments of the invention, two or more computer systems 300 coupled by communication link 320 (e.g., LAN, PTSN, or wireless network) may perform the sequence of instructions to practice the invention in coordination with one another. Computer system 300 may transmit and receive messages, data, and instructions, including program, i.e., application code, through communication link 320 and communication interface 312. Received program code may be executed by processor 304 as it is received, and/or stored in disk drive 310, or other non-volatile storage for later execution.

In the foregoing specification, the invention has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than restrictive sense.

Claims

1. A method comprising: receiving a database query language statement at an optimizer;evaluating one or more choices in a search space to identify or generate an optimal execution plan for the database query language statement;generating an annotated execution plan from the optimal execution plan by producing and storing one or more annotations in the annotated execution plan to explain one or more reasons for the optimizer selecting a choice over an alternative choice during the act of evaluating the one or more choices, whereinthe annotated execution plan is examined for identifying or generating a recommendation for one or more schema changes and the possible statement modification to rewrite the database query language statement; anddisplaying the recommendation on a display apparatus or storing the result in a tangible computer usable medium or a storage device.

2. The method of claim 1, wherein the one or more annotations for a selected choice comprises: one or more semantically different alternatives to the selected choice;reasons for not performing one or more rewrites of the database query language statement; andinformation to be displayed to a developer of the database query language statement.

3. The method of claim 2, further comprising: associating the one or more annotations for each selected choice to a corresponding operator of the database query language statement.

4. The method of claim 3, further comprising: identifying operators of the database query language statement having a cost above a threshold cost;for each costly operator, examining the one or more annotations associated with the identified operator; andgenerating a structural recommendation about the costly operator to improve the database query language statement.

5. The method of claim 4, wherein generating the structural recommendation comprising: retrieving an alternative operator from the one or more annotations;comparing a cost of the alternative operator with a cost of the costly operator; andmapping the recommendation to use the alternative operator to the costly operator in the database query language statement.

6. The method of claim 5, further comprising: rewriting the database query language statement with the alternative operator; andgenerating an execution plan for the database query language statement with the alternative operator.

7. The method of claim 1, wherein the database query language statement is a SQL statement.

8. An apparatus comprising: means for receiving a database query language statement at an optimizer;means for evaluating one or more choices in a search space to identify or generate an optimal execution plan for the database query language statement;means for generating an annotated execution plan from the optimal execution plan by producing and storing one or more annotations in the annotated execution plan to explain one or more reasons for the optimizer selecting a choice over an alternative choice during the act of evaluating the one or more choices, wherein the annotated execution plan is examined for identifying or generating a recommendation for one or more schema changes and the possible statement modification to rewrite the database query language statement; anda display apparatus configured for displaying the recommendation or a tangible computer usable medium or a storage device configured for storing the result.

9. The apparatus of claim 8, wherein said means for producing the one or more annotations comprises: means for generating one or more semantically different alternatives to the selected choice that each, if used in the database query language statement, produce the same result;means for generating reasons for not performing one or more rewrites of the database query language statement; andmeans for generating information to be displayed to a developer of the database query language statement.

10. The apparatus of claim 9, further comprising: means for associating the one or more annotations for each selected choice to a corresponding operator of the database query language statement.

11. The apparatus of claim 10, further comprising: means for identifying operators of the database query language statement having a cost above a threshold cost;means for examining the one or more annotations associated with each identified operator; andmeans for generating a structural recommendation about the costly operator to improve the database query language statement.

12. The apparatus of claim 11, wherein said means for generating the structural recommendation comprising: means for retrieving an alternative operator from the one or more annotations;means for comparing a cost of the alternative operator with a cost of the costly operator; andmeans for mapping the recommendation to use the alternative operator to the costly operator in the database query language statement.

13. The apparatus of claim 12, further comprising: means for rewriting the database query language statement with the alternative operator; andmeans for generating an execution plan for the database query language statement with the alternative operator.

14. The apparatus of claim 8, wherein the database query language statement is a SQL statement

15. A volatile or non-volatile computer readable medium storing a computer program of instructions which, when executed by a processing system, cause the system to perform a method comprising: receiving an database query language statement at an optimizer;evaluating one or more choices in a search space to identify or generate an optimal execution plan for the database query language statement;generating an annotated execution plan from the optimal execution plan by producing one or more annotations to explain one or more reasons for the optimizer selecting a choice over an alternative choice during the act of evaluating the one or more choices, whereinthe annotated execution plan is examined for identifying or generating a recommendation for one or more schema changes and the possible statement modification to rewrite the database query language statement.

16. The medium of claim 15, wherein the one or more annotations for each selected choice comprising: one or more semantically different alternatives to the selected choice that each, if used in the database query language statement, produce the same result;reasons for not performing one or more rewrites of the database query language statement; andinformation to be displayed to a developer of the database query language statement.

17. The medium of claim 16, wherein the executed instructions further perform the method comprising: associating the one or more annotations for each selected choice to a corresponding operator of the database query language statement.

18. The medium of claim 17, wherein the executed instructions further perform the method comprising: identifying operators of the database query language statement having a cost above a threshold cost;for each costly operator, examining the one or more annotations associated with the identified operator; andgenerating a structural recommendation about the costly operator to improve the database query language statement.

19. The medium of claim 17, wherein the executed instructions further perform generating the structural recommendation comprising: retrieving an alternative operator from the one or more annotations;comparing a cost of the alternative operator with a cost of the costly operator; andmapping the recommendation to use the alternative operator to the costly operator in the database query language statement.

20. The medium of claim 19, wherein the executed instructions further perform the method comprising: rewriting the database query language statement with the alternative operator; andgenerating an execution plan for the database query language statement with the alternative operator.

21. The medium of claim 15, wherein the database query language statement is SQL statement.

22. A method comprising: receiving a database query language statement at an optimizer;evaluating one or more choices for operators of the database query language statement to generate an optimal execution plan for the database query language statement;selecting a choice for each operator;generating an annotated execution plan from the optimal execution plan by producing and storing one or more annotations in the annotated execution plan to explain one or more reasons for one or more rejected choices that have been rejected by the optimizer for an operator;generating one or more recommendations for one or more schema changes and for changing one of the operators to rewrite the database query language statement based on the annotations; anddisplaying the one or more recommendations on a display apparatus or storing the result in a tangible computer usable medium or a storage device.

23. The method of claim 22, wherein generating recommendations comprises: identifying a costly operator of the database query language statement;identifying a rule from the one or more annotations for the costly operator; andidentifying a less costly operator based on the rule.

24. The method of claim 23, further comprising: recommending that a user rewrite the database query language statement by replacing the costly operator with the less costly operator.

25. A method comprising: receiving a database query language statement at a query optimizer;evaluating one or more choices in a search space at the query optimizer to generate an optimal execution plan for the database query language statement;generating an annotated execution plan from the optimal execution plan by producing one or more annotations in the annotated execution plan, at the query optimizer, to explain one or more reasons for the query optimizer selecting or rejecting a choice over an alternative choice during the act of evaluating the one or more choices, whereinthe annotated execution plan is examined for generating a recommendation for one or more schema changes and the possible statement modification to rewrite the database query language statement;examining, at the query optimizer, the one or more annotations; anddisplaying the recommendation on a display apparatus or storing the result in a tangible computer usable medium or a storage device.

26. The method of claim 25, further comprising: generating, at the query optimizer, appropriate recommendations to improve performance based on rules in a knowledge base.

27. The method of claim 26, wherein each recommendation includes a recommendation to make database schema changes.

28. The method of claim 26, wherein each recommendation is associated with a rationale and informative messages for user understandability.

29. The method of claim 26, wherein each recommendation is mapped to a corresponding SQL operator in the database query language statement.

30. The method of claim 26, wherein the method further comprising: producing rewritten statement text for a user.

31. The method of claim 26, wherein the rules in the knowledge base can be customized by a user by adding additional rules or removing existing rules.