Patent Application
20090077016
The present invention relates to the field of electronic database management.
SQL performance is a very critical component of the overall performance of a database system. Although database applications are tested and verified before delivery to customers, the performance of SQL statements in the database applications is difficult to predict. Every time an SQL statement is executed, a different execution plan may be generated by the query optimizer. Often, the execution plan is one that has not been tested before and one whose performance has not been analyzed and verified. Should the query optimizer choose a plan that does not give the best performance, the performance of the system as a whole may suffer.
SQL tuning is a process where the performance of certain SQL statements is analyzed and changes are implemented in an effort to improve the performance of the SQL statements. However, SQL tuning is complex and time-consuming, requiring expertise in query optimization, access design, and SQL design. Often, this difficult task is performed manually by the database administrator (DBA), who must also re-tune SQL statements as the workload set on the database system and the database system itself change over time.
An SQL Tuning Advisor, as described in “AUTOMATIC SQL TUNING ADVISOR”, application Ser. No. 10/936,778, filed Sep. 7, 2004, the entire contents of which is hereby incorporated by reference as if fully set forth herein, provides the DBA with the functionality of tuning SQL statements by generating tuning recommendations for the SQL statements. The SQL Tuning Advisor can, for example, perform access path analysis and recommend creating new indexes, perform statement structure analysis and recommend better written statements, and perform data statistics analysis and recommend replacing missing or stale data statistics with updated statistics.
However, using the SQL Tuning Advisor still requires significant time, effort, and involvement from DBAs. In order to use the SQL Tuning Advisor, DBAs are required to find SQL statements that are exhibiting poor performance as candidates for tuning, feed the candidate SQL statements into the SQL Tuning Advisor, and manually evaluate the results and tuning recommendations generated by the SQL Tuning Advisor to decide which tuning recommendations to implement. The SQL tuning process implemented by the SQL Tuning Advisor is itself time-consuming and, on a busy system, may need to be scheduled by the DBA. Furthermore after deciding which tuning recommendations to implements, DBAs must also monitor the performance of the database system after implementation of the tuning recommendations to check that performance gains have been achieved according to expectations. If the performance has instead deteriorated, DBAs must then perform further analysis to respond to the problem. To prepare for such a situation, DBAs also need to keep a record of prior tuning actions. Finally, this tuning process must be repeated periodically because both workload sets and database systems may change over time.
Thus, there is a need for a fully automated process for tuning SQL statements that require minimal time and effort from database administrators.
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:
In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.
Although the embodiments of the invention are described using the term “SQL”, the invention is not limited to just this particular database query language, and may be used in conjunction with other database query languages and constructs.
A process performs the following steps to provide users of database systems with a fully-automated process for tuning database query language (e.g., SQL) statements:
Additionally, the fully-automated tuning process runs in a controlled environment so that the resource-intensive tuning steps in the process do not impact other activities on the database system.
Finally, the fully-automated tuning process generates detailed reports that describe how the SQL statements are tuned and which tuning recommendations were implemented and that can be provided to DBAs.
In the fully-automated tuning process, candidate SQL statements for tuning are automatically identified. The candidate SQL statements identified for tuning are SQL statements that have been determined to impact the performance of the database system the most and for which the SQL Tuning Advisor's built-in algorithms are the most likely to help. The identification of candidate SQL statements for tuning is performed in step 102 of flow diagram 100 in
In step 102, SQL statements are identified from a workload set, which is a collection of SQL statements and performance information associated with the SQL statements. According to one embodiment, the workload set is stored as a SQL Tuning Set. SQL Tuning Sets are described in “SQL TUNING SETS”, application Ser. No. 10/936,449, filed on Sep. 7, 2004, the entire contents of which are hereby incorporated by reference. An SQL Tuning Set is a persistent database object that stores one or more database query language statements and performance information for each statement. The performance information may include, for example, execution measurements and execution context of each statement.
The SQL statements and associated performance information may come from a source such as an Automatic Workload Repository (AWR), which stores the statements executed during a specific period of time and the performance information associated with the executed statements. Automatic Workload Repositories (AWRs) are described in further detail in “AUTOMATIC WORKLOAD REPOSITORY BATTERY OF PERFORMANCE STATISTICS”, application Ser. No. 10/934,344, filed on Sep. 3, 2004, the entire contents of which are hereby incorporated. According to one embodiment, the AWR collects and stores SQL statements and associated performance information from the past week and at an interval of one hour. The SQL statements collected by the AWR may include bind values and other critical components of the SQL's compilation environment, such as optimizer parameter values. The performance information associated with the SQL statements and collected by the AWR may include: total time spent in execution, CPU time, I/O time, the number of I/Os, and the frequency of execution.
The data collected by the Automatic Workload Repository (AWR) is analyzed in step 102 to determine which SQL statements fall within the following four categories:
(1) SQL statements that are high-load over the past week;
(2) SQL statements that are high-load over any day in the past week;
(3) SQL statements that are high-load over any hour in the past week; and
(4) SQL statements that have a high response time.
The first category of SQL statements includes SQL statements that are high-load over the past week. These are SQL statements that have the highest cumulative values of execution statistics, where each statistic represents a total amount of resources consumed over multiple executions of the SQL statement. Since one goal of performance tuning is to decrease the total amount of time spent in executing a workload set, the execution times of the SQL statements are key performance metrics. The cumulative combination of execution time and frequency of execution serves as a good proxy for measuring how significant an SQL statement is in a workload set. An SQL statement whose execution time (e.g., one-tenth of a second) is small may nonetheless constitute a large part of a workload set if the statement is executed very frequently relative to other statements (e.g., one million times a week) over the course of the past week. Thus, tuning the SQL statements that have high combinations of execution times and frequencies of execution will likely improve the overall performance of the database system.
The second category of SQL statements includes SQL statements that are high-load over any day in the past week. As just discussed, high-load statements are statements that have long execution times and that are frequently executed. The first category includes statements that are high-load over the course of a past week. However, statements that are not high-load over the entire course of the past week may nonetheless significantly affect the performance of the database system. For example, a particular SQL statement may be executed only on Mondays. On Mondays, however, this particular SQL statement may be executed very frequently and may have long execution times every time it is executed. Thus, while this particular statement may not be one of the highest load statements over the course of an entire week, this particular statement may nonetheless significantly impact the performance of the database system on Mondays. The second category of high-load SQL statements captures statements like this particular statement for tuning.
The third category includes SQL statements that are high-load over any hour in the past week. The first and second categories just discussed capture SQL statements that are high-load over the course of the past week or high-load over the course of a day in the past week, but do not capture SQL statements that are high-load over an hour in the past week. For example, a particular SQL statement may be executed on only Tuesdays, and only from ten o'clock in the morning to eleven o'clock in the morning. This SQL statement is thus unlikely to be included in the first two categories. Nonetheless, if this particular SQL statement has a long execution time and is frequently executed during the one-hour interval between ten o'clock and eleven o'clock on Tuesday, it may significantly impact the performance of the database system during that one-hour interval. Therefore, tuning this particular SQL statement may significantly improve the overall performance of the database system during key time periods. These types of SQL statements are included in the third category.
The fourth category includes SQL statements which have long execution times regardless of their frequencies of execution. An SQL statement that is not frequently executed, and thus may not be included in the first three categories, may nonetheless negatively impact the response time one user experiences if it takes a very long time to execute (e.g., thirty minutes). Therefore, tuning statements like this may significantly improve overall performance. According to one embodiment, only SQL statements that have been executed at least two times over the course of the past week are included in the fourth category of SQL statements. Limiting the inclusion of SQL statements to only those statements that have been executed at least twice ensures that no tuning resources are expended on the occasional SQL statement that has been executed only once over the course of an entire week. If an SQL statement is unlikely to be executed again, tuning the SQL statement will not be an efficient use of limited system resources.
SQL tuning is a resource-intensive process, and it is likely that not all the SQL statements identified and included in the four categories described above may be tuned within the limited amount of time and resources allotted to SQL tuning. Therefore, the SQL statements identified in the four categories may be further ranked and prioritized according to their relative importance so that during the tuning process, the SQL statements with the highest priorities are executed first.
According to one embodiment, SQL statements within a single category may be ordered by performance metrics that are being optimized by the tuning process. The performance metrics being optimized may be based on the most repeatable execution statistics. For example, CPU time and buffer gets are known to repeat very reliably over multiple executions, so the SQL statements in a single category may be ordered according to a combination of CPU time and buffer get statistics.
According to another embodiment, SQL statements are tuned in an order that interweaves statements from each of the four categories. Interweaving SQL statements from each of the four categories prevents a scenario where statements from certain categories are always tuned while statements from other categories are never tuned. Interweaving may be performed by assigning different weights to different categories. A higher weight may be assigned to a category that contains SQL statements over a larger span of time. For example, an interweaving scheme may tune three statements from the first category, two statements from each of the second and third categories, and one statement from the fourth category, before returning to the first category to select further statements for tuning.
According to another embodiment, a history of SQL statements that have been tuned recently is kept. This history contains information about which SQL statements have been tuned during a specific period of time, such as the preceding week or the preceding month. An SQL statement that has been tuned recently (i.e., the SQL statement is contained in the history) is automatically not identified for tuning because it is unlikely that an SQL statement that has already been tuned recently will benefit from further tuning. However, an SQL statement that has been tuned recently may nonetheless be identified for tuning if the SQL statement meets another criterion. For example, if the SQL statement's execution statistics indicate that the SQL statement's performance has deteriorated since the last time that the SQL statement has been tuned, then this indicates that conditions in the database system that affect the SQL statement may have changed since the last time that the SQL statement was tuned, and that retuning of the SQL statement may be beneficial.
Once the SQL statements have been identified, they are tuned in order of priority. The tuning of candidate SQL statements is performed in step 104 in
According to one embodiment, the SQL Tuning Advisor uses the database system's query optimizer that is set to operate in tuning mode. Normally, the query optimizer of a database system compiles SQL statements and generates execution plans. In normal mode, the query optimizer operates with very strict time constraints, usually a fraction of a second, during which time it must find a good execution plan for an SQL statement. In tuning mode, however, the optimizer is under much less stringent time constraints, and can perform longer and more thorough analyses to determine whether the execution plan produced for an SQL statement in the normal mode can be further improved. The output of the query optimizer in tuning mode is not execution plans, but a set of estimates to better inform the cost-based optimizer with the information it requires for producing superior execution plans.
During tuning, the SQL Tuning Advisor performs four types of analyses: (1) statistics analysis; (2) SQL profiling; (3) access path analysis; and (4) SQL structure analysis.
In normal mode, the query optimizer relies on object statistics to generate execution plans. If the statistics are stale or missing, the execution plans that are generated may be poor. During tuning, the SQL Tuning Advisor checks the object statistics and generates recommendations to gather relevant statistics for objects with stale or missing statistics. The identification of inaccurate statistics for an SQL statement is described in further detail in “HIGH LOAD SQL DRIVEN STATISTICS COLLECTION”, application Ser. No. 10/936,427, filed Sep. 7, 2004, the entire contents of which are incorporated by reference herein.
In addition, the SQL Tuning Advisor also produces auxiliary information that includes statistics for objects with no statistics and statistic adjustment factors for different estimates made by the query optimizer. The auxiliary information may be collected by gathering additional information using sampling and partial execution techniques. According to one embodiment, this auxiliary information is stored in an object called an SQL Profile. Each SQL Profile is specific to and associated with a single SQL statement. SQL Profiles are described in further detail in “SQL PROFILE”, application Ser. No. 10/936,205, filed Sep. 7, 2004, the entire contents of which are incorporated by reference herein.
SQL Profiles store statistic adjustment factors that are not themselves statistics but that adjust the pre-existing statistics and/or estimates of statistics of database objects. For example, a cardinality estimate for a particular query predicate in a particular SQL statement may be 10%, while the actual cardinality for that particular query predicate is 30%. The SQL Profile for the particular SQL statement may then contain a cardinality adjustment factor of 3 for the particular query predicate, so that a query optimizer that is generating an execution plan for the particular SQL statement may consult the SQL Profile to multiply any pre-existing cardinality estimates for the particular predicate in the particular SQL statement by 3. As such, execution plans that are generated based on SQL Profiles will be based on more accurate statistical estimates and will therefore be more optimized.
According to one embodiment, the SQL Advisor generates tuning recommendations that recommend the acceptance of SQL Profiles. Once an SQL Profile for a particular SQL statement is accepted, the SQL Profile is stored persistently in a data dictionary that associates the SQL Profile with the particular SQL statement. After the SQL Profile is stored persistently, the next time that the particular SQL statement is analyzed by the query optimizer operating in normal mode, the execution plan for the particular SQL statement will be generated based on statistics that are adjusted by the adjustment factors stored in the SQL Profile that is associated with the particular SQL statement.
One advantage of using SQL Profiles is that, unlike stored outlines, an SQL Profile does not freeze the execution plan of an SQL statement. Because an SQL Profile contains adjustment factors for statistics, these adjustment factors continue to be relevant even when objects such as tables and indexes grow, shrink, are created or are deleted, and even when the data distribution or access path of the associated SQL statement changes. Therefore, an SQL Profile need not be regenerated frequently. After a long period of time, however, an SQL Profile may become outdated. At such times, a new SQL Profile may be generated for the associated SQL statement by running the SQL Tuning Advisor again on the associated SQL statement.
The SQL Tuning Advisor may also perform access path analysis. During access path analysis, the SQL Tuning Advisor explores whether a new index can significantly enhance the performance of an SQL statement. If such an index is identified, a tuning recommendation is generated to recommend the creation of the index.
Finally, the SQL Tuning Advisor performs SQL structure analysis. During SQL structure analysis, the SQL Tuning Advisor identifies common problems with the structure of SQL statements that can lead to poor performance, including syntactic, semantic, or design problems. Further details on SQL structure analysis are described in “SQL STRUCTURE ANALYZER”, application Ser. No. 10/936,426, filed Sep. 7, 2004, the entire contents of which is hereby incorporated by reference as if fully set forth herein. Relevant suggestions for restructuring SQL statements are generated as tuning recommendations.
In order to determine how the tuning recommendations generated by the SQL Tuning Advisor will affect the performance of SQL statements and to determine whether to implement the tuning recommendations on the database system, the tuning recommendations are tested in a test environment. According to one embodiment, testing and execution services such as the ones described in “TEST EXECUTION OF USER SQL IN DATABASE SERVER CODE”, application Ser. No. 12/217,249, filed on Jul. 2, 2008, the entire contents of which is hereby incorporated by reference as if fully set forth herein, are used to perform testing of the tuning recommendations.
In the test environment, the SQL statements are executed with the tuning recommendations enabled, and execution statistics such as execution time are gathered. This is performed in step 106 in
One goal of the step of selecting which tuning recommendations to implement is to choose the subset of the tuning recommendations generated from the tuning step that would improve performance the most without implementing too much change in the database system. According to one embodiment, the results from the testing step just described are analyzed, and only the tuning recommendations that result in substantial benefits are implemented. Furthermore, in order to limit the number of changes made to the database system on a daily basis, a limit on the number of actions taken during maintenance may also be configured so that tuning recommendations are implemented only if the number of changes made does not already exceed the pre-configured limit. As described above, SQL statements which have been identified as good candidates for tuning are tuned in order of priority. Therefore, the tuning recommendations for the highest-priority SQL statements would be implemented first and will unlikely be precluded from implementation by the cap on number of changes. The determination of which tuning recommendations are to be implemented is made in step 108 of
According to one embodiment, only tuning recommendations that have resulted in performance improvement that meet a specific set of criteria are implemented. In one example, only tuning recommendations that improve the execution time by at least threefold are implemented. The threefold improvement is measured in terms of specific execution statistics which are known to have consistent values across multiple executions. CPU time and buffer gets (logical I/Os) are two metrics that may be used to measure improvement because CPU time and buffer gets are both repeatable measurements that encompass important factors in SQL performance.
According to another embodiment, only tuning recommendations that meet a specific set of safety criteria are implemented. For example, in one example, only tuning recommendations that do not increase the I/O times of SQL statements are implemented, even if the tuning recommendations decrease the overall execution times of the SQL statements.
In step 110, the tuning recommendations that have been determined to be implemented are implemented. For SQL Profiles, for example, this means that the SQL Profiles are stored persistently in a data dictionary that associates the SQL Profiles with specific SQL statements. As discussed above, once a particular SQL Profile is stored persistently, the next time that the particular SQL statement is analyzed by the query optimizer operating in normal mode, the execution plan for the particular SQL statement will be generated based on statistics that are adjusted by the adjustment factors stored in the SQL Profile that is associated with the particular SQL statement.
The automatic tuning of SQL statements is a resource-intensive and time-consuming process. The tuning process is run in a controlled environment so that other activities on the machine on which the tuning process is run are not interrupted. Furthermore, in the controlled environment, the tuning process may be restricted to run only for a specified period of time.
According to one embodiment, the automatic tuning process is run in an automated maintenance task that is run on a nightly basis. The frequency, begin time, and end time that the automatic tuning process is run may be customized by the DBA.
Within the specified period of time (e.g., one hour), the tuning process performs the steps of: identifying candidate SQL statements from the AWR for tuning; tuning each SQL statement individually be calling the SQL Tuning Advisor; testing the SQL profiles generated by SQL Tuning Advisor; and optionally implementing the SQL profiles if they meet the criteria (e.g., three-fold performance improvement).
According to one embodiment, the fully-automated tuning process described above is performed by an automated maintenance task. The automated maintenance task caps the resources used by the tuning process to a specific percentage of the total resource consumption on the machine on which the tuning process is being made. The automated maintenance task also caps the total amount of time for the tuning process so that the tuning process does not run for an inordinately long period of time. Finally, the amount of time the tuning process spends tuning each individual SQL statement may be capped by the tuning process itself to prevent the tuning process from spending too much time analyzing one particular SQL statement. In addition, the total amount of time the tuning process spends tuning all the selected SQL statements may also be capped by the tuning process itself. For example, the automated maintenance task may cap the total amount of time spent by the tuning process to four hours, while the tuning process may cap the own total tuning time to one hour and cap the tuning time for a single SQL statement to twenty minutes. According to one embodiment, when any of these time limits is reached, the tuning process may be notified and in response may complete the current tuning activity in such a manner that ensures that the partial results will be consumable by the user at a later time.
According to one embodiment, a separate and independent mechanism monitors the total amount of time that the tuning process has used. If the separate and independent mechanism detects that the total amount of time used by the tuning process has exceeded the specified cap, then the tuning process is terminated.
According to another embodiment, the tuning process monitors whether or not its actions are causing any contentions with other sessions in the database system. When the tuning process detects such a contention, it will stop tuning the SQL statement currently being tuned and commence tuning the SQL statement with the next highest priority. If this action does not immediately resolve the contention, the tuning process may be aborted entirely.
After the tuning recommendations are implemented, the database system is monitored to detect when the tuning recommendations are used and to measure the benefits form the tuning recommendations. This is performed in step 112 of
According to one embodiment, information gathered during the tuning process is compiled into a report, which can then be reviewed by the DBA. This report may include, for example, a list of the SQL statements that were tuned, the tuning recommendations generated by the SQL Tuning Advisor, the results from testing the tuning recommendations, and a list of which tuning recommendations were implemented. The DBA may examine this report to implement additional tuning recommendations or to remove any tuning recommendations that were automatically implemented.
Computer system 200 may be coupled via bus 202 to a display 212, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 214, including alphanumeric and other keys, is coupled to bus 202 for communicating information and command selections to processor 204. Another type of user input device is cursor control 216, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 204 and for controlling cursor movement on display 212. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.
The invention is related to the use of computer system 200 for implementing the techniques described herein. According to one embodiment of the invention, those techniques are performed by computer system 200 in response to processor 204 executing one or more sequences of one or more instructions contained in main memory 206. Such instructions may be read into main memory 206 from another machine-readable medium, such as storage device 210. Execution of the sequences of instructions contained in main memory 206 causes processor 204 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software.
The term “machine-readable medium” as used herein refers to any medium that participates in providing data that causes a machine to operation in a specific fashion. In an embodiment implemented using computer system 200, various machine-readable media are involved, for example, in providing instructions to processor 204 for execution. Such a medium may take many forms, including but not limited to storage media and transmission media. Storage media includes both non-volatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 210. Volatile media includes dynamic memory, such as main memory 206. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 202. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications. All such media must be tangible to enable the instructions carried by the media to be detected by a physical mechanism that reads the instructions into a machine.
Common forms of machine-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punchcards, papertape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer can read.
Various forms of machine-readable media may be involved in carrying one or more sequences of one or more instructions to processor 204 for execution. For example, the instructions may initially be carried on a magnetic disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 200 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 202. Bus 202 carries the data to main memory 206, from which processor 204 retrieves and executes the instructions. The instructions received by main memory 206 may optionally be stored on storage device 210 either before or after execution by processor 204.
Computer system 200 also includes a communication interface 218 coupled to bus 202. Communication interface 218 provides a two-way data communication coupling to a network link 220 that is connected to a local network 222. For example, communication interface 218 may be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 218 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 218 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
Network link 220 typically provides data communication through one or more networks to other data devices. For example, network link 220 may provide a connection through local network 222 to a host computer 224 or to data equipment operated by an Internet Service Provider (ISP) 226. ISP 226 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet” 228. Local network 222 and Internet 228 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 220 and through communication interface 218, which carry the digital data to and from computer system 200, are exemplary forms of carrier waves transporting the information.
Computer system 200 can send messages and receive data, including program code, through the network(s), network link 220 and communication interface 218. In the Internet example, a server 230 might transmit a requested code for an application program through Internet 228, ISP 226, local network 222 and communication interface 218.
The received code may be executed by processor 204 as it is received, and/or stored in storage device 210, or other non-volatile storage for later execution. In this manner, computer system 200 may obtain application code in the form of a carrier wave.
In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.
This application claims benefit of Provisional Appln. 60/972,681, filed Sep. 14, 2007, the entire contents of which is hereby incorporated by reference as if fully set forth herein, under 35 U.S.C. § 119(e).
| Number | Date | Country | |
|---|---|---|---|
| 60972681 | Sep 2007 | US |
