Patent Application
20140310690
The present invention relates to software testing, and, in particular embodiments, to a system and method for generating automated test cases for command line based applications.
Commercial software is tested by a quality assurance (QA) group or agent before release to customers. For example, regression test and new features test are tests that are run for each release. A test framework is an environment which the QA group/agent uses to run the test cases. These test cases usually include test scripts and expected results. During run-time, test scripts are run in the application and returns the test result back to the test framework. A comparison tool in the test framework compares the actual test result with the expected result and returns a status to the QA group/agent. For QA purpose, enough test cases are needed to cover all the features in the application in order to make sure that the tested software is useable and reliable for end users.
In company environments, automated test cases have been created by QA development teams manually. QA developers need to read reference manuals and understand how to use the commands. The developers then create test matrices to make sure that all or enough options and reasonable combinations of the options are tested. The developers then create test scripts and test them in command line environment. When the test passes successfully, QA developers generate test case files, replace values with variables in test scripts of the files, put the files in the format required by the test framework, and then retest the files in the framework environment. Creating portable and re-useable test cases may require multiple days of development. When the command needs to be tested is complex, it is harder for the developer to create test cases to cover all or enough options and combinations for the command. There is a need for an improved scheme for QA testing that resolves some of the issues above, simplifies test case generation, and provide efficient and re-usable testing.
In accordance with an embodiment, a method for generating automated test cases for an application command includes establishing a knowledge base for the application command that includes global knowledge for a plurality of commands and local knowledge about the application command, parsing the application command to identify variable and option parameters in the application command, and replacing, using the knowledge base, at least some of the variable and option parameters with test values to obtain each of the test cases.
In accordance with another embodiment, a method for generating automated test cases for an application command includes reading general knowledge about a plurality of application commands from a global knowledge base for the application commands, reading local knowledge about the application command from a local knowledge base corresponding to the application command, identifying, in a syntax file of the application command, a plurality of command options and command variables, determining which of the command options to include in a test case, and replacing the command variables and any included command options in the test case with test values according to the local knowledge and the general knowledge.
In accordance with yet another embodiment, an apparatus for generating automated test cases for an application command includes a processor and a computer readable storage medium storing programming for execution by the processor. The programming includes instructions to read, from a knowledge base for the application command, global knowledge for a plurality of commands and local knowledge about the application command, parse the application command using a help command to identify variable parameters in the application command, and replace, using the global knowledge and the local knowledge, at least some of the variable parameters with test values to obtain a plurality of test cases.
For a more complete understanding of the present invention, and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawing, in which:
The making and using of the presently preferred embodiments are discussed in detail below. It should be appreciated, however, that the present invention provides many applicable inventive concepts that can be embodied in a wide variety of specific contexts. The specific embodiments discussed are merely illustrative of specific ways to make and use the invention, and do not limit the scope of the invention.
System and method embodiments are provided for creating automated test cases for an application based on knowledge about the application. The applications may be a software, program, or code, including one or more computer oriented command lines in any suitable computer language or the like (e.g., pseudo-code). A knowledge base is built for one or more application commands to be tested. The knowledge base can include global knowledge (e.g., about different application commands) as well as local knowledge about the application command that is tested. The embodiments also use the output of a function or command that is applied to application command lines to identify the keywords and options of the command. This function can be a help command, which is currently available in most of the command line applications for developers, such as Structured Query Language (SQL) and Hive in Hadoop. However, any suitable function or command that is currently available (or may be available in future releases) to developers and can serve this purpose may also be used. The help command or function can be used on the application command lines and the resulting output is scanned or parsed to obtain keywords and options (e.g., command variables or parameters) for the application/command. The options are then assigned suitable values to generate test cases, as described below. The output is parsed and the test cases are generated in accordance to the knowledge established in the knowledge base(s).
The test cases for the application or command(s) can be generated automatically by a testing tool, e.g., a program or software, that is configured for this purpose. The tool receives (e.g., reads) the command to be tested as input and returns (e.g., writes) one or more test cases as output. To generate test cases, the tool uses knowledge about the command from one or more knowledge bases, which may include built-in knowledge, global knowledge, and local knowledge, or any combination thereof. The knowledge bases (described further below) may be separate data bases or combined into one or fewer bases. For example, the built-in knowledge and/or the global knowledge may be based on External Specification (ES) or user manual of the application under test. The ES is created by a project manager or developer(s) and includes knowledge about how end users use the application or software. A user manual may include all the knowledge in ES. The local knowledge base includes more specific knowledge about the command that is tested. The information in the knowledge base(s) can be built prior to generating the test cases for the application.
The built-in knowledge base includes knowledge needed for writing the code of the test tool to generate the test cases. For example, the knowledge includes that a two-byte integer value should be a value between −32768 to 32767. Further, when a user inputs a two-byte integer value, it should be a set of characters 0-9 in addition to “−” and “+”, and “−” and “+” can only be the first character. The naming convention for the application command is also included in this knowledge base. For example, in order to parse a command in the test tool, the parsing function needs to know that when it encounters a “[” character, it needs to find the matching “]” character, and the content between these two characters is an option in the parsed command.
The global knowledge base includes knowledge or information shared by different application/commands. The knowledge is not needed for writing the code of the test tool to generate test cases. An Extensible Markup Language (XML) file (or other suitable file format) can be used to add this knowledge to the testing tool. For example, the global knowledge indicates that a table name (in the application command) should be a string starting with a character ‘a’ to ‘z’ or “_” in addition to other specified or pre-determined characters. The same knowledge may be needed for different commands, such as create table, create view, create index, select, update, and delete commands. If this knowledge is put in a local knowledge base, then there may be significant duplicate information in local knowledge bases corresponding to the different commands. If duplicating the information in different local knowledge bases is acceptable, then the information of the global knowledge base can be put instead in the local knowledge bases. However, saving this shared information in a global knowledge base is more efficient (e.g., to save storage and process time). The information of the built-in knowledge base on the other hand is not put in the local knowledge base for the test tool since this information is already available and used in test tool prior to testing (prior to configuring the test tool).
The local-knowledge base includes knowledge or information specific to the application commander that is tested. The steps 120 and 130 can be implemented in any order. In another embodiment, steps 120 and 130 can be combined to read global/local knowledge from a single data base. However, step 110 is implemented first to establish or read the built-in knowledge to write the test tool's code. Subsequently, the global/local knowledge is read to parse the application command and generate test cases. The knowledge bases are previously established. However, in another embodiment, one or more of the knowledge bases can be created during the implementation or the test case generation process. For example, the local knowledge base can be created in step 130.
The tool then reads (in step 140) information in the command, such as keywords and options, from an output (e.g., a syntax file) generated using a help (or similar) command on the tested command. The help command reads the tested command as input and returns the output or syntax file as output. Alternatively, the output or syntax file is generated in step 140. The tool then parses the output or syntax file (in step 150) and generates an option array. During the parsing process, the tool gets keywords and options in the command. The tool can use the knowledge base(s) (e.g., in steps 110, 120, and 130) to parse the output file using the help command. The knowledge may include syntax conventions such as:
At step 160, the tool scans the array and decides which option(s) to use, for instance based on generating a random number or based on a pre-determined percentage value for option usage. At step 170, the tool generates expected result and test script(s) based on the option array and test framework. During this step, the tool converts the options in the command to real values based on the information in the knowledge base. More details about the steps of method 100 are described below.
In an example, a “create table” command in a SQL application is tested. The “create table” command is parsed using the help command in SQL (e.g., in step 140 of method 100) providing the following output syntax file:
There are five sections in the above output from using the help command to process the “create table” command. The first two sections include two create table commands. The remaining sections include information about the column_constraint, table_constraint, and like_option. The two create table commands can be used as input to create automated test cases, as described further below. However, knowledge about the “create table command” is needed before generating the automated test cases. For the first of two create table commands, a plurality of options are found, including table type option (data type), table name option, exists option, column definition options, inherits option, with option, on commit option, tablespace option, distribute by option, and to option. For the second option in this list, the knowledge of the table name is needed. For the fourth option, the knowledge of the column name, parent table, and data type is needed. This information needs to be in a knowledge base. For example, this information is read from a local knowledge base (e.g., in step 130 of method 100), before creating the test cases.
For example, an XML script can be used to present, in the local knowledge base, the knowledge about table name, column name, and the other needed information above. For example, the knowledge indicates that the data type needs to be one of the data types that are supported and the parent table needs to be a name of a pre-created table on the system. Additionally, if the on commit option is used, then a temp or temporary option is set to “true” and if the distribute by option is used, then the inherits option is set to “false”. The XML script can have the following form:
After creating or obtaining the knowledge above, the “create table” command can be processed or parsed by the help command using this knowledge to provide the output syntax file described above. The information in the output file can then be divided (e.g., as part of step 150 of method 100) to keywords, variables, and options. Dividing the information can be in the following form:
In the information above, “CREATE”, “TABLE”, “(” and “)” are the keywords in the command, while “table_name” and “column_name” are the variables in the command. The other information includes the options for the command. According to the information, the column name definition can occur multiple times (e.g., defining the column name is repeatable), while the other information can occur once. After obtaining this knowledge or information about the “create table” command, the information can be put in an internal array, for instance as shown in Table 1.
In Table 1, items 2, 7, 9, 10, 11, 13, and 14 have options. In an embodiment, an option handling function can be used to process these options and determine possible values that can be assigned. For example, option number 2 in Table 1 can be further divided as shown in Table 2. For this option, there can be six possible values for the table type, which are “ ” (no specified type), “UNLOGGED”, “GLOBAL TEMPORARY”, “LOCAL TEMPORARY”, “GLOBAL TEMP” and “LOCAL TEMP”.
The same analysis can be used for the other options. After generating all possible values for all options in the “create table” command, the tests cases can be generated. One scheme to generate the test cases comprises generating first a random number that determines which option(s) to use in the command. When generating the test cases, in addition to using random numbers, the values 0 and 1 may be used to allow the group of generated test cases to include none of the options (e.g., for number 0) and all the options (e.g., for number 1). For the rest of the test cases, the generated random number indicates which combination of options to use. In another scheme, each option is assigned a value in the local knowledge base that indicates the percentage usage of that option of all considered test cases. For example, if the knowledge base indicates that 80% of the test cases should use the “distribute by” clause, then the “distribute by” option is selected in 80% of all the test cases.
Since the test cases are generated automatically by the testing tool, various combinations of options can be easily covered in test cases. Creating the test cases using the internal array (e.g., as in Tables 1 and 2) may also ensure having successful test results without erroneous values. The following are examples of the test cases generated as described above for the “create table” command:
Negative test cases that are expected to provide failure results can also be generated. Similar to the test cases of the “create table” command, test cases can be generated by the testing tool for other commands. The tool can generate test cases for commands in different applications, for example SQL, Hadoop Hive, HBase or other applications. For instance, in the output of the “create table” using the help command in Hive can have the following form:
The CPU 310 may comprise any type of electronic data processor. The memory 320 may comprise any type of system memory such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), a combination thereof, or the like. In an embodiment, the memory 320 may include ROM for use at boot-up, and DRAM for program and data storage for use while executing programs. In embodiments, the memory 320 is non-transitory. The mass storage device 330 may comprise any type of storage device configured to store data, programs, and other information and to make the data, programs, and other information accessible via the bus. The mass storage device 330 may comprise, for example, one or more of a solid state drive, hard disk drive, a magnetic disk drive, an optical disk drive, or the like.
The processing unit 301 also includes one or more network interfaces 350, which may comprise wired links, such as an Ethernet cable or the like, and/or wireless links to access nodes or one or more networks 380. The network interface 350 allows the processing unit 301 to communicate with remote units via the networks 380. For example, the network interface 350 may provide wireless communication via one or more transmitters/transmit antennas and one or more receivers/receive antennas. In an embodiment, the processing unit 301 is coupled to a local-area network or a wide-area network for data processing and communications with remote devices, such as other processing units, the Internet, remote storage facilities, or the like.
While this invention has been described with reference to illustrative embodiments, this description is not intended to be construed in a limiting sense. Various modifications and combinations of the illustrative embodiments, as well as other embodiments of the invention, will be apparent to persons skilled in the art upon reference to the description. It is therefore intended that the appended claims encompass any such modifications or embodiments.
