Patent Application
20060230320
The present invention relates generally to computer software testing; and more particularly to a system and method for automatically generating test cases for computer software.
Reliable and successful software is built through sound, efficient and thorough testing. However, software testing is labor intensive and expensive and accounts for a substantial portion of commercial software development costs. At the same time, software testing is critical and necessary to achieving quality software. Typically, software testing includes test suite generation, test suite execution validation, and regression testing.
Test suite generation involves creating a set of inputs which force the program or sub-program under test to execute different parts of the source code. This generated input set is called a “test suite.” A good test suite fully exercises the program's functionality including the individual functions, methods, classes, and the like.
Unit testing process tests the smallest possible unit of an application. For example, in terms of Java, unit testing involves testing a class, as soon as it is compiled. It is desirable to automatically generate functional unit tests to verify that test units of the system produce the expected results under realistic scenarios. This way, flaws introduced into the system can be pinpointed to single units when functional unit tests are maintained for regression.
Conventional unit test generators create white-box and black box unit tests that test boundary conditions on each unit. Moreover, existing automatically generated unit tests may be using test stimulus that does not represent a realistic input in the system. Thus, the extra, unnecessary generated unit tests produce “noise” or unimportant errors. Furthermore, these unit tests may not be testing the functionality that is critical to the rest of the system.
A GUI-based record-and-playback testing can determine if the system is functioning correctly as a whole. However, when a problem is introduced in the system, it cannot locate the source of the problem. This requires development resources to manually narrow down the problem from the system level to the individual unit causing the problem.
Therefore, there is a need for unit tests that are capable of pinpointing flaws to single units, while the functional unit tests are maintained for regression.
In one embodiment, the present invention is a method and system for generating test cases for a computer program including a plurality of test units. The method and system execute the computer program; monitor the execution of the computer program to obtain monitored information; and generate one or more test cases utilizing the monitored information.
In one embodiment, the present invention is a method and system for generating test cases for a computer program including a plurality of test units. The method and system execute the computer program; monitor the execution of the computer program to obtain execution data; analyze the execution data to identify run time objects used by the computer program; store state of the identified objects in an object repository. The invention is then capable of generating one or more test cases utilizing the stored execution data and information about the identified objects.
In one embodiment, the present invention is a method and system for generating test cases for a computer program including a plurality of test units. The method and system store a plurality of test cases; select a test case form the plurality of stored test cases; creating a parameterizes test case by parameterizing selected fixed values in the selected test case; and varying the parameters of the selected test case. For example, a first parameter is selected and heuristically swept, while the rest of the parameter are kept fixed.
In one embodiment, the present invention automatically generates unit tests by monitoring the system program being executed under normal, realistic conditions. Stimulus to each test unit is recorded when the test units are exercised in a correct context. State information and results of external calls are recorded so that the same context can be later replicated. Unit tests are generated to recreate the same context and stimulus. Object state and calling sequences are reproduced the same as in the executing system. This produces realistic unit tests to be used in place of, or in addition to system level tests.
In one embodiment, the present invention is a method for test generation including; observing an application when being executed and creating unit test case for one or multiple objects based on information gathered from the execution. Examples of recorded stimulus include input parameter values to function calls, return values of calls from one function to another, call sequence and base object information for object-oriented functions, and data field values. The invention then stores the gathered information about the executed objects during execution of the application in an object repository and utilizes the stored information in the object repository for unit test case(s) generation. The generated unit test cases are used, for example, for boundary testing and/or regression testing. The invention takes a unit test case and analyses, parameterizes and runs it with different parameters to increase test coverage for the application or find errors in the application.
When designing an application, functionality is broken down into components so that they can be isolated and clearly defined. The same paradigms are applied when testing an application. At the lowest functional level, automated unit tests, such as those created by the present invention, provide a good level of testing for method level functionality. However, as functional blocks become larger, they become more inter-related and the associated tests become more sequential. These sequential tests are the type of tests that developers manually implement and the type of tests that the present invention automates by monitoring the application. The question for sniffing becomes ‘what needs to be monitored to create the test?’
Given a functional block with the following steps: A: Load configuration file, B: Perform Operation, and C: Display result, one option for testing would be to test each of these steps independently, using sniffing to create test cases for each step. For example; sniff step A, generate test cases, validate test cases, sniff step B, generate test cases, validate test cases, finally, sniff step C, generate test cases, and validate test cases. This process results in a series of functional unit tests that test each step and, by inference, each of the previous steps. This means that the tests for step C will test the entire functional block, including steps A and B.
A second option is to perform sniffing on just step C. This enables the efficient creation of functional tests that exercise the functionality of the entire block.
The present invention provides the software developers with the option to generate tests with or without automatically generated stubs. Therefore, automatically generated stubs should only be used when the generated tests are going to be re-run outside of the ‘live’ environment. Stubs are objects (and methods) that mimic the behavior of intended recipients and enable the isolation of the code under test from external resources. This allows for a unit test to be re-deployed independently of a ‘live’ environment. However, when creating and executing functional tests, it is often useful to access the external resources and run these tests within a ‘live’ environment.
Once a functional test has been created, the present invention can also enable the test to be parameterized, wherein the code can be automatically refactored to enable a wide variety of test values to be used. For example, given a test for the previous example:
<CODE>
<GENERATED TEST>
The present invention can refactor this test to be as follows:
<REFACTORED CODE>
Allowing the developer to extend the functional test by simply supplying more values to the parameterized test:
<EXAMPLE TEST CASE>
In other words, by monitoring the final logical point in a functional block, the present invention automates the creation of functional tests for that block and the steps within it. These tests can then be executed within the ‘live’ environment (without stubs) and using parameterization, the tests can run over a range of different data values increasing the level of functionality tested.
For example, data can be acquired for processes ran on Java VM using DI (Debugger Interface), PI (Profiler Interface), or TI (Tool Interface) for Sun Microsystem's™ JDK. Alternatively, the source or the binary code can be instrumented. Also, the combination of the above mentioned data acquisition means can be employed.
The driver program then initializes a recorder module 1011. Control events 1007 and 1009 are sent to the recorder. These events may be sent by the driver 1002, the monitored program, or both. Example of control events include, “Start Recording” 1010, and “Stop Recording” 1012. Events also control the granularity of recorded data. For example, “Record method calls”, “Record method calls and objects”, etc. Execution data 1008 is then sent to the recorder 1011.
Recorder 1011 may send control events to the monitored program 1005 or the driver 1002. These events may be, for example, data granularity control events like, turning on/off object recording, execution control events like, “suspend execution” or “kill”. Execution data is then processed by the recorder and stored in an Execution Record Database 1012. The tested program is prepared for recording (1003) by appending arguments for the launch to enable the required program type interfaces. The prepared program is then launched in 1004, and terminated in 1006.
In one embodiment, sequence is implied form the temporal recording of the sequence of calls, that is, no notion of child/parent calls is recorded per se, but rather, is implied from the recorded sequence). The Recorder Event Listener 2005 writes events sequentially to the Execution Record Database 2008, which preserves the order of events for later processing by a test generation system.
In one embodiment, objects may be added to the Object Repository using one or more of the following methods,
In one embodiment, for each tested method, the Test Generating System 4003:
In one embodiment, the input stimulus to generated unit tests include:
In one embodiment, the outcomes are:
In one embodiment, the object inputs and outcomes are generated based on calling sequences and filtering data. Test generation system has an option to limit number of calls in the sequence leading to the object creation to improve performance. Effectively, the object states which require more than a maximal allowed number of method calls are not used in test generation. Objects from the Object Repository may contain a snapshot of the recorded state and can be reloaded in a unit test at some point using the Object Repository API.
In one embodiment, filtering data for generation and generation options may include:
As an example, during execution of a JAVA application, the present invention monitors the Java Virtual Machine and produces functional unit tests based on what it observes by generating unit tests in Java source code that use the JUnit framework and contain test stimulus derived from recorded runtime data. These tests can then be validated and executed as part of the testing infrastructure to ensure that the code is operating to specification.
The following example describes usage of some embodiments of the present invention utilizing a test tool, for example, Jtest™ from Parasoft Corp.™.
If there is already an executable module or application, Jtest™ provides a fast and easy way to create the realistic test cases required for functional testing, without writing any test code. A running application that Jtest™ is configured to monitor can be execised. Jtest™ tool observes what methods are called and with what inputs, then it creates JUnit test cases with that data. The generated unit test cases contain the actual calling sequence of the object and primitive inputs used by the executing application. If code changes introduce an error into the verified functionality, these test cases will expose the error.
One way to use this method for functional testing is to identify a point in development cycle where the application is stable (e.g., when the application passes the QA acceptance procedures). At this point, the acceptance procedure is completed as Jtest™ monitors the running application and creates JUnit test cases based on the monitored actions. In this way, one can quickly create a “functional snapshot”: a unit testing test suite that reflects the application usage of the modules and records the “correct” outcomes. This functional snapshot test suite may be saved independent of the reliability test suite, and run nightly. Any failures from this test suite indicate problems with the application units' expected usage.
To generate realistic functional test cases from a running module/application in Jtest™:
After the application exits, unit tests will be generated based on what was monitored while the application was executing. The JUnit test cases that are created are saved in the same location as the test cases that were generated based on code analysis that Jtest™ performs.
To generate realistic functional test cases by exercising the sample Runnable Stack Machine application:
After the application exits, unit tests will be generated based on what was monitored while the application was run. The JUnit test cases that are created are saved in a newly created project Jtest Example.mtest. mtest projects are created when test cases are generated though monitoring.
To view the generated test cases:
To use the same monitoring technique to generate additional test cases for this application:
It will be recognized by those skilled in the art that various modifications may be made to the illustrated and other embodiments of the invention described above, without departing from the broad inventive scope thereof. It will be understood therefore that the invention is not limited to the particular embodiments or arrangements disclosed, but is rather intended to cover any changes, adaptations or modifications which are within the scope and spirit of the invention as defined by the appended claims.
This patent application claims the benefits of U.S. Provisional Patent Application Ser. No. 60/669,281, filed on Apr. 7, 2005 and entitled “System And Method For Test Generation,” the entire content of which is hereby expressly incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 60669281 | Apr 2005 | US |
