Patent Application
20080228805
Presently, two major methods of testing a computing system are that of static and dynamic testing. In static testing, a sequence of operations is specifically selected. A number of expected reactions are determined. A selected sequence of operations is then executed, and the tester verifies that observed reactions match the expected reactions. Static tests are relatively easy to write and usually are the first type of testing done.
In dynamic testing, a code's reaction to variables that change with time is examined. Two types of dynamic testing are stochastic testing and model-based testing. In stochastic testing, a sequence of operations is randomly selected, executed against a system, and the reactions are observed. The system under test typically passes individual tests, but the goal of stochastic testing is to see if the system can pass a large number of individual tests.
Model-based testing allows for a more focused method than that of stochastic testing. In model-based testing, a large number of sequences of operations are randomly selected according to what operations are likely or possible based on previous operations. These operations are then executed against a system, and the reactions are observed. Generally, model-based testing applies tests randomly selected from all possible sequences of operations within a system's search space to a system.
There exist many limitations though to the current state of technology with respect to testing systems. While static tests are easy to write, they are difficult to keep up-to-date due to the rapid increase in the demand for enhanced tests. Another limitation to static testing is the pesticide paradox problem, wherein only the bugs in the code covered by the static tests are found and fixed. Problems that might be found by a large number of randomly selected tests, such as is done in stochastic testing, may be missed.
While stochastic testing may address the pesticide paradox problem, it also has its limitations. For example, while stochastic testing randomly selects a large number of sequences of operations to execute, this method is not very focused. Additionally, while the refined model-based testing allows for a more focused test, it involves a very broad search space, parts of which may be tested unnecessarily.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
A method for testing a system is disclosed. The technology initially accesses a database having a plurality of stand-alone pieces-of-code, wherein each piece-of-code includes a sequence of operations to be performed on a system. One or more of a plurality of stand-alone pieces-of-code within a database are selected to generate a set of at least two stand-alone pieces-of-code, based on a testing scenario. A variable test sequence from a set of at least two stand-alone pieces-of-code is then generated. This variable test sequence is then utilized to test a system based on a testing scenario.
Embodiments of the present technology enable reusable stand-alone pieces-of-code to be arbitrarily combined within a framework in different ways so as to cover different scenarios, while being subject to tester-defined conditions. The stand-alone pieces-of-code can be dynamic in nature, which avoids the pesticide paradox problem associated with a static test. Furthermore, since the stand-alone pieces-of-code are configured to be reusable, any enhancements to stand-alone pieces-of-code may be employed by any users of the testing system. Hence, alternative embodiments of the present technology permit stand-alone pieces-of-code to be reused, easy to up-date due to its' dynamic nature, and constrained and controlled for more focused testing.
The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the technology for providing a method for testing a system and, together with the description, serve to explain the principles discussed below:
The drawings referred to in this description should be understood as not being drawn to scale except if specifically noted.
Reference will now be made in detail to embodiments of the present technology for providing a method for testing a system, examples of which are illustrated in the accompanying drawings. While the technology for providing a method for testing a system will be described in conjunction with various embodiments, it will be understood that they are not intended to limit the present technology for providing a method for testing a system to these embodiments. On the contrary, the present technology for providing a method for testing a system is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope the various embodiments as defined by the appended claims.
Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present technology for providing a method for testing a system. However, the present technology for providing a method for testing a system may be practiced without these specific details. In other instances, well known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the present embodiments.
Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present detailed description, discussions utilizing terms such as “accessing”, “selecting”, “generating”, “utilizing”, “storing”, “reusing”, “choosing”, “segmenting”, “determining”, and “receiving” or the like, refer to the actions and processes of a computer system, or similar electronic computing device. The computer system or similar electronic computing device manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission, or display devices. The present technology for providing a method for testing a system is also well suited to the use of other computer systems such as, for example, optical and mechanical computers.
With reference now to
System 100 of
System 100 also includes computer usable non-volatile memory 110, e.g. read only memory (ROM), coupled to bus 104 for storing static information and instructions for processors 106A, 106B, and 106C. Also present in system 100 is a data storage unit 112 (e.g., a magnetic or optical disk and disk drive) coupled to bus 104 for storing information and instructions. System 100 also includes an optional alphanumeric input device 114 including alphanumeric and function keys coupled to bus 104 for communicating information and command selections to processor 106A or processors 106A, 106B, and 106C. System 100 also includes an optional cursor control device 116 coupled to bus 104 for communicating user input information and command selections to processor 106A or processors 106A, 106B, and 106C. System 100 of the present embodiment also includes an optional display device 118 coupled to bus 104 for displaying information.
Referring still to
System 100 is also well suited to having a cursor directed by other means such as, for example, voice commands. System 100 also includes an I/O device 120 for coupling system 100 with external entities. For example, in one embodiment, I/O device 120 is a modem for enabling wired or wireless communications between system 100 and an external network such as, but not limited to, the Internet. A more detailed discussion of the present technology for providing a method for testing a system is found below.
Referring still to
The computing system 100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the present technology. Neither should the computing environment 100 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the example computing system 100.
The present technology may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The present technology may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer-storage media including memory-storage devices.
As an overview, in one embodiment, the present technology provides a method for utilizing a database's stand-alone pieces-of-code for testing a system. Moreover, in one embodiment the stand-alone pieces-of-code are provided such that a testing apparatus can easily combine and reuse the stand-alone pieces-of-code. The testing apparatus of the present technology may also combine disparate pieces-of-code in a focused manner to create variable test sequences. These variable test sequences are then applied to a system.
With reference now to
Stand-alone pieces-of-code 215 . . . n represents a predetermined number of stand-alone pieces-of-code besides that of pieces-of-code 215a, 215b, or 215c. For example, there may be just one stand-alone piece-of-code represented by 215a, twenty five stand-alone pieces-of-code represented by 215a through 215y, or any number over twenty five stand-alone pieces-of-code represented by 215 . . . n.
In one embodiment, stand-alone pieces-of-code 215 . . . n are reusable pieces-of-code that perform a sequence of operations. In one embodiment, each stand-alone pieces-of-code 215 . . . n may contain any number of sequences of operations to be performed on system 100. For example, stand-alone piece-of-code 215a may contain one sequence of operations, or it may contain 1000 sequences of operations. There may be any number of stand-alone pieces-of-code 215 . . . n within database 210.
Referring again to
Referring still to
Referring now to 230 of
In general, the term testing scenario describes a targeted search area and environment which a user wishes to test. Factors to be considered regarding a targeted search area and testing environment include, but are not limited to, the following: time constraints; search space breadth (set of all sequences of operations possible to test); importance of specific areas of the search space; desire to perform an arbitrary application of a test to a specific search area; desire to perform repeatable tests; and the operations needed to connect stand-alone pieces-of-code 215 . . . n in order to perform a test.
In one embodiment, the stand-alone pieces-of-code selector 230 receives input for selecting the pieces-of-code from an operating system such as operating system 122 of
In still another embodiment, the stand-alone pieces-of-code selector 230 receives input for selecting the pieces-of-code from a user. A user may enter input regarding a particular selection of stand-alone pieces-of-code 215 . . . n, by any number of means described in
Additionally, stand-alone pieces-of-code selector 230 selects and uses operations such as but not limited to addition, subtraction, multiplication, and division to connect stand-alone pieces-of-code 215 . . . n in order to perform intended tests.
Referring again to
Referring now to
A combination of stand-alone pieces-of-code, such as 215a, 215d, and 215z, is called a scenario definition. A scenario definition is a particular combination of scenario stand-alone pieces-of-code. Scenario stand-alone pieces-of-code are stand-alone pieces-of-code with one or more sets of variations.
Referring now to
Scenario stand-alone pieces-of-code 215 . . . n may be made up of the following two classes: operators and verifiers. Referring to
Referring again to
System 100 may be any part of example computer system 100 that may be tested for expected results and/or accuracy. For example, variable test sequences may be executed on operating system 122, applications 124, modules 126, and or/data 128.
In one embodiment, testing apparatus 200 has a variable test sequence store which is configured to store a record of a variable test sequence for repeatability. In another embodiment, testing apparatus 200 generates a summary of a test result. For example, after variable test sequence utilizer 250 executes variable test sequences upon system 100, test results are generated. A summary of the test result is generated and sometimes stored. Then this summary of a test result may be accessed by stand-alone pieces-of-code selector 230.
With reference now to
Referring now to 305 of
Referring now to 310 of
A testing scenario describes a targeted search area and environment which a user wishes to test. Factors to be considered regarding a targeted search area and testing environment include, but are not limited to, the following: time constraints; search space breadth (set of all sequences of operations possible to test); importance of specific areas of the search space; desire to perform an arbitrary application of a test to a specific search area; the desire to perform repeatable tests; and the operations needed to connect stand-alone pieces-of-code 215 . . . n in order to perform a test. For example, stand-alone pieces-of-code selector 230 selects and uses operations such as, but not limited to, the following: addition; subtraction; multiplication; and division to connect stand-alone pieces-of-code 215 . . . n in order to perform intended tests regarding calculator functions.
In one embodiment, stand-alone pieces-of-code selector 230 chooses a sequence of operations to be performed on system 100 based upon a functionality of stand-alone pieces-of-code 215 . . . n. For example, the functionality of stand-alone pieces-of-code 215 . . . n may relate to the following: what system 100 provides; and what a user anticipates to be an expectation from people who would want to reuse stand-alone pieces-of-code 215 . . . n. For example, in testing a PDA's addition functions of its integrated calculator, the code selector 230 chooses stand-alone pieces-of-code 215 . . . n which contain negative numbers, along with an addition operation. In this manner, a PDA calculator's addition functions relating to negative numbers may be tested.
In another embodiment, a stand-alone pieces-of-code selector 230 makes a selection of one or more of a plurality of stand-alone pieces-of-code 215 . . . n based on input from operating system 122. For example, operating system 122 is pre-programmed to analyze the targeted search area and testing environment in order to determine what scenario to select. Stand-alone pieces-of-code selector 230 is communicatively coupled to operating system 122, and receives guidance based on the scenario determined by operating system 122.
In still another embodiment of the present technology, stand-alone pieces-of-code selector 230 makes a selection of one or more of a plurality of stand-alone pieces-of-code 215 . . . n based on user input. A user may enter input regarding a particular selection of stand-alone pieces-of-code 215 . . . n, by any number of means described in
Referring now to 315 of
With reference now to 320 of
One embodiment of the present technology provides a method for testing a system involving storing a record of an executed variable test sequence. For example, after variable test sequence utilizer 250 executes a variable test sequence on a system I 00, the executed variable test sequence is stored on testing apparatus 200 or on an element communicatively coupled with testing apparatus 200.
After database 210 stores a variable test sequence, testing apparatus 200 may access database 210 and its recorded executed variable test sequence. Furthermore, testing apparatus 200 may then reuse the variable test sequence which has already been executed and recorded in order to provide a repeatable test. For example, a user may want to repeat a test in order to test a particular function of a system that previously did not provide an expected result after the first variable test sequence was executed. Therefore, the user would want to access database 210 and retrieve a specific recorded variable test sequence in order to have variable test sequence utilizer 250 execute this same specific certain variable test sequence upon system 100 or other system.
In another embodiment of the present technology, a summary of a test result generator of testing apparatus 200 generates a summary of the test result after variable test sequence utilizer 250 executes a variable test sequence upon system 100. Furthermore, the present technology provides a record of the summary of the test result. In another embodiment the summary of the test result is stored by a summary of a test result store. However, this generated summary of the test result may be stored on database 210 or some other element communicatively coupled to example computing system 100. This summary of the test result may be accessed by for example, stand-alone pieces-of-code selector 230. Stand-alone pieces-of-code selector 230 might access this summary of a test result in order to determine if a test needs to be repeated so as to check for expected results and/or accuracy.
With reference now to
Referring now to 405 of
For example, if a user anticipates that a calculator's functions will be tested using stand-alone pieces-of-code, the user will utilize the full range of possible operating functions to segment these functions into individual compartments. For example, if the user anticipates testing the addition function of a calculator, the user may segment the sequence of operations into a stand-alone piece-of-code 215a containing positive numbers, stand-alone piece-of-code 215b containing negative numbers, stand-alone piece-of-code 215c containing decimal numbers, and stand-alone piece-of-code 215d containing hexadecimal numbers.
Referring now to 410 of
For example, a class type may be one involving decimal numbers and one involving hexadecimal numbers. All of the numbers within stand-alone pieces-of-code 215a are decimal numbers, and all of the numbers within stand-alone pieces-of-code 215f are hexadecimal numbers. When testing the multiplication function of a calculator on a PDA, a stand-alone pieces-of-code selector 230 may choose stand-alone pieces-of-code 215a, a multiplication operation, and then stand-alone pieces-of-code 215f.
Referring now to 415 of
Referring now to 420 of
In one embodiment of the present technology, performing a method for building database 210 for testing system 100 includes database 210 storing stand-alone pieces-of-code 215 . . . n based on class type.
In one embodiment, performing a method for building database 210 for testing system 100 includes database 210 receiving a variable test sequence from a variable test sequence store. For example, database 210 receives a variable test sequence from another element of testing apparatus 200 or from a source external to testing apparatus 200. The variable test sequence store stores a variable test sequence. The variable test sequence store may be integrated within testing apparatus 200 or may be external to and communicatively coupled to testing apparatus. The variable test sequence store receives variable test sequences from one or more of the elements of testing apparatus 200, such as variable test sequence generator 240, variable test sequence utilizer 250, or even system 100 itself.
Thus, the present technology provides a computer implemented method for testing system 100. Moreover, the present technology's enablement of testing apparatus 200 to selectively combine and reuse already developed stand-alone pieces-of-code 215 . . . n reduces the time it takes to develop and run a test on system 100. Since stand-alone pieces-of-code 215 . . . n are reusable, any enhancements to individual stand-alone pieces-of-code 215 . . . n are usable by any of the users of pieces-of-code 215 . . . n. Additionally, the present technology enables a tester to focus a test on a section of system's 100 broad search space so that important dimensions of a search space are covered, considering the limited time and resources available for the test. Furthermore, the present technology enables a test to be repeated. Repeating tests enables a system to debug a problem as well as to determine if a bug is fixed.
Although the subject matter has been described in a language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
