Currently, when software is purchased there is no direct indication of the quality of the software. It is generally recognized that quality assurance is primarily a function of testing. However, software testing standards are not consistent. Some organizations like the International Organization for Standardization with their ISO 90003 standard, or the Software Engineering Institute with their Capability Maturity Model Integration measure the maturity of the software creation process of an organization but do not determine the testing done on a application basis. Others like the Institute of Electrical and Electronics Engineers (IEEE 1044-1993) or the British Standard (BS 7925-2:1998), have direct standards for unit level testing only but still do not determine the testing actually performed on a application basis. Still other organizations like British Standards only govern testing terminology, per BS 7925-1.
Unfortunately, these existing standards do not provide a clear, concise indication to the public of exactly how much testing has been performed on a particular application. Thus, the public is left with only anecdotal measures of software quality.
A system and method are disclosed for generating a specific level of software testing of algorithms and applications. A test plan, including input parameter values, expected output parameter values, and dataset size, is entered. The test plan is then executed, and results of the test are scored in accordance with predetermined software testing level definitions, yielding one of four possible testing levels achieved by the tested software.
System 100 is coupled to a host management system 145, which provides management of system functions, and issues system requests. Algorithm execution module 125 initiates execution of kernels invoked by algorithms that are executed. Management system 145 can be any external client computer system which requests services provided by the present system 100. These services include the presently-described automatic software test level generation.
The software for system services that are indicated below as being initiated by various corresponding menus and ‘buttons’ may be stored in data and program storage area 190.
Software Testing Levels
There are four software testing levels indicated by the present method—level 0, level 1, level 2, and level 3. These levels are described below.
Level 0 Testing
A level 0 test indication means that there is not enough information available to the system to perform a complete test. Typically this means that there is at least one code block or segment within an algorithm or application that has no unit level testing performed. According to McCabe, et al., the minimum testing which should be done to any multipath software code is at least one test per linear software block. Level 0 testing means that the McCabe minimum testing criteria is not provably met.
Level 1 Testing
A level 1 test indication means that the McCabe minimum testing criteria is met. The McCabe minimum testing criteria only insures that if a code block is entered with some input value and the expected output value is obtained then the test passes. The McCabe minimum test criteria does not insure that all boundary conditions of each transformed output variable for each code block have been tested. Level 1 test means that both the McCabe minimum testing criteria is met and the minimum, median, and maximum values for each transformed output data parameter have been individually tested as well.
Level 2 Testing
A level 2 test indication means that the Howard Minimum testing criteria has been met. Level 0 and level 1 testing does not insure that every code block entry method has been tested. Since a code block can be called in multiple ways, from different “other” code blocks, and since the overall system's conditions may effect the performance of a code block, the present ‘Howard Minimum’ test requires that each code block entry method have Level 0 and Level 1 testing performed. This is equivalent to testing a state at each state transition in a finite state machine, rather than only testing the states themselves.
Level 3 Testing
A level-3 test indication means that not only has every code block had every boundary condition of every code block tested, but that the sequencing (transitioning) between the code blocks is also correct. In level-3 testing, the system checks manually-entered expected transitions against the ones found when the code is tested.
Generating Level 0-Level 2 Tests
In an exemplary embodiment, level 0 through 2 tests are generated by first accessing the desired project followed a right mouse click on a specific process which displays an options list. Selecting a test procedure button then causes a test procedure pop-up display to be shown. As shown in
If all test processes (code blocks) pass when executed and there is at least one test procedure for every process, but there is at least one process does not have a high, medium, and low data set for the process then that project is at Test Level 1. If a project has high, medium, and low input parameters for all state transitions on every process then that project is at Test Level 2.
Generating a Level 3 Test
In an exemplary embodiment, selecting an appropriate button causes a test plan screen to be displayed. An MPT kernel is equivalent to a McCabe cyclomatic complexity basic code block, which is termed herein a ‘McCabe code block’. An MPT algorithm is comprised of MPT kernels and/or other MPT algorithms.
As shown in
In order to view, create, or edit an actual test, either the View Test or the Create New Test button on screen 300 is selected. Selection of either button will cause test plan screen 400 to be displayed.
The Index # column contains a list of all possible test steps for one of three data set groupings: small, medium, or large. These groupings allow for one, two, or three dataset sizes to be selected to be tested, at step 220. If only one or two test dataset sizes are selected then, at most Test Level 2 can be obtained. If all three dataset sizes are tested then, up to Test Level 3 can be obtained. A developer can opt for only level 0 through level 2 testing by not entering a test plan.
It should be noted that the present test procedures are associated with design, as opposed to code. Only after the code has been attached to the design is it possible to execute the test procedures. In
Selection of a “submit” button 401 causes the test plan to be executed on the selected code segment (kernel or algorithm), at step 225. There are two possible types of resultant errors: procedure and transition.
At step 230, scoring application 191 ‘scores’ the results of test plan execution in accordance with the definitions of the software testing levels described above, and indicates the actual test level 240 achieved by the tested code.
Having described the invention in detail and by reference to specific embodiments thereof, it will be apparent that modifications and variations are possible without departing from the scope of the invention defined in the appended claims. More specifically, it is contemplated that the present system is not limited to the specifically-disclosed aspects thereof. It should be noted that the present system is not limited to the specific file names, formats and instructions presented herein. The methods described herein may be executed via system 100 or other systems compatible therewith.
This application claims priority to U.S. Patent Application Ser. No. 61/656,835, filed Jun. 7, 2012, the disclosure of which is incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
20030014734 | Hartman et al. | Jan 2003 | A1 |
20050160321 | Cleaveland et al. | Jul 2005 | A1 |
20050251432 | Barker et al. | Nov 2005 | A1 |
20060010428 | Rushby et al. | Jan 2006 | A1 |
20070079189 | Jibbe | Apr 2007 | A1 |
20080209567 | Lockhart et al. | Aug 2008 | A1 |
20090199047 | Vaitheeswaran et al. | Aug 2009 | A1 |
20110088014 | Becker et al. | Apr 2011 | A1 |
20120060144 | Novak et al. | Mar 2012 | A1 |
20130339930 | Xu | Dec 2013 | A1 |
20140181793 | Kaliappan | Jun 2014 | A1 |
Entry |
---|
PCT Application PCT/US2013/044811 International Search Report and Written Opinion dated Sep. 13, 2013, 9 pages. |
Number | Date | Country | |
---|---|---|---|
20130332777 A1 | Dec 2013 | US |
Number | Date | Country | |
---|---|---|---|
61656835 | Jun 2012 | US |