Patent Grant
8566805
The present invention is related to copending application Ser. Nos. 12/558,327 and 12/558,274, the contents of which are incorporated herein by reference in their entireties.
The present invention generally relates to a method and system of calibrating estimates for software projects, and more particularly, to a method and system to provide continuous calibration estimation and improvement options across a software integration life cycle.
While software systems continue to grow in size and complexity, business demands continue to require shorter development cycles. This has led software developers to compromise on functionality, time to market, and quality of software products, for example. Furthermore, the increased schedule pressures and limited availability of resources and skilled labor can lead to problems such as inefficient testing, high development and maintenance costs, and the like. This may lead to poor customer satisfaction and a loss of market share for software developers.
To improve product quality many organizations devote an increasing share of their resources to testing and identifying problem areas related to software and the process of software development. Accordingly, it is not unusual to include a quality assurance team in software development projects to identify defects in the software product during and after development of a software product. By identifying and resolving defects before marketing the product to customers, software developers can assure customers of the reliability of their products, and reduce the occurrence of post-sale software fixes such as patches and upgrades which may frustrate their customers.
Testing and identifying problem areas related to software development may occur at different points or stages in a software development lifecycle. For example, a general software development lifecycle includes a high level requirements/design review, a detailed requirements/design review, code inspection, unit test, system test, system integration test, performance test and user acceptance test. However, as the software development lifecycle proceeds from high level requirements/design review to user acceptance test, costs for detecting and remedying software defects generally increases, e.g., exponentially. As such, software developers seek to detect and remedy software defects as early in the software development lifecycle in an effort to avoid the increased costs of detecting and remedying these software defects later in the software development lifecycle.
Complex system integration development is very expensive and considered high risk, as the majority of defects are found later in the life cycle due to the methodologies used to test projects. For example, test projects are planned inefficiently/ineffectively because there is no solution that provides real time insight necessary to find and fix defects as early as possible. For this reason, consistently accurate defect projection modeling in complex system integration testing is considered impossible absent significant project history. That is, conventionally, historical defect data for a project (e.g., a software code project) must be available in order to accurately project future defect patterns and trends for the project. This historic defect data may aid an organization in detecting software defects, in order to project future defect patterns and trends for the project.
It is known, though, that at the earliest stages of the software development lifecycle (when defects are less costly to identify and remedy), historical defect data for the project may not yet be available. That is, at the earliest stages of the software development lifecycle, no testing has yet occurred to provide the necessary historical defect data for the project. Thus, although most complex system integration project owners would like to accurately project defect rates to effectively plan release development, test, and launch activities and costs, this is often not possible since there is little or no applicable project history available, e.g., earlier in the design project.
Also, differences between historic project information and a current project frequently mean the projections using historical data will not be very accurate when compared against the actual current project results. Furthermore, the approach of using historical projects for estimation provides no guidance or useful insight into how to best adjust strategic development plans while the project is underway to reflect changed conditions. As a result, detailed estimation planning is rarely performed on many of the large and complex efforts that would most benefit from such planning. And, even when estimation planning is used, such planning remains static throughout the project, even though defects in rates other than those planned for may have been discovered during each phase of the project, and each deviation from defect expectations directly effects the defect expectations in each subsequent phase of the project, including production.
Accordingly, there exists a need in the art to overcome the deficiencies and limitations described hereinabove.
In a first aspect of the invention, a method is implemented in a computer infrastructure having computer executable code tangibly embodied on a computer readable storage medium having programming instructions. The programming instructions are operable: compare actual data associated with a phase of a project to expected results used to develop a strategic development plan for the project; and calibrate efforts in the strategic development plan for future activities of the project based on the actual data.
In another aspect of the invention, a system comprises a calibration tool operable to calibrate a strategic development plan originally based on expected results obtained from a user or expert system. The calibration tool calibrates efforts noted in the strategic development plan for future activities in a project using actual data obtained from a current phase of the project.
In an additional aspect of the invention, a computer program product comprising a computer usable storage medium having readable program code embodied in the medium is provided. The computer program product includes at least one component operable to: receive expected results to develop a strategic development plan for a software integration project; compare actual test data associated with a phase of the software integration project to the expected results; and dynamically calibrate efforts for future activities based on the actual data when the actual data does not match the expected test results.
In a further aspect of the invention, a computer system for calibrating efforts of a strategic plan, the system comprises a CPU, a computer readable memory and a computer readable storage media. Additionally, the system comprises: first program instructions to receive expected results to develop a strategic development plan for a software integration project; second program instructions to compare actual test data associated with a phase of the software integration project to the expected results; and third program instructions to dynamically calibrate efforts for future activities based on the actual data when the actual data does not match the expected test results. The first, second and third program instructions are stored on the computer readable storage media for execution by the CPU via the computer readable memory.
The present invention is described in the detailed description which follows, in reference to the noted plurality of drawings by way of non-limiting examples of exemplary embodiments of the present invention.
The present invention generally relates to a method and system of calibrating estimates for software projects, and more particularly, to a method and system to provide continuous calibration estimation and improvement options across a software integration life cycle. In embodiments, the present invention provides a method and system to support ongoing, dynamic calibration of a strategic development plan for a complex integration project based on actual data received from a current phase of a project (prior to completion of the entire software integration project). More specifically, the calibration is based on actual data obtained from a test of a completed phase of a project. This information can then be used to adjust an original strategic development plan for future phases in the software integration project. For example, testing procedures, team make-up and other resources can be reallocated or calibrated based on the information gleaned from actual data of a completed phase of the project. This calibration, in turn, will be used to optimize future phases of the software integration project.
As an example, in embodiments, the present invention comprises performing an initial estimation of the entire software integration project based on some general information the user (or expert system) provides to the system of the present invention. The general information may include, for example, complexity of the project, personnel decisions, initial distributions, and other resource allocations, as well as the maturity of the organization. Maturity of the organization refers to the capabilities of an organization in performing complex integration projects. For example, a mature organization will have extensive experience and successful outcomes in organizing, performing and completing complex integration projects. Initial estimates may include, for example, expected projected defects based on an organization's maturity level, projected time schedules and personnel decisions, as well as the allocation of other resources, based on each phase of a project.
Once a particular phase of the project is completed, the actual data obtained from testing such phase is acquired and, using this information, the system and method (e.g., model) updates the overall estimate/projection (originally developed plan for the project). If the actual result based on the estimation does not meet expectations, e.g., cost estimates, resource allocation, time schedules, etc., the model will provide suggestions on how the strategic development plan can be adjusted to satisfy the expected quality goal, schedule goal, etc. This can be repeated until the project is completed such that ongoing calibration and improvement options can be provided to the user, e.g., software developer.
Illustratively, upon completion of a “system test” phase, the actual data of the “system test” phase is captured. A calibration tool of the present invention will then use this information to adjust the future phases of the project based on the current actual data, whereas an optimization tool of the present invention will adjust the original plan to achieve the desired quality goal, schedule goal, etc. As a result, the strategic development plan for subsequent activities after “system test” is adjusted so that it now better reflects what will realistically occur on the project. Advantageously, having this information in real time provides essential insight to project stakeholders so that they can make meaningful and practical adjustments as needed while the project is still underway. Without such insight, projects are left to wait until the project has completed, at which time it would be possible to derive various “lessons learned” that can only then be applied to future projects, resulting in a far slower quality improvement cycle.
As will be appreciated by one skilled in the art, the present invention may be embodied as a system, method or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.
Any combination of one or more computer usable or computer readable medium(s) may be utilized. The computer-usable or computer-readable medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following:
The computer-usable or computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
In the context of this document, a computer-usable or computer-readable medium may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-usable medium may include a propagated data signal with the computer-usable program code embodied therewith, either in baseband or as part of a carrier wave. The computer usable program code may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc.
Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network. This may include, for example, a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
The computing device 14 includes a processor 20, memory 22A, an I/O interface 24, and a bus 26. The memory 22A can include local memory employed during actual execution of program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution. The bus 26 provides a communications link between each of the components in the computing device 14. In addition, the computing device includes random access memory (RAM), a read-only memory (ROM), and an operating system (O/S). In general, the processor 20 executes computer program code (e.g., program control 44), which can be stored in the memory 22A and/or storage system 22B. While executing the computer program code, the processor 20 can read and/or write data to/from memory 22A, storage system 22B, and/or I/O interface 24.
The computing device 14 is in communication with the external I/O device/resource 28 and the storage system 22B. For example, the I/O device 28 can comprise any device that enables an individual to interact with the computing device 14 or any device that enables the computing device 14 to communicate with one or more other computing devices using any type of communications link. The external I/O device/resource 28 may be for example, a handheld device, PDA, handset, keyboard, etc.
The computing device 14 further includes an estimation tool 25, a calibration tool 30, optimization tool 35, and report generation tool 40. The estimation tool 25, calibration tool 30, optimization tool 35, and report generation tool 40 can be implemented as one or more program code in the program control 44 stored in the memory 22A as separate or combined modules. The program control 44 controls and/or executes the processes of the invention. In embodiments, calibration, estimation and optimization profiles may be stored in the storage system 22B, which may be, for example, a database. For example, rules used to determine calibration and optimization results, as discussed further below, may be stored in the storage system 22B.
In embodiments, the estimation tool 25 estimates the test effect according to the test plan (test effort and the test effort distribution), test organization and the system under testing, where the test plan and test effect has data for each project phase. The estimation tool 25, in embodiments, can also estimate the effects of a calibration and/or optimization performed by the calibration tool 30 and/or optimization tool 35. This can be performed by computer simulations discussed below. In this way, if the calibration and/or optimization do not meet the stated goals/objectives of the developer, the strategic development plan will not be updated and further iterations to the strategic development plan can be performed by the calibration tool 30 and/or optimization tool 35.
The calibration tool 30 is operable to calibrate the strategic development plan (that was originally developed from, for example, the estimated data generated from the estimation tool) using actual data obtained from a current phase of a project (e.g., obtained from a test). For example, using actual data obtained from a test in a software integration project, the calibration tool 30 will adjust the efforts (e.g., resources, time schedules, etc.) for future activities within the software integration project, e.g., adjust the estimated effort (add or reduce effort) for each trigger in each activity. That is, the calibration tool 30 can, for example, provide a percentage change to the estimated effort needed for a particular phase in the project in order to show the developer how to best adjust resources for future activities in the integration project (e.g., add or reduce effort for particular stages in a project). In embodiments, a “trigger” indicates where (and possibly, how) a defect was discovered (e.g., the circumstances surrounding the defect discovery), which is further defined in application Ser. No. 12/558,274.
The optimization tool 30 is operable to adjust the original strategic development plan to achieve a desired quality goal, schedule goal, etc. based on the calibration results. As a result, the original strategic development plan for subsequent activities can be adjusted to better reflect what will realistically occur on the project based on past performance data of the complex integration project. The calibration, optimization and estimation can be shown graphically or otherwise using the report generation tool 40. The report generation tool 40 can also be used to show any of the interfaces described herein.
In one example, testing procedures, team make-up and other resources can be reallocated or calibrated based on the information obtained from actual data gleaned from a completed phase of the project. Thus, advantageously, the method and system of the present invention leverages the actual data as the project is progressing, and is operable to translate that information into precise and meaningful adjustments to the original strategic development plan, in real time. This approach provides significantly greater control and ability to proactively manage risk than the alternative approach of relying on historic project information alone.
The computing device 14 can also comprise any general purpose computing article of manufacture capable of executing computer program code installed thereon (e.g., a personal computer, server, etc.). However, it is understood that the computing device 14 is only representative of various possible equivalent-computing devices that may perform the processes described herein. To this extent, in embodiments, the functionality provided by the computing device 14 can be implemented by a computing article of manufacture that includes any combination of general and/or specific purpose hardware and/or computer program code. In each embodiment, the program code and hardware can be created using standard programming and engineering techniques, respectively.
Similarly, the computing infrastructure 12 is only illustrative of various types of computer infrastructures for implementing the invention. For example, in embodiments, the server 12 comprises two or more computing devices (e.g., a server cluster) that communicate over any type of communications link, such as a network, a shared memory, or the like, to perform the process described herein. Further, while performing the processes described herein, one or more computing devices on the server 12 can communicate with one or more other computing devices external to the server 12 using any type of communications link. The communications link can comprise any combination of wired and/or wireless links; any combination of one or more types of networks (e.g., the Internet, a wide area network, a local area network, a virtual private network, etc.); and/or utilize any combination of transmission techniques and protocols.
In embodiments, a service provider, such as a Solution Integrator, could offer to perform the processes described herein. In this case, the service provider can create, maintain, deploy, support, etc., the computer infrastructure that performs the process steps of the invention for one or more customers. These customers may be, for example, any business that uses technology. In return, the service provider can receive payment from the customer(s) under a subscription and/or fee agreement and/or the service provider can receive payment from the sale of advertising content to one or more third parties.
Furthermore, the invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. The software and/or computer program product can be implemented in the environment of
In particular, at step 200, actual test results of the system test phase are obtained and compared to expected results. The expected test results can be based on user input using information such as, for example, a maturity level of the organization, expected resources to be used at different phases, resource allocations, expected defects, etc. If the actual test results are not the same as the expected test results, at step 205, the calibration tool will calibrate/change the documentation, e.g., overall strategic development plan, in real time, to adjust the overall strategic development plan based on the actual test results. For example, at step 210, the calibration tool provides a new/update worksheet (strategic development plan), e.g., adding additional resources, code alterations, etc. for downstream phases of the project such as, for example, SIT testing, performance testing or UA testing. The outputs from the calibration tool can be obtained from the database 22B using, for example, the actual test results, maturity levels of the organization, etc. In this way, the calibration tool will renew the overall strategic development plan and documentation.
At step 215, the optimization tool will apply the calibration to future phases/activities in order to optimize the overall strategic development plan. For example, when a defect pattern develops, the optimization tool can suggest mitigating circumstances in order to compensate for such defect patterns, etc. In embodiments, the optimization tool can obtain such mitigating circumstances from the database (storage 22B), which can store different mitigating circumstances matched to known defect patterns, maturity levels of the organization and actual test results, as compared to the expected test results. That is, using a rules based engine, the optimization tool can produce suggested changes to the strategic development plan.
At step 220, the optimization results and inputs from, e.g., a strategic development plan, are provided to an estimation module. The estimation module, in embodiments, will estimate the results that can be obtained by using the mitigating circumstances from the optimization tool. This can be performed by, for example, running simulations using the different mitigating circumstances in order to estimate test results of a next phase of the project, e.g., SIT testing, at step 225. If the result based on the estimation does not meet expectations, e.g., cost estimates, schedules, etc., the model will provide additional calibrations and suggestions on how the strategic development plan can be adjusted to satisfy the expected quality goal, schedule goal, etc., at step 205 This can be repeated until the project is completed and/or results are accepted to the developer.
In this way, the actual test results provided to the calibration tool can be used to dynamically change (calibrate) the overall strategic development plan, in real time. If the changes are not acceptable to the developer(s), though, the calibration tool can provide different calibrations to provide further calibrations and/or optimization results until an acceptable calibration is provided. This can be performed prior to changing the strategic development plan via the use of the estimation tool. As shown in
In particular, at step 300, the system of the present invention receives input from a developer. This information may include, for example, test efficiency, test effort, and effort distribution by activities, amongst other input. More specifically, TABLE 1 shows exemplary inputs and definitions of such inputs provided at step 300.
This input can be provided to the calibration tool and estimation tool. At step 305, the estimation tool estimates the test effect according to the strategic development plan (e.g., test effort and the test effort distribution), test organization and the system under testing. For example, the estimation tool can create an estimation based on the maturity of the organization and/or other user inputs which can include the inputs provided at step 305. The strategic development plan and test effect has data for each project phase. After one activity is completed, the actual data is input to the system.
At step 310, the calibration tool compares the actual data and estimated (or expected) data together to perform a reconciliation of the strategic development plan with a defined set of rules. That is, the calibration tool reconciles the deviation between the actual data and defect projection by activities in order to update the strategic development plan of the project. The rules can be defined by users (or by an expert system), where each rule can be composed of three parts: symptom, cause and treatment. For example, TABLE 2 shows an exemplary rule set that shows a calibration (treatment) for different symptoms. The rule set of TABLE 2, as well as other rule sets contemplated by the present invention can be stored in the database 22B of
After the calibration is performed, the adjusted inputs are recalculated to more accurately reflect the current situation based on the actual data. These calculated inputs may be, for example, test efficiency, SUT, test efforts, or effort distribution, amongst other inputs, each of which may have been changed based on the variables obtained by the actual results.
At step 315, the optimization tool can be leveraged to optimize the current strategic development plan according to the expected quality, schedule or other goal according to the results of the calibration tool and/or other inputs. The optimization tool can use the results of the calibration tool, in addition to goals set by the developer as well as other constraints such as, for example, scheduling, etc. to provide suggestions to the developer. The suggestions, as discussed herein, can be based on rules associated with causes and symptoms known to be associated with the actual test results. As shown in
In particular, the process begins at step 400. At this stage, information can be entered into the system in order to estimate defects. The inputs can be, for example, the following inputs shown in TABLE 3.
At step 405, the estimation tool estimates a total defect volume for a particular phase in the software integration project. That is, the estimation tool estimates the expected defects for a particular phase in the software integration project. These estimates may be, for example, best guess estimates based on the data provided, for example, in TABLE 3, as well as other data such as, for example, historical data. The estimation tool can also use an already existing project or a profile which characterizes the potential defect volume and distribution information as a reference.
In further embodiments, the outputs from the calibration tool and/or optimization tool can be provided to the estimation tool prior to the development of a new strategic development plan. In this way, the estimation tool can simulate different scenarios for the developer to determine whether the stated goals and/or objectives are met. If the simulations are acceptable, the strategic development plan can be updated by the system and method of the present invention. If the simulations are not acceptable, additional inputs can be fed into the calibration tool until an acceptable solution is found.
In embodiments, there are two factors that can be used to determine the defect volume. For example, the first factor is the defect density and the second factor is the project size. In embodiments, the defect density is determined by the reference project and the ratio between the characteristics of the current project and those of the reference project. The characteristics should have reasonable coverage of relevant factors which could affect the defect density. In further embodiments, the potential defect volume is equal to the accumulation of defect density multiplied by project size.
At step 410, the program control will determine whether additional follow on activities are required, i.e., cycle through the estimation process for each activity such as, for example, design reviews, code inspections, system tests, etc., depending on what activities are to be included in the analysis. Once the estimation has been performed for all of the activities, the process will then perform the calibration. More specifically, the program control will determine whether actual defects have been received from the activity in order to calculate or measure a defect deficiency in the calibration tool. If no activity is required, the process ends at step 415.
If additional activities are required, the process continues to step 420. At step 420, the discovered defects are estimated according to the devoted effort, e.g., resources allocated to the particular phase of the integration project. This estimate can be calculated, for example, on additional information provided to the developer such as, for example, alterations to the code, historical defects or other triggers (e.g., known defects). The residual defects may be the defects that end up in production of the integrated software project, and which will not significantly affect the end product.
As further shown in
DisDefUpp is the upper limit of the percentage of the discovered defects, totalDefs is the total defects that can be discovered by the current trigger, and DDE is the defect discovery efficiency.
More specifically,
The user interface of
Additionally, the test cases 635 and executed test cases 640 will be used to populate the status field 605. For example, the total number of executed test cases completed with relation to the total test cases is shown as a percentage in the status field 605. The type field 645 will show the type of defect found and possible types of correction. For example, an initialization/assignment type (e.g., if/then statement) will signify a minor defect such as, for example, an initialization line has a typographical error. In embodiments, the types of defects can be listed in order of complexity; although, other formats are also contemplated by the present invention.
In embodiments, the actual data can be listed as separated catalogues. The catalogues can be, for example, the actual data and the calibration results. For example, the first catalogue is directed to inputs and the second catalogue is directed to outputs. For the first catalogue, the calibration tool uses the actual data input by the user (or the actual data transferred from another tool) to replace the original input. For the second catalogue, the calibration tool uses a set of rules to adjust the inputs. These rules can be, for example, stored in the database of
The output of the calibration tool may be referred to as symptom, cause, and treatment, as discussed above. For example, the “symptom” is a statement to specify a deviation. The “cause” defines which inputs do not match with the actual results data and the “treatment” defines how to adjust the value of the input to perform the calibration. In this way, the calibration tool can be used as a rule based reconciliation mechanism to dynamically calibrate the strategic development plan. An example rule is shown in TABLE 4, below.
In the interface shown in box 805, suggestions can be provided to add or reduce effort for each trigger in each activity. The box 800 can be formatted to include a plurality of activities and triggers, where an intersection (cell) of the activity and trigger may be represented by a percent change as calibrated by the calibration tool. The cells can be manually adjusted in order to meet certain goals. For example, the SIT/Coverage intersection cell can be adjusted from 50% to 40%. By making this change, any of the other triggers associated with the SIT activity can be readjusted so that the total percentage remains 100%. In this way, it is possible to shift efforts to an early stage of the integration project in order to remove defects at an earlier point in the project life cycle. For example, if a change in one trigger is made, then one or more adjustments in other applicable triggers for that activity are adjusted such that the total of all triggers in that activity equals 100%. If the user makes a change, though, the system and method suggests distributions based on maturity or project profile information, or due to some calculation performed, such as any reconciliation or calibration invoked by the user, for example.
In one example, in the situations where total effort (total Person Days (PD) for instance) can be adjusted, the optimization tool makes recommendations on adjusting effort for each activity in order to meet the user (or expert system) defined target for the percentage of defects that will escape to production. On the other hand, if the total effort is fixed, then the optimization component (tool) tries to find the most efficient allocation of effort among the activities so that the total discovered defects prior to moving to production is maximized, leading to the lowest possible defect escape rate in production. Thus, the optimization tool adopted in both cases is dynamically programmed, which is an appropriate choice to shift or adjust efforts for individual trigger and activity. In mathematics and computer science, dynamic programming is a method of solving problems that exhibit the properties of overlapping sub problems and optimal substructure. That is, dynamic programming is an optimization technique that transforms a maximization (or minimization) problem involving “n” decision variables into n problems having only one decision variable each.
The following are illustrative examples of implementations of the present invention using, for example, predefined rules stored in a database (e.g., storage system 22B of
If given the effort required, the problem is solved as follows:
1. Remove The Wasteful Effort. In this solution, although the effort is devoted, there are no discovered defects according to the simplified relationship between effort and discovered defects.
2. Add/Reduce Effort To Make The Total Effort Reach The Expected Value. In this solution, a dynamic programming algorithm is applied to add/reduce the effort to/from the cells. Each cell represents a trigger in an activity.
3. Shift Effort. In this solution, the effort will be shifted from a cell (trigger in one activity) with a low DDE to a cell with high DDE. The shifting can be based on a dynamical programming algorithm.
Remove Wasted Effort
For each cell, if the devoted effort is more than the effort that is required to discover the residual defects (totalDefs*DisDefUp<effort*DDE), there is wasted effort. And the wasted effort is equal to the required effort subtracted from the devoted effort (effort−(totalDefs*DisDefUp)/DDE).
Add Effort
The problem is specified as follows:
Reduce Effort
The problem is specified as follows:
Shift Effort
The problem is specified as follows:
If given an expected quality goal (such as y % defects escape to field), the problem is solved as follows:
1. Remove The Wasteful Effort. Although the effort is devoted, there are no discovered defects according to our simplified relationship between effort and discovered defects.
2. Add/Reduce Discovered Defects To Make The Residual Defect Rate In Production Reach The Expected Value. In this solution, a dynamic programming algorithm is applied to add/reduce the discovered defects to/from the cells.
3. Shift Defect Removal. In this solution, the discovered defects will be shifted from the cell (cell=one Trigger in one Activity) with low DDE to the cell with high DDE. The shifting algorithm is also based on a dynamic programming algorithm.
Remove Wasted Effort
For each cell, if the devoted effort is more than the effort that is required to discover the residue defects (totalDefs*DisDefUp<effort*DDE), there is wasted effort. And the wasted effort is equal to subtract the required effort from the devoted effort (effort−(totalDefs*DisDefUp)/DDE).
Add Discovered Defects
The problem is specified as follows:
Reduce Discovered Defects
The problem is specified as follows:
Shift Discovered Defects
The problem is specified as follows:
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims, if applicable, are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principals of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated. Accordingly, while the invention has been described in terms of embodiments, those of skill in the art will recognize that the invention can be practiced with modifications and in the spirit and scope of the appended claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5539652 | Tegethoff | Jul 1996 | A |
| 5905856 | Ottensooser | May 1999 | A |
| 6332211 | Pavela | Dec 2001 | B1 |
| 6442748 | Bowman-Amuah | Aug 2002 | B1 |
| 6456506 | Lin | Sep 2002 | B1 |
| 6477471 | Hedstrom et al. | Nov 2002 | B1 |
| 6519763 | Kaufer et al. | Feb 2003 | B1 |
| 6546506 | Lewis | Apr 2003 | B1 |
| 6601233 | Underwood | Jul 2003 | B1 |
| 6725399 | Bowman | Apr 2004 | B1 |
| 6889167 | Curry | May 2005 | B2 |
| 6901535 | Yamauchi et al. | May 2005 | B2 |
| 6988055 | Rhea et al. | Jan 2006 | B1 |
| 7200775 | Rhea et al. | Apr 2007 | B1 |
| 7231549 | Rhea et al. | Jun 2007 | B1 |
| 7334166 | Rhea et al. | Feb 2008 | B1 |
| 7451009 | Grubb et al. | Nov 2008 | B2 |
| 7630914 | Veeningen et al. | Dec 2009 | B2 |
| 7788647 | Martin et al. | Aug 2010 | B2 |
| 7809520 | Adachi | Oct 2010 | B2 |
| 7861226 | Episkopos et al. | Dec 2010 | B1 |
| 7886272 | Episkopos et al. | Feb 2011 | B1 |
| 7917897 | Bassin et al. | Mar 2011 | B2 |
| 7984304 | Waldspurger et al. | Jul 2011 | B1 |
| 8191044 | Berlik et al. | May 2012 | B1 |
| 20010052108 | Bowman-Amuah | Dec 2001 | A1 |
| 20020078401 | Fry | Jun 2002 | A1 |
| 20020188414 | Nulman | Dec 2002 | A1 |
| 20030018952 | Roetzheim | Jan 2003 | A1 |
| 20030033191 | Davies et al. | Feb 2003 | A1 |
| 20030058277 | Bowman-Amuah | Mar 2003 | A1 |
| 20030070157 | Adams et al. | Apr 2003 | A1 |
| 20030196190 | Ruffolo et al. | Oct 2003 | A1 |
| 20040205727 | Sit et al. | Oct 2004 | A1 |
| 20040267814 | Ludwig et al. | Dec 2004 | A1 |
| 20050071807 | Yanavi | Mar 2005 | A1 |
| 20050102654 | Henderson et al. | May 2005 | A1 |
| 20050114828 | Dietrich et al. | May 2005 | A1 |
| 20050144529 | Gotz et al. | Jun 2005 | A1 |
| 20050209866 | Veeningen et al. | Sep 2005 | A1 |
| 20050283751 | Bassin et al. | Dec 2005 | A1 |
| 20060047617 | Bacioiu et al. | Mar 2006 | A1 |
| 20060248504 | Hughes | Nov 2006 | A1 |
| 20060251073 | Lepel et al. | Nov 2006 | A1 |
| 20060265188 | French et al. | Nov 2006 | A1 |
| 20070100712 | Kilpatrick et al. | May 2007 | A1 |
| 20070112879 | Sengupta | May 2007 | A1 |
| 20070174023 | Bassin et al. | Jul 2007 | A1 |
| 20070234294 | Gooding | Oct 2007 | A1 |
| 20070283325 | Kumar | Dec 2007 | A1 |
| 20070283417 | Smolen et al. | Dec 2007 | A1 |
| 20070300204 | Andreev et al. | Dec 2007 | A1 |
| 20080010543 | Yamamoto et al. | Jan 2008 | A1 |
| 20080052707 | Wassel et al. | Feb 2008 | A1 |
| 20080072328 | Walia et al. | Mar 2008 | A1 |
| 20080092108 | Corral | Apr 2008 | A1 |
| 20080092120 | Udupa et al. | Apr 2008 | A1 |
| 20080104096 | Doval et al. | May 2008 | A1 |
| 20080162995 | Browne et al. | Jul 2008 | A1 |
| 20080178145 | Lindley | Jul 2008 | A1 |
| 20080201611 | Bassin et al. | Aug 2008 | A1 |
| 20080201612 | Bassin et al. | Aug 2008 | A1 |
| 20080255693 | Chaar et al. | Oct 2008 | A1 |
| 20090070734 | Dixon et al. | Mar 2009 | A1 |
| 20100005444 | McPeak | Jan 2010 | A1 |
| 20100145929 | Burger et al. | Jun 2010 | A1 |
| 20100211957 | Lotlikar et al. | Aug 2010 | A1 |
| 20100275263 | Bennett et al. | Oct 2010 | A1 |
| 20100332274 | Cox et al. | Dec 2010 | A1 |
| 20110296371 | Marella | Dec 2011 | A1 |
| 20120017195 | Kaulgud et al. | Jan 2012 | A1 |
| 20120053986 | Cardno et al. | Mar 2012 | A1 |
| Entry |
|---|
| Holden et al. “Improving Testing Efficiency using Cumulative Test Analysis”, 2006, IEEE, 5 pages. |
| Ian Holden, “Improving Testing Efficiency using Cumulative Test Analysis”, 2006, 25 slides, retrieved from http://www2006.taicpart.org/presentations/session5/3.pdf, pp. 1-25. |
| Ponaraseri et al., “Using the Planning Game for Test Case Prioritization”, 2008, Retrieved from http://selab.fbk.eu/tonella/papers/issre2008.pdf, 10 pages. |
| Paolo Tonella, “Publication List”, 2012, Retrieved from http://selab.fbk.eu/, 15 pages. |
| Office Action dated Nov. 5, 2012 in U.S. Appl. No. 12/558,274, 12 pages. |
| Office Action dated Nov. 8, 2012 in U.S. Appl. No. 12/558,260, 17 pages. |
| Office Action dated Dec. 20, 2012 in U.S. Appl. No. 12/558,147, 18 pages. |
| Office Action dated Nov. 8, 2012 in U.S. Appl. No. 13/595,148, 14 pages. |
| McGarry, J. et al., “Practical Software Measurement: A Guide to Objective Program Insight”, http://pdf.aminer.org/000/361/576/practical—software—measurement.pdf, Naval Undersea Warfare Center, Version 2.1, Part 1 to Part 4, 1996, 299 pages. |
| Jonsson, G., “A Case Study into the Effects of Software Process Improvement on Product Quality”, Jun. 2004, Master's Tesis in Computer Science—University of Iceland, 93 pages. |
| Office Action dated Oct. 11, 2012 in U.S. Appl. No. 12/558,327, 12 pages. |
| Notice of Allowance dated Aug. 31, 2012 in U.S. Appl. No. 12/558,375, 16 pages. |
| Hurlbut, “Managing Domain Architecture Evolution Through Adaptive Use Case and Business Rule Models”, 1997, pp. 1-42. |
| Final Office Action dated Nov. 23, 2012 in U.S. Appl. No. 12/557,886, 42 pages. |
| Boehm, B., “Software Engineering Economics”, IEEE Transactions on Software Engineering, vol. SE-19, No. 1, Jan. 1984, pp. 4-21. |
| Basili, V. et al., “Comparing the Effectiveness of Software Testing Strategies”, IEEE Transactions on Software Engineering, vol. SE-13, No. 12, Dec. 1987, pp. 1278-1296. |
| Office Action dated Jun. 14, 2012 in U.S. Appl. No. 12/557,886, 38 pages. |
| Ambler, S., “Choosing the Right Software Method for the Job”, http://web.archive.org/web/20090219074845/http://agiledata.org/essays/differentStrategies.html, retrieved Jun. 7, 2012, pp. 1-14. |
| Unknown, “ASTQB—ISTQB Software Testing Certification : ISTQB Syllabi”, http://web.archive.orb/web/20090423053623/http://www.astqb.org/educational-resources/syllabi-management5.php, retrieved Jun. 7, 2012, pp. 1-12. |
| Office Action dated Oct. 3, 2012 in U.S. Appl. No. 12/558,382, 11 pages. |
| Office Action dated Dec. 7, 2012 in U.S. Appl. No. 12/558,324, 15 pages. |
| Office Action dated Apr. 13, 2012 in U.S. Appl. No. 12/558,324, 10 pages. |
| Office Action dated Apr. 27, 2012 in U.S. Appl. No. 12/558,375, 10 pages. |
| Office Action dated Nov. 23, 2012 in U.S. Appl. No. 12/558,263, 36 pages. |
| Kwinkelenberg, R. et al., “Smartesting for Dummies”, Oct. 14, 2008, Wiley, 36 pages. |
| Lazic, L. et al., “Cost Effective Software Test Metrics”, WSEAS Transactions on Computers, Issue 6, vol. 7, Jun. 2008, pp. 559-619. |
| Hou, R. et al., Optimal Release Times for Software Systems with Scheduled Delivery Time Based on the HGDM, IEEE Transactions on Computers, vol. 46, No. 2, Feb. 1997, pp. 216-221. |
| Notice of Allowance dated Apr. 15, 2013 in related U.S. Appl. No. 12/558,274, 20 pages. |
| Final Office Action dated Apr. 3, 2013 in related U.S. Appl. No. 12/558,327, 11 pages. |
| Final Office Action dated May 13, 2013 in related U.S. Appl. No. 12/558,382, 12 pages. |
| Notice of Allowance dated Apr. 24, 2013 in related U.S. Appl. No. 12/558,260, 9 pages. |
| Final Office Action dated Mar. 29, 2013 in related U.S. Appl. No. 12/558,263, 54 pages. |
| Ulrich, “Test Case Dependency Processing in Robot Framework”, https://groups.google.com/forum/?fromgroups#!topic/robotframework-users/twcycBNLXI4, Google, Feb. 16, 2009, pp. 1-4. |
| Notice of Allowance dated Apr. 2, 2013 in related U.S Appl. No. 12/558,147, 10 pages. |
| Final Office Action dated Jun. 13, 2013 in related U.S. Appl. No. 13/595,148, 8 pages. |
| Chan et al., “A Tool to Support Perspective Based Approach to Software Code Inspection”, Proceedings of the 2005 Australian Software Engineering Conference, IEEE, 2005, 8 pages. |
| Notice of Allowance dated Jun. 24, 2013 in related U.S. Appl. No. 12/558,327, 6 pages. |
| Final Office Action dated Jul. 31, 2013 in related U.S. Appl. No. 12/558,382, 13 pages. |
| Final Office Action dated Jul. 18, 2013 in related U.S. Appl. No. 12/558,324, 15 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20110066887 A1 | Mar 2011 | US |
