Patent Application
20230259449
The present disclosure generally relates to testing of computer program code. Particular implementations provide automated or semi-automated evaluation of test code with respect to changes made to code tested by the test code.
Determining and correcting the source of software errors or performance issues, commonly referred to as debugging, remains a major problem in software development. Several studies have estimated that debugging consumes over fifty percent of software development time and costs. Although software configuration management systems and formalized debugging approaches can assist in debugging efforts, debugging can remain a tedious, time consuming task.
In order to try and reduce bugs in released software code, software development processes typically include the development of various tests for code under development. Once test code is defined, it can be periodically executed against code being developed, which can be referred to as target code, production code, or tested code.
In some cases, the volume, and potentially complexity, of test code exceeds, including potentially by orders of magnitude, that of the target code. When target code is changed, it can affect one or more tests written for that target code. Finding test code that is affected by a change to target code can be time consuming, as can updating the test code to account for the changes to the target code. This problem can be compounded when given production code is subjected to multiple tests. Accordingly, room for improvement exists.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
Techniques and solutions are described for automatically updating test code based on changes, including additions or deletions, to production code. Requests to analyze production code and a modification to production code are received. The modification is applied to the production code to provide modified production code. Test code specific for the production code is executed on the modified production code. It is determined that a test specified in the test code was failed by the modified production code. A condition responsible for test failure is identified, where the condition specifies at least one expected value or at least one expected result. A modification to update the test code is determined, where the modification includes an actual value or an actual result obtained during execution of the failed test. The modification to the test code is applied to provide modified test code.
The present disclosure also includes computing systems and tangible, non-transitory computer readable storage media configured to carry out, or including instructions for carrying out, an above-described method. As described herein, a variety of other features and advantages can be incorporated into the technologies as desired.
Determining and correcting the source of software errors or performance issues, commonly referred to as debugging, remains a major problem in software development. Several studies have estimated that debugging consumes over fifty percent of software development time and costs. Although software configuration management systems and formalized debugging approaches can assist in debugging efforts, debugging can remain a tedious, time consuming task.
In order to try and reduce bugs in released software code, software development processes typically include the development of various tests for code under development. Once test code is defined, it can be periodically executed against code being developed, which can be referred to as target code, production code, or tested code.
In some cases, the volume, and potentially complexity, of test code exceeds, including potentially by orders of magnitude, that of the target code. In one particular software design paradigm, tests for target code are written before the actual target code itself. In at least some cases, particular test code is written to test a particular aspect or use case of the test code. For instance, a method or function may be called by multiple different processes or use scenarios of a software application. The input, processing of the input, or output can change depending on the particular process or scenario. In order to isolate potential problems with the target code, different test code may be written for different processes or scenarios. Similarly, it is often desirable to write tests in a way such that a single aspect of the target code is being tested by each test, which can help isolate the source of a bug.
As an example, consider production code that performs processing on four input parameters and produces a single output parameter. At least four separate tests may be desirable—one for each of the input parameters, where one input parameter can be tested (e.g., by providing different values for the input parameter during the testing) and the output can be compared with an expected output. The other three input parameters can have fixed values, and thus the behavior of the production code, such as providing or deviating from the expected output, can be attributed to the input parameter under test.
Multiple tests for the same input parameter may be desirable, such as having different tests for different application processes or use scenarios. Input parameters may influence one another, and thus it may be desirable to have tests where two input parameters are allowed to vary (e.g., by providing different sets of input for the two input parameters), and the other two parameters are kept constant (or otherwise provided with values that provide a desired or expected behavior). It can be seen that even for relatively simple production code the length and complexity of test code can be much greater than the production code being tested.
When target code is changed, it can affect one or more tests written for that target code. Finding test code that is affected by a change to target code can be time consuming, as can updating the test code to account for the changes to the target code. This problem can be compounded when given target code is subjected to multiple tests.
As an example, consider production code that generates possible messages, which can be messages displayed to a user, error codes, etc. Test code may be written that checks to make sure one or more messages are produced by the testing, or that a message produced in testing is in a list of expected messages. The production code is modified in a way that a new message is produced—a message that was not included in the test code. Several issues can arise. One issue is that, if the test code is not modified, the tests may fail even though the production code is operating as intended. However, it can be difficult to identify in advance (before modifying the target code) the test code that needs to be adjusted—particularly if a software developer is not familiar with/did not write the test code or the test code is not well documented. Even once the relevant test code is identified, the test code must typically be manually updated by the developer.
This scenario illustrates the earlier observation of how a large percentage of software development cost and effort is related to testing/bug correction. However, such testing may be very important, as releasing software that does not operate correctly can have serious consequences—consider software that is used in areas such as transportation, medical care, or banking. Accordingly, room for improvement exists.
As used herein, a bug, also referred to as a coding error or software defect or fault, refers to code behavior that produces unintended or undesirable behavior. Bugs can result from failure to follow the correct syntax of a programming language. Bugs can also result from improper code semantics or code design, such as mathematical operator errors (e.g., division by zero), variables that may go out of bounds (e.g., a variable that causes an array to go out of bounds), memory leaks, security flaws, or bugs that result from improper interaction of software modules or components. Categories of bugs can include arithmetic errors, logic errors, syntax errors, resource errors, threading errors, interface errors, or errors resulting from the interaction of different programmers. Bugs can be associated with these and other types. In some cases, error messages or codes, including error messages from exception handling routines, can provide an indication of a bug, including its type and possible information as to its cause.
Typically, software developers test code at various stages of development. Testing code can involve more than simply executing code and looking for errors. For example, in many cases, software development occurs in a componentized or modular fashion. Different developers, or development teams, may be working on different parts of a larger program, or various components may be in different stages of development. If the components interact, it can be difficult to test a component whose operation depends on one or more other components that are not available, not fully developed, or are not suitably tested and debugged (which may, for example, make it more difficult to determine which component contains a bug).
One way to deal with the issue of incomplete or unavailable components, or to isolate tests to a single functional software code unit, is to hard code values (e.g., input values, which can be actual parameters or arguments) that may be externally supplied during “normal” software execution, such as values provided by user input, values from another data source (e.g., a database system that may not be available during testing), or values that may be returned by other components (e.g., return values from calling a function or method, or provided in response to an API call or similar mechanism, which values can be associated with an interdependent code unit—hard coding isolates the software unit being tested from another software unit that, during runtime, would supply input values). In some cases, rather than hard coding values, a developer may wish to replace or disable certain code sections that might be desired in the final version of the code, but which should not be run during testing.
When debugging, it can sometimes be difficult to determine if code is functioning properly. Even if an error is detected, it can be difficult to determine the source of an error. In some cases, it can be useful to output information, such as variable values, during program execution that is different than (or in addition to) any output that might be provided during normal program execution. A developer can manually cause such values to be checked or displayed (e.g., using cout statements in C++). Again, however, this process can be time consuming and can result in code that is more cluttered, complex, and difficult to read. When checks are performed, such as on variables, the nature of the check can vary over time, which may cause the check to be rewritten or edited, further increasing development time and costs.
The present disclosure provides techniques that assist in updating test code based on changes to target code. In one aspect, the present disclosure provides techniques that identify test code that may need to be updated based on a change to target code. For example, test code can be executed on changed target code and failed tests can be flagged for possible adjustment. In another aspect, the present disclosure provides techniques that can assist in updating test code that is affected by a change to target code. For example, a deviation from expected behavior can be used to suggest how test code may be altered to conform with a change to target code.
A developer may wish to avoid updating actual target code until test code has been adjusted to account for a planned change to the target code. In one aspect, the present disclosure provides for making a copy of at least a portion of target code that includes a modification to the target code. Test code and test code modifications can be executed against the target code copy—simulating the effect of a change to test/target code. In some cases, changes to test code can also be made to a copy of the test code. Once the operation of test/target code has been confirmed, such as by a developer, the changes to the target code/changes to test code can be made to the original target/test code.
In some cases, suggested changes to test code can be automatically identified and applied. However, typically a developer will wish to manually confirm appropriate behavior of both modified target code and modified test code before committing the changes. Thus, disclosed embodiments can provide a developer with a summary of suggested changes to be made to test code to accommodate a change to target code. The developer can choose to have some or all of the changes applied to the test code automatically (or the developer can manually make the changes, in another implementation). Optionally, the developer can opt to re-execute the altered test code to confirm the test code executes as desired before committing the changes to the test code or to the target code.
In the example above of production software without four input variables, using disclosed technologies, test code can be executed against the modified production code. A disclosed innovation, implemented in software, can identify one or more tests that failed. For example, the failed tests may indicate that a message is included in an output file that is not an expected output. Even this information can save a developer valuable time.
Software-implemented aspects of the present disclosure can then identify the source of the failure and suggest a modification to the test code to address this failure. In the example, the message that was not found in the list can be the source of the failure, and an adjustment to the test code can be to include the missing message in the list of expected messages. This change can be made automatically, or it can be suggested to a developer and the developer can choose to have the change made automatically, can make the change manually, or the developer can choose to reject the suggestion. Optionally, the test code can be re-executed to confirm that the test code now acts as expected. As additional changes are made to the production code, the developer can have some confidence that any new test failure is not due to prior code changes. Similarly, even when evaluating the original production code changes, the disclosed techniques help the developer ensure that any test code failures are due to actual errors/issues with the production code rather than resulting from test code that has not been updated to account for “legitimate” changes to production code behavior.
Examples 2-4 describes various ways in which target code can be tested, including having target and test code features implemented as classes. Although described techniques can be used with separate test code, they can also be used when test code is embedded within target code. For example, assertions can be included in test code that is applied to target code, or can be included in the target code itself, including when target code is rewritten or modified for testing purposes. Examples 5-8 provide details regarding disclosed techniques for evaluating test code.
The original source code segment 100 represents a portion of code for a method 110, such as a method to search for available economy class seats on a particular flight itinerary. The code for the method 110 includes a modifiable code segment in the form of a replaceable or substitutable code segment 112, which can also be referred to as a SEAM (such as in products available from SAP SE, of Walldorf, Germany). A replaceable code segment 112 can be indicated in various ways, such as by including a statement 114 indicating a starting position for the replaceable code segment, and a statement 116 indicating an end position for the replaceable code segment. The replaceable code segment 112 can be selectively executed, replaced with other code, or simply not executed (which can be equivalent to replacing the replaceable code segment with “empty” code, or commenting out or otherwise rendering unexecutable the replaceable code segment).
The test code segment 102 includes the statements 114 and 116 indicating the replaceable code segment 112, but the code of the replaceable code segment has been replaced by substitute code 120. Substitute code 120 can be, for example, code provided by a developer in order to generate the test code segment 102. The test code 102 can be automatically generated by replacing the replaceable code segment 112 with the substitute code 120.
Original code segment 100 includes code checkpoints 124, 126, 128, which can also be referred to as SPOTs. The code checkpoints 124, 126, 128 can be indicated by a particular keyword (e.g., “TEST-SPOT”) and can include one or more identifiers, such as a name of the checkpoint (e.g., “check_currency) and a variable (e.g., “ls_flight-CURRENCY”). Note that code is not otherwise associated with the checkpoints 124, 126, 128 in the original source code segment 100.
The test code segment 102 illustrates how code for checkpoints, such as checkpoints 124, 126, 128, can be added. A user interface, such as in the form of a wizard, can be provided to allow a user to select which checkpoints to use or activate, and to specify how values should be checked, such determining the status of a variable, including whether the variable has been assigned and its value relative to another value (e.g., a value supplied by the developer via the user interface). The test code 102 can be generated by automatically adding appropriate code proximate the checkpoint 124, 126, 128. For example, test code 102 shows checkpoint code 130 added after checkpoint 124, while checkpoints 126 and 128 have not been activated. Checkpoint code 130 indicates that a variable is to be checked as equal to a certain value, and an error message displayed if the variable is not equal to the value. In other cases, the test code can be manually generated in whole or part.
Both the original code segment 100 and the test code segment 102 include a modifiable code segment in the form of a redirectable method call 140, which can also be referred to as STUBs (including as implemented in products of SAP SE, of Walldorf, Germany) The redirectable method call 140 can be a method for which a substitute implementation can be created during testing, and the replaceable call can be substituted with a call to the substitute implementation. When the original code segment is deployed in a production environment, the redirectable method call 140 can be left unchanged and can call a production method.
The redirectable method call 140 can be indicated as such using a starting statement 142 and an ending statement 144. In at least some cases, the starting and ending statements 142, 144 are used to designate a redirectable method call 140. In some cases, having a commented redirectable method call in the original code segment would typically require the original code segment 100 to be changed prior to being executed in a production environment (e.g., the commenting would have to be removed to make the removable method call active). Thus, the use of the starting and ending segments 142, 144 can facilitate code testing by allowing redirectable method calls 140 to be located and handled without requiring the original code segment 100 to be modified prior to deployment.
The redirectable method call 140 can include an identifier 148, such as a name. The identifier 148 can be used to provide an option to a user to activate the redirectable method call 140 during testing, which can then be directed to call a non-production method, or not to activate the test method call, which can then be directed to call the production method during testing. A user can supply code for the non-production method. In the test code 102, the redirectable method call 140 has been changed to substitute method call 150.
In at least some aspects, the original code segment 100 and the test code segment 102 are separately maintained. That is, when testing is to be performed, the original code segment 100 is analyzed, such as to determine the presence, and, typically, identifiers, for the replaceable code segment 112, the code checkpoints 124, 126, 128, and the redirectable method call 140. For the replaceable code segment 112 and code checkpoints 124, 126, 128, if they are activated for testing, the replaceable code segment is replaced with substitute code 120, and suitable code, such as checkpoint code 130, is added for activated code checkpoints.
Thus, testing can produce the test code segment 102 such that the original code segment 100 need not be modified. Maintaining the test code segment 102 separately from the original code segment 100 can result in the original code segment being easier to read and maintain, as it does not include much of the test-specific code of the test code segment. Similarly, particularly when source code, or other human-readable code is shipped, the code can omit much of the test-specific code (or, in further cases, all test-specific code can be removed, such as by removing lines having starting and ending segments 142, 144). As the original code segment 100 typically has fewer lines than the test code segment 102, it can also be smaller (whether compiled or not) than the test code segment. In addition, having fewer lines of code, the production code can be more secure.
In some cases, testing can be facilitated by components of a software framework. For example, the ABAP programming language includes a library of test functionality, in the form of assertions that can be made for other software components. An example of this is the assertions class 222, which contains a number of assertion types 224. Assertions can be conditions that are evaluated with respect to other software, and which typically return a Boolean value depending on whether the assertion holds or not with respect to given software being evaluated. Evaluation of assertions can also return exceptions (or other types of error messages or error handling mechanism), rather than Boolean values. Or, a Boolean value (e.g., a value indicating that a desired result was not obtained) can be used to raise an exception (e.g., if return value==FALSE, then raise exception).
Assertions can be called by methods of other classes, such as by test methods 218a, 218b. At least for some types of assertions, a call to a member function of the assertions class 222 can include an actual value obtained by executing a test method 218a, 218b and an expected value. In the case of an “assert equals” assertion, a value of FALSE can be returned if the actual and expected values are not the same (which in turn can indicate that the test has failed). Various types of assertions can be analyzed using disclosed techniques, including for determining how test code might automatically be updated based on changes to target code including, without limitation:
It can be seen that simply calling the method 232 may be insufficient to determine that it is operating as intended. For example, assume that a value of “3” is provided by a test. The method 232 will return a value of “9,” which is not the intended result of “6.” While an analysis of the testing of the class 230 might indicate 100% code coverage, since all lines of code are the class 230 are executed by the test with the value of “3,” the test is not “good,” since it fails to determine that the result provided by the test is not the intended result.
The test class definition 240 defines a test method 242, m_add, for testing the method 232. The test class implementation 250 contains an implementation 252 of the test method 242. The implementation 252 declares an object 254 as an instance of the class 230, and a variable 256 to hold a result of a call to the method 232 using the object 254. At line 258, the variable 256 is assigned the value of calling the ADD method 232 with a value of “4.”
However, the test class implementation 250 includes an assertion 260 that calls an “assert_equals” method of a unit test class in the ABAP programming language. The assertion 260 specifies an expected value 262, the variable 256, and a message 264 to be displayed if the expected value does not equal the value of the variable 256 when the assertion is made. In this case, the test class implementation 250 would identify that the value returned by calling the method 232 with a value of “4,” is not the expected value of “8,” but is rather “16.” The test would thus fail, indicating to a developer that the method 232 may need correction.
Another type of parameter is an export parameter, which can be a parameter of a software component or a return value provided by the software component. Although an export parameter may assign a value to a variable, such as a variable passed by reference as a parameter, a software component does not use a current value of the export parameter in the scope of the calling code. Some parameters can have values that are both used and modified by a software component, and can be referred to a “changing” parameter or an input/output parameter. Changing parameters can be equivalent to passing a variable by value or reference to a software component without an indication that the variable cannot be modified. As used herein, “input value” refers to a value that is provided to a software component for manipulation, such as an input parameter or a changing parameter. An “output value” refers to a value that is changed by a software component and returned to a caller (or otherwise used in another software component), such as an output parameter or a changing parameter.
In one aspect, software testing, such as using assertions or equivalent functionality, is used to determine whether input or changing parameters have an expected effect on an export or changing parameter.
The definition 300 includes examples of input parameters 304 (shown as 304a-304c), a changing parameter 306, and export parameters 308 (shown as 308a, 308b). The parameters 304, 306, 308 are used with a method 312. Code 322 that is part of a first implementation 320 of the method 312 may represent a portion of a larger code section of the implementation 320. The code 322 includes the “location” export parameter 308a. The code 322 can be analyzed to determine what affects the final value assigned to the export parameter 308a. It can be seen that conditional operations 326 assign a value to “location” based on a value of the changing parameter 306, “cost.” In turn, it can be seen that the value of “cost” is determined using the “make” input parameter 304b and the “model” input parameter 304c. Thus, “cost,” “make,” and “model” can be said to influence the final value of “location.”
Code 322 also modifies a value (e.g., an input value) for the “cost” changing parameter 306. As described, cost 306 is modified by the input parameters 304b, 304c. So, input parameters 304b, 304c can be said to influence the final value of “cost.” In the code 322, the input parameter 304a, “color,” does not affect a value of “cost” or a value of “location.”
Code 342 is associated with a second implementation 340 of the method 312 and can be analyzed in a similar manner as the code 322. In the code 342, it can be seen that input parameter 304a and input parameter 304c influence the value of the export parameter 308b, “VehicleClass.”
At 404, a developer selects (or has selected for them) production code for modification. The production code can be a file, such as a file representing all or a portion of a software program. For instance, the file can be a header file or an implementation file for a class definition or other software object/abstract data type. Or, the file can be a main program or a file that represents a particular routine that can be called by a main program (such as a library file).
A copy of all or a portion of the production code is made at 408. The copy can be in the form of a file having a file name different that than the source file having the selected production code or can be maintained in memory. The developer makes changes to the copy of the production code at 412, or the changes can be made in an automated or semi-automated manner (e.g., the developer can specify a find/replace, a find/add, or a find/delete operation, or software corrections specific in an update file can be applied, including, without limitation, a “note” as implemented in software available from SAP SE, of Walldorf, Germany) In other implementations, step 408 is omitted, and changes are made to the original production code at 412, although such changes may be discarded if not specifically saved/committed by a user or process.
All or a portion of test code relevant to the modified production code is executed at 416. The test code can be in the form of unit tests, including as implemented in software available from SAP SE, of Walldorf, Germany Test code affected by the changes to production code is determined at 420. At least in some cases, the test code was known to execute successfully against the production code prior to the modifications made at 412. Thus, if any of the tests implemented in the test code fail when the test code is executed at 416, it can be assumed at 420 that the failure is due to one or both of errors in the changes to the production code or errors in the test code (where the error can be that the test code fails to account for changes to the production code). In some cases, the changes to the production can be simple enough such that it can be safely assumed that failed tests are due to the test code not being updated to account for changes to the production code. So, 420 can include identifying failed tests.
At 424, it is determined why tests of the test code failed. In some cases, this information can be obtained from a test itself. That is, the test code itself may have error logging/reporting functionality which can be leveraged by software carrying out the method 400. In other cases, software carrying out the method 400 can be programmed to identify one or more reasons for test failure. In either event, one way of determining a reason for test code failure can be that one or more executed results of test code were not satisfied by actual execution results. For example, particular test code may produce four expected outputs. If three actual outputs matched the expected outputs, but one actual output value did not match the corresponding expected output value for an output parameter, that output parameter can be identified as a reason for test failure. More particularly, it can be identified that the failure with respect to that output was due to the actual output value (say “X”) not matching the expected value (for example, “Y,” or of “X” not being in a list of acceptable output values).
Changes can be proposed to test code at 428, such as based on the reasons for test code failure identified at 424. For instance, in the example above, the proposed change can be to change the expected output to “X,” or adding “X” to a list of expected outputs. The changes can be accepted or rejected, such as by a developer, at 432. In some cases, if the changes are accepted, they are saved or committed at 436, where the saved changes are typically both for the production code and the test code, however optionally only one set of code changes is saved at 436. Saving the changes can include saving a temporary file using the name of an original file or replacing code in an original file with changed code.
After 436, the method 400 can end, or it can loop to 404 where different production code is selected for modification, which can be in the same code (e.g., file) as the previously changed code in some cases. For example, if multiple changes to a single set of production code are desired, the method 400 can be performed after less than all of the changes are made, which can help identify changes to test code that might be needed and help to isolate needed changes to test code as opposed to test code failures that might be due to erroneous production code changes or incompatibilities between production code changes. In other words, this incremental approach can help allow test code to be developed in a “normal” manner of helping to identify issues with production code. In other cases, the next production code can be different code than previously tested (e.g., code in a different file). In some cases, the code in a different file can optionally interact with code in a file with code previously analyzed using the method 400, such as where the newly selected production code calls, or is called by, the previously analyzed code, or where the files have a header/implementation relationship.
Returning to 432, if the changes are rejected, it can be determined at 440 whether the developer wishes to modify the production code or the test code. A rejection to production code may be made, for example, if a user determines that the reason for test code failure was not due to a problem with the test code, but rather with the production code. A rejection to test code changes may be made, for example, if it is determined that the test code change does not, or does not by itself, resolve failure of the relevant test. In some cases, an option can be provided to accept suggested changes to test code but to then, such as manually, make further changes to the test or production code.
If it is determined at 440 that changes to test code are being rejected, the method 400 can proceed to 444 where the test code is further adjusted or modified, which can be in place of or in addition to changes suggested at 428. The method 400 can then return to 416 where the modified test code is re-executed on the modified production code.
If it is determined at 440 that changes to the production code are being rejected, the method 400 can return to 412, where different or additional changes can be made to the production code by the developer. It could be that both the test code and the production code are to be further modified.
The test framework 508 includes a controller 512. The controller 512 can be responsible for carrying out, including coordinating, various operations regarding changes to production or test code, including the method 400 of
The controller 512 can communicate with a number of components, including a user interface 516 and an application program interface 520. The user interface 516 can allow a user to perform various actions, such as selecting production code to be modified, defining or modifying test code or production code, updating or evaluating test code based on changes to production code, receiving information regarding test results, or accepting or rejecting changes to test or production code. The application program interface 520 can facilitate the performance of similar processes as the user interface 516, but with respect to various applications (which can be part of the test framework 508 or external to the test framework), including in response to user input received using user interfaces associated with such applications.
The controller 512 communicates with (or in some cases includes) a test adjustment system 520. The test adjustment system 520 can include software components (which can be implemented, for example, as discrete operational units, such as classes, that are coordinated by a central program acting as the test adjustment system) that implement disclosed technologies, such as carrying out all or a portion of the method 400 of
The code change simulator 524 can make copies of code, such as production or test code. For example, in response to user input, such as through the user interface 516, the code change simulator 524 can make a copy of selected production code. The code change simulator 524 can also make copies of test code relevant to selected production code. In particular, the code change simulator 524 can make copies of test code identified as relevant to changed production code, such as making a copy of test code for tests which were identified as failing in response to changed production code, or test code updated to account for changes to the production code (such as to have the test code perform properly with the updated production code).
The test code execution component 528 can execute test code associated with the modified production code. The execution can be in response to user input received through the user interface 516. The user can, in some cases, select to execute test code once they have made desired changes to production code.
Test execution results are analyzed by the test result analyzer 532. The test result analyzer 532 can determine what tests failed, why the test failed, and in at least some cases can also suggest revisions to test code. If a user chooses to accept changes to test code, or manually makes changes, the changes can be implemented in the test code by the code adjuster 536. The code adjuster 536 can also commit changes made to the production code, such as in response to user input through the user interface 516 indicating that the user is satisfied with the changed production code, and typically also any revised test code.
In carrying out disclosed innovations, the test code adjustment system 520, or its components 524, 528, 532, 536, can read information from, and write information to, a code repository 550. The code repository 550 can include production code 554 and test code 558, which can be read by the code change simulator 524 and written to by the code adjuster 536, for example. In at least some cases, the code repository 550 can also store copies of production code 562 and test code 564, such as copies made by the code change simulator 524, or modifications made by the test result analyzer 532.
In any event, the test pseudocode 700 include input data 704 that will be provided to the production pseudocode 600 when the test pseudocode is executed, such as indicated at 708. 708 also indicates that the output of calling the production pseudocode 600 is a message 712. At 716, the test pseudocode 700 checks to see if the message 712 is in a list 720 of expected messages.
It can be seen from the production pseudocode 600 and the test pseudocode 700 that the message 628 added to the production pseudocode is not in the list 720. Thus, check 716 will evaluate to “FALSE,” indicating that the test has failed, even though the production pseudocode 600 does not contain an error (at least for the specific code being discussed). The failure of the test provided by the test pseudocode 700 can be used to identify that the test pseudocode may need to be updated in view of the changes to the production pseudocode 600, the reason for test failure (the error message 628 is not in the list 720), and a possible solution (adding the error message 628 to the list 720).
At 820, if/when the test code executes successfully, a copy of the production code to be modified is made. Optionally, relevant test code is also copied. Once a developer is finished entering changes, the test code is reexecuted at 824. At 828, a notification of failed tests is provided, and a test failure analysis and adjustment routine is called at 832. Note that in at least some cases it can be expected that test code will fail if changes are made to production code, as that can indicate robust test code. The pseudocode 800 can then loop, with adjustments being made to test code, automatically, manually, or the combination of the two, until the test code does not fail when executed at 824.
If the test code did not fail at 824, an indication of test success is provided at 836, along with an indication that a developer may wish to review the test code to confirm whether it is of sufficient quality/robustness.
Once the test code passes after adjustment, or if the test code did not initially fail at 824, the copied production code or test code can be copied over the original code, committing the changes.
In a similar manner, string manipulation functions can be used to replace the relevant portion of the test code, including changing the nature of the condition or changing an expected value of the condition. In particular,
In the particular example shown, it can be seen how the original test code 1030 did not include a warning for a “msgno” value of “015,” which is added by the modified test code 1032.
At 1110, a request to analyze production code is received. A modification to the production code is received at 1115. The modification is applied to the production code at 1120 to provide modified production code.
At 1125, test code specified for the production code is executed on the modified production code. It is determined at 1130 that a test specified in the test code failed. A condition in the production code responsible for test code failure is determined at 1135, where the condition specifies at least one expected value or at least one expected result.
At 1140, a modification is determined to the test code to update the condition such that the at least one expected value or the at least one expected result includes an actual value or an actual result obtained during execution of the test code. The modification is applied to the test code at 1145 to provide modified test code.
With reference to
A computing system 1200 may have additional features. For example, the computing system 1200 includes storage 1240, one or more input devices 1250, one or more output devices 1260, and one or more communication connections 1270. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system 1200. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing system 1200, and coordinates activities of the components of the computing system 1200.
The tangible storage 1240 may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium which can be used to store information in a non-transitory way, and which can be accessed within the computing system 1200. The storage 1240 stores instructions for the software 1280 implementing one or more innovations described herein.
The input device(s) 1250 may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computing system 1200. The output device(s) 1260 may be a display, printer, speaker, CD-writer, or another device that provides output from the computing system 1200.
The communication connection(s) 1270 enable communication over a communication medium to another computing entity. The communication medium conveys information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media can use an electrical, optical, RF, or other carrier.
The innovations can be described in the general context of computer-executable instructions, such as those included in program modules, being executed in a computing system on a target real or virtual processor. Generally, program modules or components include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. Computer-executable instructions for program modules may be executed within a local or distributed computing system.
The terms “system” and “device” are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on a type of computing system or computing device. In general, a computing system or computing device can be local or distributed, and can include any combination of special-purpose hardware and/or general-purpose hardware with software implementing the functionality described herein.
In various examples described herein, a module (e.g., component or engine) can be “coded” to perform certain operations or provide certain functionality, indicating that computer-executable instructions for the module can be executed to perform such operations, cause such operations to be performed, or to otherwise provide such functionality. Although functionality described with respect to a software component, module, or engine can be carried out as a discrete software unit (e.g., program, function, class method), it need not be implemented as a discrete unit. That is, the functionality can be incorporated into a larger or more general purpose program, such as one or more lines of code in a larger or general purpose program.
For the sake of presentation, the detailed description uses terms like “determine” and “use” to describe computer operations in a computing system. These terms are high-level abstractions for operations performed by a computer, and should not be confused with acts performed by a human being. The actual computer operations corresponding to these terms vary depending on implementation.
The cloud computing services 1310 are utilized by various types of computing devices (e.g., client computing devices), such as computing devices 1320, 1322, and 1324. For example, the computing devices (e.g., 1320, 1322, and 1324) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smart phones), or other types of computing devices. For example, the computing devices (e.g., 1320, 1322, and 1324) can utilize the cloud computing services 1310 to perform computing operators (e.g., data processing, data storage, and the like).
Although the operations of some of the disclosed methods are described in a particular, sequential order for convenient presentation, it should be understood that this manner of description encompasses rearrangement, unless a particular ordering is required by specific language set forth below. For example, operations described sequentially may in some cases be rearranged or performed concurrently. Moreover, for the sake of simplicity, the attached figures may not show the various ways in which the disclosed methods can be used in conjunction with other methods.
Any of the computer-executable instructions for implementing the disclosed techniques as well as any data created and used during implementation of the disclosed embodiments can be stored on one or more computer-readable storage media. The computer-executable instructions can be part of, for example, a dedicated software application or a software application that is accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software can be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or in a network environment (e.g., via the Internet, a wide-area network, a local-area network, a client-server network (such as a cloud computing network, or other such network) using one or more network computers.
For clarity, only certain selected aspects of the software-based implementations are described. It should be understood that the disclosed technology is not limited to any specific computer language or program. For instance, the disclosed technology can be implemented by software written in C++, Java, Perl, JavaScript, Python, Ruby, ABAP, SQL, Adobe Flash, or any other suitable programming language, or, in some examples, markup languages such as html or XML, or combinations of suitable programming languages and markup languages. Likewise, the disclosed technology is not limited to any particular computer or type of hardware.
Furthermore, any of the software-based embodiments (comprising, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) can be uploaded, downloaded, or remotely accessed through a suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, software applications, cable (including fiber optic cable), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communication means.
The disclosed methods, apparatus, and systems should not be construed as limiting in any way. Instead, the present disclosure is directed toward all novel and nonobvious features and aspects of the various disclosed embodiments, alone and in various combinations and sub combinations with one another. The disclosed methods, apparatus, and systems are not limited to any specific aspect or feature or combination thereof, nor do the disclosed embodiments require that any one or more specific advantages be present, or problems be solved.
The technologies from any example can be combined with the technologies described in any one or more of the other examples. In view of the many possible embodiments to which the principles of the disclosed technology may be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be taken as a limitation on the scope of the disclosed technology. Rather, the scope of the disclosed technology includes what is covered by the scope and spirit of the following claims.
