Patent Application
20090292956
Software developers create software using one or more software development programs. Throughout the software development process, software developers typically test their own code to ensure that it is operating as expected. Furthermore, one or more testers can conduct further testing to ensure that the software program overall operates as expected, and on the various platforms being supported.
For large software projects, there can be multiple software developers writing source code for a given software program. It can be difficult for a given software developer to know whether an error that is occurring during testing resulted from the code he wrote, the code he just changed, or from the code written by one or more of the other developers. As a result, a given software developer often spends a lot of time investigating errors that he did not even cause.
Various technologies and techniques are disclosed for using historical trends from prior tests to prioritize how failures are reported in later tests. After a user changes a software development project, one or more tests are run to detect failures during execution of the tests. Any detected failures are analyzed in comparison with historical failures for the software development project across tests run for multiple users. Any detected failures are categorized as new or old. New failures are reported with a different emphasis than old failures, such as with new failures being reported as a higher priority than old failures.
In one implementation, the newness of the error and whether or not it was introduced by the current user helps determine how the error gets reported. Current failure information is gathered from one or more automated tests of a software development application for a current user. Historical failure information is gathered from failures occurring in past automated test runs of the software development application for all users. Current failure information is compared to historical failure information to determine if any failures have occurred in past test runs. Any failures that have not occurred in past automated test runs are reported in a first priority grouping of failures. Any failures that have occurred in past automated test runs for the current user are reported in a second priority grouping of failures. Any failures that have occurred in past automated test runs for other users are reported in a third priority grouping of failures.
This Summary was provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
The technologies and techniques herein may be described in the general context as an application that prioritizes failures from tests based upon historical trends, but the technologies and techniques also serve other purposes in addition to these. In one implementation, one or more of the techniques described herein can be implemented as features within a software development program such as MICROSOFT® Visual Studio, or from any other type of program or service that performs testing and/or reporting of software test runs.
As noted in the background section, for large software projects, there can be multiple software developers writing source code for a given software program. It can be difficult for a given software developer to know whether an error that is occurring during testing resulted from the code he wrote, the code he just changed, or from the code written by one or more of the other developers. In one implementation, techniques are described for prioritizing failures that are reported to developers and/or other users from the result of testing. The failures are compared with historical failures and then prioritized so that the user knows which failures he should focus on more than the others. The failures are then reported in an order of priority. The failures listed highest in priority are more likely caused by something that was just changed with the current set of changes being tested or from that developer's prior changes (as opposed to another developer's changes).
When a given developer is ready to test changes he has just made to the software development project, he can run a series of automated or manual tests to confirm that the changes operate as expected. Any failures that occur during the automated or manual testing are stored in a data store 106 of historical failures 108. In the example shown in
In one implementation, the modified software development project must be tested before the developer or other user is even allowed to check in his latest changes into the central data store 106 for publication to other users. In another implementation, the tests can be run after the developer or other user has checked in his latest changes into the central data store for publication to other users. Once the tests are run, if any failures are identified, then the data store 106 is consulted to retrieve any historical failures 108 that are similar to or the same as the current failure. An analysis process is performed to categorize any current failures as new failures or old failures, and prioritize the failures.
The failure results are then reported or otherwise displayed to the user based upon the results of the prioritization. For example, new failures can be shown with a higher priority than old failures. As another example, failures caused by changes of the current user can be shown with a higher priority than failures caused by other users. In one implementation, the prioritized test failure results are emailed to one or more users, such as after an automated testing tool finishes the testing. In such an implementation, the automated testing tool can be responsible for sending the email or other notification to any users that requested notice of the prioritized test results. In another implementation, the failure results can be accessed using an interactive testing tool or other tool that can display a test failure report. These steps for analyzing historical data and prioritizing failures are described in further detail in
Turning now to
Similar failures can be determined in one of various ways, such as by performing a string comparison to determine that a given failure only differs from another failure by a certain number of characters, and thus they should be considered similar. Another technique for determining whether two failures are similar can be by consulting a database that contains an index of failures that are known to be similar to each other. Other techniques for determining that failures are similar can also be used. Once the historical failures have been identified that are the same or similar to any current failures, then the failures can be categorized in priority.
Failure A is compared to failure B (stage 312). If the failure has not occurred in the past (decision point 314), then the failure is categorized as a possible new failure/high priority failure (stage 318). If the failure has occurred in the past (decision point 314), and the failure has only occurred in the current user's jobs (decision point 316), then the failure is categorized as a possible new failure/high priority failure (stage 318). If the failure has occurred in the past (decision point 314), and the failure has occurred in other user's jobs (decision point 316), then the failure is categorized as a lower priority failure (stage 320).
The result of this process is that failures are categorized into three groupings of failures. The first priority grouping includes failures that have not occurred in past test runs before (i.e. that are brand new failures resulting from the current changes). The second priority grouping includes failures that have occurred in past automated test runs for the current user, but not other users. The third priority grouping includes failures that have occurred in past test runs for other users. In other implementations, a different number of categories and/or logic can be used to group the failures in a priority order. These three are just included as one non-limiting example to illustrate how different levels of priority can be assigned based upon comparison of the current failure to historical trends. Furthermore, the process described in
As shown in
Additionally, device 500 may also have additional features/functionality. For example, device 500 may also include additional storage (removable and/or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated in
Computing device 500 includes one or more communication connections 514 that allow computing device 500 to communicate with other computers/applications 515. Device 500 may also have input device(s) 512 such as keyboard, mouse, pen, voice input device, touch input device, etc. Output device(s) 511 such as a display, speakers, printer, etc. may also be included. These devices are well known in the art and need not be discussed at length here.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. All equivalents, changes, and modifications that come within the spirit of the implementations as described herein and/or by the following claims are desired to be protected.
For example, a person of ordinary skill in the computer software art will recognize that the examples discussed herein could be organized differently on one or more computers to include fewer or additional options or features than as portrayed in the examples.
