The technical field relates to tracing relationships between software artifacts. More particularly, the field relates to correlating artifacts such as, software design artifacts to determine a relationship.
During the lifecycle of large software projects several software design artifacts, such as, but not limited to, requirements specifications, test plans, use-case documents and class descriptions of object-oriented programming languages are generated. Any change in one such artifact could necessitate a change in many of the other software design artifacts as well. The ability to assess the impact of a change, in any one of the software design artifacts, on the rest of the project artifacts is a critical aspect of software engineering. However, as software design artifacts are created and updated their relationship to other artifacts is not always fixed and also not always apparent without the use of some sorts of explicit links established manually by the developers.
Maintaining user-specified explicit links between software design artifacts has many costs. For instance, such methods add substantial overhead in terms of computing resources needed to maintain the links throughout the software development lifecycle. In addition to the computing resources such methods require personnel, such as developers, business analysts, architects and other stake holders to spend valuable time to maintain the manual links.
Establishing and maintaining explicit links between related software design artifacts throughout the software development cycle also means that such relationships have to be continuously established and maintained through the software development lifecycle. Any gaps in such an effort will mean that certain relationships may be missed. Also, these methods do not lend themselves easily to ascertaining the relationships between software design artifacts in advanced stages of the software development lifecycle when a large number of such artifacts are already in place.
Thus, there is a need for automating the process of determining and quantifying relationships between various software design artifacts generated during the software development lifecycle.
Described herein are methods and tools for determining traceability information between concepts of different software design artifacts. In one aspect, the concepts are components of software design artifacts that capture aspects of a design of a software application, such as a requirements specification, a test plan, an object-oriented class framework specification, and use-case documentation. Concepts can be, for example, such software design details as individual class descriptions, individual classes, and individual test cases.
In one aspect, an automated tool parses software artifacts to extract individual concepts and key terms comprised therein. In other aspects, the concepts are compared to each other to quantify a similarity between them based on their key terms. In some other aspects, key terms are selected to be included in the similarity analysis by, for instance, removing certain words and phrases from the similarity analysis because they are too common in the English language and, hence, unhelpful in determining substantive similarity between concepts being compared.
The similarity analysis can be based on frequency of selected key terms and in another aspect, the value of the term frequency can be weighted by a term significance factor, such as the Inverse Document Frequency (IDF). The IDF, among other things, desirably accentuates or attenuates the importance of selected key terms in a similarity analysis based on their ubiquity within a selected domain of an enterprise.
In yet another aspect, similarity of key terms is not based on syntactical similarity but it also based semantic meanings of the key terms which are determined based at least in part on domain ontology including Entity description and Entity relationships, for instance.
In further aspects, the tools described herein are capable of implementing an impact analysis for quantifying the impact that changes in one concept may have on another concept based on their quantified relationship. Also, the relationship quantifier can be used by the tools described herein to provide a summary report of the various relationships between concepts such that a software designer can be made aware of the various relationships at a higher summary level and can then account for such relationships when designing the software.
In yet another aspect, key terms associated with a set of test cases comprised therein can be identified and compared with key terms of a set of requirements to establish a minimal sub-set of the entire set of test cases needed to cover the requirements. In one aspect, the method for establishing a minimal sub-set of test cases begins with maintaining a data structure with key terms associated with the set of requirements and another data structure having a list of the test cases that have key terms in common with the key terms of the requirements. The maintaining of the data structure, in this example, begins with iteratively updating the key term data structure by removing those key terms found in common with test cases being analyzed and updating the minimal test case data structure by adding test cases whose key terms are removed from the key term data structure. This is continued until test cases are exhausted or until key terms of the requirements are accounted for in the test cases listed in the minimal test case data structure. The process of software testing is made more efficient by identifying the minimal number of test cases needed to address the functionality of the software application as specified in the requirements.
The quantified implicit relationships derived through computation can be combined with relationships that are explicitly established to derive other relationships. Explicit relationships between concepts are provided by modeling tools, for instance. Some of the explicit relationships can also be specified manually by a user. The quantified implicit relationships can be combined with explicit relationships to determine relationships between concepts that would otherwise have to be computed through a similarity analysis or defined explicitly, for instance. Thus, unifying links across several sets of concepts can be established by combining relationships that are expressed explicitly with implicitly expressed relationships that are derived through computation.
Additional features and advantages will become apparent from the following detailed description of illustrated embodiments, which proceeds with reference to accompanying drawings.
FIGS. 12A-B are flow diagrams describing an exemplary method for determining a minimal set of test cases for testing a given set of requirements.
For instance, for a requirements artifact, the component concepts can comprise specific individual descriptions of various requirements related to a software application. Such requirements can be functional in nature, such as a particular expected response from an application for a particular type of request. Such requirements can also be security related requirements, such as restrictions on access to certain functionalities of the software for selected users. For a test plan document, the concepts can include individual test cases. For class description artifacts, the concepts can include descriptions of individual classes, such as their methods, attributes, and hierarchical relationships to other classes, for instance. For use-case artifacts, the concepts include individual use-case descriptions.
In one embodiment, relationships (e.g., 130 and 135) between concepts (e.g., 112 and 122) of different software design artifacts 120 and 110 can be quantified based on a relationship quantifier (e.g., a quotient) that quantifies the relationship based on an analysis of similarity between the concepts. According to one embodiment, the evaluation of the similarity is based on determining common occurrences of selected key terms used in the concepts being compared. In this manner, relationships that are only implicit during the course of software development activity can be identified. Armed with such information, a software engineer can make better decisions about his or her design changes.
For instance, the quantification of the relationships allows for impact analysis whereby the impact that a change in a requirement can have on a related test case can be established based on a relationship quotient between the requirement and the test case. Also, given a high degree of correlation between two concepts, a software engineer could be informed that he or she should make changes to a second concept, if he or she is changing the highly correlated first concept, for instance.
Although methods and system for establishing traceability between software artifacts are described herein with respect to artifacts related to software designs, the principles described herein can be applied to a variety of artifacts and is not restricted just to software design artifacts. For example, this method can also be used to determine relationships between E-mails and to build conversation threads, between various artifacts related to various entities and associated products.
Accordingly, at 240, term significance weights are desirably calculated for at least some of the key terms identified at 230 to quantify the significance of key terms prior to a similarity analysis between at least two concepts for establishing a relationship quotient quantifying their relationship. At 250, similarity between two concepts can be quantified, in this example, by determining at least how many of the key terms the concepts being analyzed have in common. The term significance weights are used in such a similarity analysis to adjust the significance of the various key terms to avoid the similarity to be based too heavily on terms that may not be important to the software design in question.
Non-template based freeform extraction of concepts from a software design artifact is also possible. For instance, the fact that selected terms and phrases are used in various types of concepts can be leveraged to determine concepts. Requirements typically use terms, such as “should”, “shall”, and “must.” Other techniques, such as using topic-detection to determine a change in context, or the topic of a paragraph, or a flow of paragraphs, can also be used.
User-assisted extraction of concepts is also possible. One such method would be to mark the concepts in the document using pre-defined style formats or ‘smart tags’. For example, a new style format (say ‘Concept’) can be created. The text representing a concept can be formatted with this new style (‘Concept’). These concepts can then be extracted by the tool.
Once the individual concept descriptions are extracted (e.g.,
Furthermore, some of the words extracted from a concept can be stemmed prior to conducting a similarity analysis with another concept. By stemming it is meant breaking down words to their root, or one or more syllables, or word portions and then treating such words broken down to the same stemmed root as if they are identical during a similarity analysis. This is so, because words with similar root forms, such as “saving” or “savings”, may have similar meaning in usage such that their presence in concepts being analyzed should be accounted for as if they are identical during a similarity analysis. Thus, at 540, stemming methods are applied to the words by a stemmer 315 implemented by the traceability engine 310 in
There are many methods that can be suitably used for stemming. One such method is the Porter's stemming algorithm. At 550, a listing of the set of key terms comprising words and phrases of a concept to be used in a similarity analysis with another concept is generated. In one embodiment, this list may comprise words and phrases that been processed through the stop list processing (530), or have been processed through the stemming methods (540), or both.
During the extraction of the key terms at 510, additional attributes can be extracted about the key terms using parts-of-speech analysis, word-sense-disambiguation, and word patterns, for instance. These attributes can describe the key terms and their specific form and/or context of usage in the concept. For example, “watch” when used in its noun sense would be an “object”, while in its verb sense it would be an “activity”. It is also possible to identify roles and relationships in an entity to make sense of words that describe the entity. Entity relationships describe relationships between actors (e.g., “Supervisor”, and “Sales manager”), locations (e.g., “Dallas”, and “Northern plant”), organizations (e.g., “ABC Corp”), specific persons (e.g., “John”), and objects (e.g., “sales order”, “invoice”).
The extraction of the attributes could be aided by domain ontology or a form of thesaurus that captures information that further describes the domain entity belongs to. The domain ontology could be a searchable database and have information like “sales manager” is a specific variant of “manager” and that “operator” reports to a “supervisor”. The domain ontology could also have descriptions and relations pertaining to the domain/industry/organization etc., for example, information about organizations, product-related information and so for the which are typically referred to as Entity Descriptions and Entity Relationships.
Such information improves the ability of deciphering a key term's meaning not just based on its syntactic representation but also based on its semantic use. For instance, in a statement such as, “John, from the dispatch section, sent the package”, the relationship role of John in the organization can be deduced. The relationship of “dispatch section” to the “send” activity can also be deduced with help of the available domain ontology. Furthermore, if certain roles or relationships are not represented exactly in a current form of the domain ontology they can be inferred instead and added to the domain ontology to further enhance the domain ontology.
Thus, domain ontology comprising information related to Entity Descriptions and Entity Relationships can be used to enhance the similarity analysis at least because similarity analysis based on key terms will not be based solely on syntactical similarity between key terms but it is also based on semantic meaning of the key terms. For instance, the key term “watch” as an “activity” will be not be considered equal to “watch” as an object. Further, based on entity relationship data of a domain ontology database, for instance, it can be determined that the term “sales manager” is a specific form of “manager” and hence, in a similarity analysis they can be considered to be similar key terms. Also based on the domain ontology data, the key terms “order” and “invoice” can be established to be related terms even though they are not similar syntactically. Thus, at least for the purposes of a similarity analysis, these words can be similar.
When a domain ontology or a dictionary is not complete, heuristics can be used to derive the additional information based on the context. A degree of confidence or certainty can also be associated with the descriptions inferred using heuristics. For instance, in a sentence like “Asimo moved towards the door”, ‘Asimo’ is likely to be an actor, a person or a system, based on the activity ‘move’. But in a sentence “Asimov wrote science fiction”, ‘Asimov’ is most likely a person from the activity ‘write’. Furthermore for instance, it can be inferred that a “Zonal manager” is a variant of manager, even when “zonal manager” is not defined in the ontology. In “A log entry is made by the system”, “log entry” can be assumed to be an object as the verb used is “made” and also from the article ‘a’ in ‘a log entry’. “The requestor sends a notice to the Order system.” Here, the requestor is the subject, doing some action. So, he can be a person/role or a system. From the word pattern ‘ . . . or’, it is more likely that the requestor is a role. For example, in a phrase like “the X” or “a X”, the term “X” is less likely to be a person but could be a role, object etc. Such information regarding key terms can assist in a similarity analysis.
Once the lists of key terms are available for the various concepts (e.g., 550), a similarity analysis can be conducted in order to establish a quantifiable relationship between the concepts. Such an analysis can include applying term significance weights to ensure the analysis is done in a manner that determines the similarity between concepts based on appropriate significance accorded to occurrences of the various key terms on which the analysis relies. A Cosine Measure analysis is one of the many different possible similarity measurement analyses that are suitable for determining similarity between two sets of key terms and, thereby, establishing a relationship between two concepts corresponding thereto.
The use of Cosine Measure (also referred to as Dot Product) for similarity analysis is described below with reference to examples. The correlation between two concepts is calculated using Cosine Measure as follows:
Where, fx,j denotes the frequency (Number of occurrences) of word j in concept x. Here in this example, the concepts whose relationship is being measured are x and y.
Consider x and y as the following two concepts comprising a use-case description and possibly a related class description:
Suppose the list of key terms in Use-case1 and Class1 and the frequency of their occurrence therein are as listed in Table 1 below.
Applying the Cosine Measure formula listed above to the data provided in Table 1 yields the following:
Thus, based on the selected key terms listed in Table 1, the relationship between Use-case1 and Class1 is quantified as 0.65. The Cosine Measure quantifies the similarity-based relationship between two concepts between the values of 0 to 1 with the closest relationship being quantified as 1 and the least close relationship being quantified as 0.
Also, as noted above, information from domain ontology including Entity Descriptions and Entity Relationships can be used to enhance the similarity-analysis measurement.
One modification to the Cosine Measure method for determining similarity between concepts relates to addressing the fact that, in some software designs, the similarity analysis between concepts may depend far too much on certain key terms that are ubiquitous in many of the artifacts related to the particular type of enterprise. As a result, concepts that may not have a significant substantive relationship may nevertheless be determined to be similar based on the occurrence of such ubiquitous terms. One way to address this problem is to calculate the term significance weights for at least some of the key terms and apply these term significance weights as coefficients in a similarity analysis (e.g., 240). For instance, these term significance weights can be applied as coefficients of the key term frequencies in a Cosine Measure analysis to weight the term frequency numbers.
For instance, consider the following set of requirement concepts:
The term frequency for the various key terms of requirement listed above as (iii) may be as follows:
Considering the requirement (iii), the term “Account” appears thrice, so it is the most important word from a term frequency perspective. On the other hand, the terms “Savings Account” and “Current Account” appear only once in this requirement. But, if the similarity analysis takes a broader perspective (e.g., across several requirements), it can be determined that the term “Account” is in fact too ubiquitous a term. Thus, considering only the term frequencies during a similarity analysis (e.g., Cosine Measure) would lead to the term “Account” to gain an inflated level of importance and, hence, adversely affecting the similarity analysis.
Calculating a term significance weight such as the IDF, for the key terms can, however, add a proper weight to the importance of the key terms during a similarity analysis. One exemplary IDF calculation is as follows:
Accordingly, the IDF for the various key terms of the requirement (iii) listed above is as follows:
Thus, the IDF for “Account” is 0.0 and “Savings Account” is 0.4. So now when term significance weights, such as the IDFs, are applied as coefficients to the term frequency in a similarity analysis such as the Cosine Measure, the key term “Account” becomes less important while the key term “Savings Account” becomes more important, which is desirable.
The relationship quotients that result from a similarity analysis between concepts of different software design artifacts can be captured in the form of the exemplary matrix 700 as shown in
One exemplary user interface 800 for sharing the relationship data in a user-friendly manner, as shown in
Although it can vary, in one example, a normalized relationship quotient value range of 0.7 to 1 indicates a High level relationship, a range of 0.5 to 0.7 indicates a Medium level relationship, and the range of 0 to 0.5 indicates a Low level relationship.
An impact analysis translates the quantified relationships between concepts to a level of impact a change in one concept can have on another concept based on the value of the relationship quotient. Based on this assessment, a software developer can gauge the impact of changes proposed for one aspect of the design might have on other aspects of the design.
Suppose there exists a set of requirements {R1, R2 . . . . Rn} and corresponding sets of design documents {D1, D2 . . . Dn}, class specifications {C1, C2 . . . Cn}, and test plans {T1, T2 . . . Tn}. Now, if requirements R2 is changed, all the other designs, class specifications, and test plans corresponding to requirement R2, may have to be changed. Finding these other concepts (i.e., requirements, designs, class specifications, test plans, etc.) corresponding to a particular concept facilitates the determination of which of the other concepts may also require change. Using quantified matrices (e.g., 700 of
Based on the data from Table 2 above, the impact of change in concept R2 on D7, T5, T8, D4 and C6 will be high. Whereas concepts D3 and C9 will experience a small impact, while C3 is expected to experience almost no impact. This can help in planning tasks more accurately for a scope change request during the development process, for instance.
The impact analysis can also be performed once key terms have been added or deleted to alter a concept. The corresponding concepts that contain those added or deleted key terms can be considered in an impact analysis (e.g., including term weights). Furthermore, if one concept is changed by adding key term for instance, an impact analysis will inform a user which are the other concepts that may be affected and these concepts may be presented to a user for consideration. In one specific example, an added key term to a requirement may necessitate addition of a test case. However, a post addition impact analysis might reveal existing test cases that already address the testing related to the added key term. In this case, these test cases are presented to a user for consideration.
This can be implemented in a user interface which can instantly inform the user about the extent of impact and the impacted concepts, whenever a concept is modified.
Once the key terms of a concept are extracted (e.g., as in
Once the missed key terms between selected sets of concepts are available, other useful analysis of the data can be performed to further improve the software development process. During software testing, many redundant test cases that address substantially the same functionality of software application are often implemented. Accordingly, it is desirable to find a minimal set of test cases that cover the requirements of a software application is desirable. One manner in which this can be automated is by post-processing the output from the comparison analysis (e.g., 1100 of
Consider a set of requirements {R1, R2, R3, R4, R5, R6} with a corresponding set of test cases {T1, T2, T3, T4, T5, T6} having the following key terms:
Complete coverage of key terms of requirements in Table 3 above in the test by a minimal set of test cases can be achieved by creating a set of missed key terms (MK) which initially contains all the key terms that appear in all the requirements. Thus, such a set MK is initialized, in this example, to comprise {K1, K2, K3, K4, K5, K6, K7, K8, K9, K10, K14}. Also a set of minimal test cases is created and initialized to null, for instance, as TC comprising { }. Later the test case with the most number of key terms appearing in MK is added to the minimal set TC. In this example, the test case T3 has the highest number of key terms in the current make up of the set MK so as a result, TC now comprises {T3}. Also, all those key terms present in the test case T3 can now be removed from MK. As a result, MK would then comprise {K4, K5, K7, K8, K9, K10, K11, K14}. We continue repeating these steps until MK is empty or none of the remaining key term elements of MK are covered by any test case. Thus, in the above example, the minimum set of test cases that covers most of the key terms of the requirements {R1, R2, R3, R4, R5, R6} that the given set of test cases {T1, T2, T3, T4, T5, T6} can possibly cover is determined to be TC comprising {T3, T4, T5, T6}.
In this example, key terms {K9 and K14} are not covered by any of the test cases {T1, T2, T3, T4, T5, T6}. Thus, as a result of this analysis, a developer can be informed of the lapse and he or she can add alter the existing test cases or add new test cases to address the gap. On the other hand the developer may decide not address the gap for various design specific reasons.
Identification of the minimal test case for testing a given set of requirements allows software developers to reduce the number of redundant test cases, thus, making the process of creating and testing software more efficient. An automated tool can calculate and present such information with minimal effort on part of the user.
Although the examples regarding minimal test case analysis are presented with respect to a comparison of key terms associated with test case concepts and requirement concepts these methods are equally applicable to any combination of concept types.
The quantified trace relationships that are computed by a traceability engine (e.g., 310 of
If some explicit links are already in place (e.g., either user maintained or modeling tool maintained in-process relationships), there is a benefit to using them in order to establish relationship traceability links between concepts that otherwise may have to be computed.
The unified links and the relationships they describe, once established can be analyzed much like the computationally derived relationships. Such relationships can also be displayed or otherwise reported to a user via a user-interface. The form in which information regarding the display is shared can be varied. For instance, it could be in the form of the exemplary relationship diagram 1300 of
With reference to
A computing environment may have additional features. For example, the computing environment 1700 includes storage 1740, one or more input devices 1750, one or more output devices 1760, and one or more communication connections 1770. An interconnection mechanism (not shown) such as a bus, controller, or network that interconnects the components of the computing environment 1700. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing environment 1700, and coordinates activities of the components of the computing environment 1700.
The storage 1740 may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, CD-RWs, DVDs, or any other medium which can be used to store information and which can be accessed within the computing environment 1700. For instance, the storage 1740 can store instructions for the software 1780 implementing methods of automated relationship traceability between software design artifacts.
The input device(s) 1750 may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computing environment 1700. For audio, the input device(s) 1750 may be a sound card or similar device that accepts audio input in analog or digital form, or a CD-ROM reader that provides audio samples to the computing environment. The output device(s) 1760 may be a display, printer, speaker, CD-writer, or another device that provides output from the computing environment 1700.
The communication connection(s) 1770 enable communication over a communication medium to another computing entity. The communication medium conveys information such as computer-executable instructions, compressed graphics information, or other data in a modulated data signal.
Computer-readable media are any available media that can be accessed within a computing environment. By way of example, and not limitation, with the computing environment 1700, computer-readable media include memory 1720, storage 1740, communication media, and combinations of any of the above.
The various tools and systems, such as but not limited to, the traceability engine (e.g., 310 in
Having described and illustrated the principles of our invention with reference to the illustrated embodiments, it will be recognized that the illustrated embodiments can be modified in arrangement and detail without departing from such principles. Elements of the illustrated embodiment shown in software may be implemented in hardware and vice versa. Also, the technologies from any example can be combined with the technologies described in any one or more of the other examples.
In view of the many possible embodiments to which the principles of the invention may be applied, it should be recognized that the illustrated embodiments are examples of the invention and should not be taken as a limitation on the scope of the invention. For instance, various components of systems and tools described herein may be combined in function and use. We therefore claim as our invention all subject matter that comes within the scope and spirit of these claims.