To aid in the creation of computer programs and components of computer-executable instructions, various tools have been developed that seek to identify potential problems. One such set of tools is directed to the testing of computer programs and components by executing them under controlled circumstances and observing the results. The testing of computer-readable code seeks to examine the actual performance of the code when executed and identify any unexpected actions on the part of the code, or any unexpected results. Traditionally, such unexpected actions or results are referred to as “bugs” in the code. Once identified through testing, a targeted examination of the relevant computer-executable instructions can be undertaken to identify the cause of the unexpected actions or results and, hopefully, corrective measures can be identified and performed.
Another set of tools used to examine computer programs and components are verification tools that seek to analyze the computer-readable code itself, rather than its actual behavior. Such verification tools can identify issues in the code itself, such as failures to conform to guidelines regarding the expected behavior of the code, failures to conform to the specifications of the code's interaction with external elements, failures to adhere to specific coding conventions, or other similar issues. Verification tools often comprise a verification engine that can perform the actual verification, and one or more rules that can define or indicate the relevant expected behavior, specification of interaction with external elements, coding convention, or other issue that is to be verified. Often, the computer-readable code being verified may require some supporting structure. For example, the verification of a component may require at least some aspects of the other elements that component is designed to interact with, or the component may not be able to be verified properly. Such execution environment issues can be addressed by environmental models that can be associated with relevant rules and can be provided to the verification engine, or they may even be addressed by different types of verification engines themselves.
Traditionally, code verification tools sought to generate an abstraction of the computer program or component being verified, often in the form of a mathematical model. Such efforts can be non-linear and, as a result, depending on the complexity and quantity of the computer-readable code being verified, verification attempts can consume all available memory resources, or can otherwise take so long that, from a practical perspective, the verification attempt will not finish. Traditional solutions to such inefficiencies were directed to the verification engine itself, as a faster verification engine can improve the efficiency with which any computer-readable code, simple or complex, is verified.
Aspects of the verification of computer-executable instructions are often complex and multi-faceted. In particular, verification that computer-executable instructions conform to specific expected behaviors, specifications or coding conventions may be more easily conceptualized as multiple correlated verifications, each directed to a simpler issue. Code verification tools that can first check if certain conditions are met prior to verifying a particular set of computer-executable instructions, can avoid verifications that may be irrelevant to that particular set of computer-executable instructions, thereby increasing the efficiency of the overall verification process. In addition, code verification tools that can process an overall verification as a series of correlated verifications, each directed to a simpler issue, can render such verifications easier for human users to set up and can make it more likely that the verification will succeed.
In one embodiment, therefore, a verification system can accept the specification of “precondition rules” that can limit the circumstances in which particular verification rules are applied to a set of computer-executable instructions being verified. The precondition rules can obtain information regarding the computer-executable instructions being verified and, on the basis of the obtained information, a particular set of rules can be selected, or deselected, against which the computer-executable instructions can be verified.
In another embodiment, an environment model representing the computer-executable instructions which interact with the computer-executable instructions being verified can likewise be selected, or deselected, prior to verification based on the results from one or more precondition rules. The environment model can be associated with a particular rule, and can, thereby, be selected, or deselected, together with the associated rule, depending on the results of the evaluation of the precondition rules. Alternatively, a single rule can be applied to verify computer-executable instructions using any one of a multiple of different environment models and, in such a case, the environment model can be selected based on the precondition rules independently of the rule with which the computer-executable instructions are being verified.
In a further embodiment, the precondition rules can themselves comprise further precondition rules, such that some precondition rules either are, or are not, evaluated based on the results of precondition rules to those precondition rules. Such multiple layers of precondition rules can be evaluated using any of a multiple of evaluation mechanisms. In one embodiment, a “lazy” evaluation mechanism can examine a precondition rule to identify any precondition rules to that precondition rule. Such an examination can proceed until a terminal level of precondition rules is reached, at which point those precondition rules can be evaluated. Precondition rules dependent on those precondition rules can then be evaluated, and the evaluation can proceed in such a manner until the first level of precondition rules is reached, at which point, analysis can proceed to the next precondition rule at that level. In an alternative embodiment, an “eager” evaluation mechanism can examine some or all of the multiple levels of precondition rules and can categorize the precondition rules according to level. Subsequently, evaluation of the precondition rules at the terminal level can be performed, followed up evaluation of the precondition rules at the immediately higher level, and proceeding in such a manner until the precondition rules at the first level are evaluated. While such an evaluation mechanism can result in a simplified evaluation strategy, it can also result in the evaluation of precondition rules that end up being irrelevant for other reasons and can, thereby, introduce a measure of inefficiency.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
Additional features and advantages will be made apparent from the following detailed description that proceeds with reference to the accompanying drawings.
The following detailed description may be best understood when taken in conjunction with the accompanying drawings, of which:
The following description relates to the verification of computer-executable instructions by first referencing precondition rules to inform the selection, or deselection of various verification mechanisms. Evaluation of the precondition rules can provide information regarding the computer-executable instructions to be verified, thereby enabling the selection of appropriate verification rules and environment models, and possibly even evaluation engines, and the deselection of inappropriate verification rules and environment models. Evaluation of the precondition rules can be performed using a “lazy” mechanism, whereby levels of precondition rules associated with one first level precondition rule are evaluated before levels of precondition rules associated with another first level precondition rule are evaluated, or the evaluation of precondition rules can be performed using an “eager” mechanism, whereby precondition rules are evaluated by level, commencing with a terminal level.
The techniques described herein focus on, but are not limited to, the utilization of precondition rules by a set of static verification tools, such as may be developed for, and provided to, a wide range of consumers. Indeed, the techniques described herein are equally applicable to customized verification solutions. As such, references to verification engines, environment models, rules, precondition rules, and modules are not meant to be limited to the particular examples described, but are instead intended to encompass any suitable such elements.
Turning to
In some cases the verification of the module 30 by the verification engine 40 can utilize an abstraction of computer-executable instructions that can invoke the module, an abstraction of computer-executable instructions that can provide support for the module, or a combination thereof. For example, a peripheral device driver, such as would execute within the context of an operating system, may be verified with reference to function calls, such as would be made by a higher level application to invoke the peripheral device driver, and operating system support, such as would be provided to the peripheral device driver by the operating system in response to requests from the peripheral device driver.
For purposes of verification of the module 30 with respect to a particular rule 20, a scenario model 50 can abstract the relevant features and functionality of higher-level software applications or other computer-executable instructions that can invoke, or make requests of, the module. Similarly, for purposes of verification of the module 30 with respect to a particular rule 20, a platform model 60 can abstract the relevant features and functionality of lower-level operating systems or other computer-executable instructions that can provide information or support to the module. For ease of reference, the scenario model 50, the platform model 60, or combinations thereof, whatever may be relevant to the rule 20, can be referred to as the environment model 10. Thus, in the system 99 of
As described herein, the verification engine 40, the module 30, the rule 20 and the environment model 10, comprising the scenario model 40 and the platform model 60, can each comprise computer-readable instructions, computer-readable data, or combinations thereof. As such, although not required, the descriptions herein will be in the general context of computer-executable instructions, such as program modules, being executed by one or more computing devices. More specifically, the descriptions will reference acts and symbolic representations of operations that are performed by one or more computing devices or peripherals, unless indicated otherwise. As such, it will be understood that such acts and operations, which are at times referred to as being computer-executed, include the manipulation by a processing unit of electrical signals representing data in a structured form. This manipulation transforms the data or maintains it at locations in memory, which reconfigures or otherwise alters the operation of the computing device or peripherals in a manner well understood by those skilled in the art. The data structures where data is maintained are physical locations that have particular properties defined by the format of the data.
Generally, program modules include routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the computing devices need not be limited to conventional personal computers, and include other computing configurations, including hand-held devices, multi-processor systems, microprocessor based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. Similarly, the computing devices need not be limited to a stand-alone computing device, as the mechanisms may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
With reference to
The computing device 100 also typically includes computer readable media, which can include any available media that can be accessed by computing device 100 and includes both volatile and nonvolatile media and removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computing device 100. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.
The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computing device 100, such as during start-up, is typically stored in ROM 131. RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation,
The computing device 100 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media discussed above and illustrated in
Of relevance to the descriptions herein, the computing device 100 may operate in a networked environment using logical connections to one or more remote computers. For simplicity of illustration, the computing device 100 is shown in
Turning back to
Turning to
For ease of illustration and description,
A particular rule, such as rule 20, can seek to verify a conformance to a particular guideline regarding the expected behavior of the module 30, conformance by the module 30 to particular specifications defining its interaction with external elements, conformance by the module 30 to specific coding conventions, or other similar issues. Because such guidelines, specifications, coding conventions and the like are often expressed in conditional form, a rule seeking to verify conformance to them can likewise often be conceptualized as a series of related rules that can be conditionally applied. For example, a simple guideline regarding the implementation of particular interfaces related to the power policy applied to a particular hardware device can be in a conditional form of: IF a module is a power policy owner for a particular device, THEN the module must implement particular interfaces related to power policy ownership. A slightly more complex variant of the same guideline can be in an alternative condition form of: IF a module is a power policy owner for a particular device, THEN the module must implement particular interfaces related to power policy ownership, AND IF a module is not a power policy owner for a particular device, THEN the module must be able to respond to particular instructions regarding the power policy of the device. A rule, such as rule 20, seeking to verify conformance with this exemplary guideline can be conceptualized as two related rules: the first rule seeking to verify implementation of the specified interfaces if the module 30 is the power policy owner for a particular device, and the second rule seeking to verify the module's ability to respond to the specified instructions if the module is not the power policy owner. Taken together, these two rules can be thought of as a single rule, such a rule 20, that verifies conformance to this exemplary guideline regarding power policy ownership.
Rule A 255, rule B 265 and rule C 275 illustrated in
In an alternative embodiment, rule B 265 and rule C 275 can merely be modified versions of rule A 255. Likewise, environment model B 260 and environment model C 270 can be modified versions of environment model A 250. Rule B 265 and rule C 275, as modified versions of rule A 255, need not be preexisting and can, instead, be generated after evaluation of the precondition rules 210. In such a manner, the selected rule can be modified based on the results of the evaluation of the precondition rules 210. Similarly, environment model B 260 and environment model C 270, as modified versions of environment model A 250, need not be preexisting and can, instead, be generated based on the evaluation of the precondition rules 210. In one embodiment, such a modification of rule A 255 into rule B 265 or rule C 275, or a modification of environment model A 250 into environment model B 260 or environment model C 270, can be performed by the preprocessor 240 or the verification engine 40 as part of the selection process described below.
The selection of a rule, such as rule A 255, rule B 265 and rule C 275, and potentially an associated environment model, such as environment model A 250, environment model B 260 and environment model C 270, with which the module 30 will be verified by the verification engine 40 can be informed by one or more precondition rules 210. Returning to the above-enumerated exemplary power policy ownership guideline, the precondition rules 210 can be directed to the determination of whether the module 30 is a power policy owner or not. For example, a precondition rule, such as precondition rule 210, can determine if the module 30 comprises the functionality expected from a power policy owner. If, using the precondition rule 210, the results 230 indicate that the module 30 is a power policy owner, then the environment model A 250 and the rule A 255, for example, can be selected by the verification engine 40 and it can verify the module 30 with respect to rule A. Alternatively, if, using the precondition rule 210, the results 230 indicate that the module 30 is not a power policy owner, then the environment model B 260 and the rule B 265, for example, can be selected by the verification engine 40 and it can verify the module 30 with respect to rule B. In such a manner, the precondition rules 210 can enable the selection of a subset of rules and, possibly, environment models as well by which the module 30 can be verified by the verification engine 40, thereby rendering the overall verification system 200 more efficient.
In one embodiment, the precondition rules 210 can be processed by an evaluation tool, such as the evaluation tool 220. Depending on the precondition rule 210, the evaluation tool 220 can process the precondition rule by referencing metadata associated with the module 30, by directly examining the module, including its structure, properties and even its behavior, or by performing other actions with respect to the module. As indicated previously, in one embodiment, the evaluation tool 220 can be no different than the verification engine 40 and can process the precondition rule in the same manner as the verification engine would process a rule, such as one or more of the rules 255, 265 and 275. The precondition rules 210 can be in the form of computer-executable instructions that can be understood by the evaluation tool 220 and can instruct the evaluation tool to determine information about the module 30. For example, a precondition rule 210 to determine if the module 30 is a power policy owner can comprise computer-executable instructions directing the evaluation tool 220 to determine if the module calls specific functions, or if the module itself comprises, and has the capability to respond to, specific types of functions.
In one embodiment, the precondition rules 210 can additionally comprise computer-executable instructions that can instruct the evaluation tool 220 to reach a conclusion, reflected in the results 230, based on the information obtained. For example, if the evaluation tool 220 determines that the module 30 comprises a specific type of function, the precondition rule 210 can comprise computer-executable instructions that can cause the evaluation tool to reach the conclusion that the module is not a power policy owner. In such an embodiment, the results 230 can indicate the conclusion that the module 30 is not a power policy owner. In an alternative embodiment, however, the results 230 can merely indicate the results of various determinations performed by the evaluation tool 220, as dictated by the precondition rules 210, leaving the ultimate evaluation of such results to the verification engine 40.
Irrespective of whether the results 230 comprise conclusions or mere evaluation data, they can be utilized by the verification engine 40 to select a rule, and possibly an associated environment model, with which the module 30 can be verified. In one embodiment, the verification engine 40 can utilize a preprocessor 240 that can evaluate conditional statements associated with the rules 255, 265 and 275, and with the environment models 250, 260 and 270, and can select an appropriate rule, and, if appropriate, a corresponding appropriate environment model, with which the module 30 can be verified by the verification engine 40.
Returning to the above example regarding an overall rule 20, designed to verify compliance with power policy ownership guidelines by the module 30, such a rule can, as indicated previously, be divided into a rule A 255, and an associated environment model A 250, that can be applicable if the module is a power policy owner and a rule B 265, and an associated environment model B 260, that can be applicable if the module is not a power policy owner. The results 230 from the evaluation tool 220 can, in one embodiment, indicate whether or not the module 30 is a power policy owner since, as indicated previously, the precondition rules 210 can have provided instructions for the evaluation tool 220 to obtain information about the module 30 relevant to the determination of whether the module is a power policy owner. In such an embodiment, the preprocessor 240 can evaluate a conditional statement that can specify that if the results 230 indicate that the module 30 is a power policy owner, then the preprocessor is to select rule A 255 and its associated environment model A 250 and if the results indicate that the module is not a power policy owner, then the preprocessor is to select rule B 265 and its associated environment model B 260.
In an alternative embodiment, the results 230 can comprise only the data obtained by the evaluation tool 220 without specific conclusions, such as whether the module 30 is a power policy owner. In such an embodiment, the precondition rules 210 may not have comprised relevant instructions by which the evaluation tool 220 can have reached a specific conclusion, or the evaluation tool may not have evaluated those relevant instructions even though they may have been present in the precondition rules. If the results 230 comprise merely data obtained by the evaluation tool 220, the preprocessor 240 can evaluate the conditional statements that can be contained in the precondition rules 210, or in another entity, such as header files, and can itself reach those conclusions that may be required for selection from among the rules 255, 265 and 275. For example, if the results 230 comprise an indication of whether or not the module 30 comprises a specific type of function, the preprocessor 240 can evaluate a conditional statement indicating that if a module comprises that type of function, then the module is not a power policy owner. The preprocessor 240 can, therefore, itself reach a conclusion, such as, for example, whether the module 30 is a power policy owner or not, and can then, in the manner described above, select from among the rules 255, 265 and 275, and associated environment models 250, 260 and 270.
As indicated previously, the evaluation tool 220, preprocessor 240 and the verification engine 40 were shown in
In the embodiment represented by the verification system 300 of
In such a manner, the evaluation tool 220 functionality of the verification engine 40 can first perform a complete evaluation of the module 30, to the extent instructed by the precondition rules referenced in the “precondition.h” 310 header file, and the verification engine 40 can then utilize the preprocessor 240 functionality to select one or more rules with which the module 30 will be verified, based on the results contained in the “pre-results.h” 320 header file. Thus, as shown, the “pre-results.h” 320 header file can be provided back to the verification engine 40 to enable the preprocessor 240 functionality of the verification engine to reference the results contained in that header file, together with the requirements of one or more rules, as specified in rule header files, such as the “rule.h” 330 header file, to determine which rules should be used to verify the module 30. Once those rules are selected, the module 30 can be verified and the verification engine 40 can produce the results 70.
The flow diagram 400 of
The evaluation of the precondition rules at step 420 can be performed in a straightforward manner if none of the precondition rules themselves have precondition rules. In such a case, evaluation of the precondition rules at step 420 can proceed in, for example, a linear fashion, with one precondition rule being evaluated after another, without regard to the order in which they are evaluated. However, if the precondition rules themselves have precondition rules, then the evaluation can utilize various mechanisms that are directed to multiple layers of precondition rules. While two such mechanisms are specifically illustrated and described below, the descriptions are meant to be exemplary only and are not meant to be limiting. Any mechanism by which a structure of related precondition rules is traversed in a logically consistent order can be utilized.
As indicated previously, a precondition rule, such as precondition rules 210 of
For ease of reference, precondition rule dependences can be ordered into “levels” such that a “first level” precondition rule can have no other precondition rules whose application is dependent on the results of the “first level” precondition rules. Similarly, a “second level” precondition rule can have its results be utilized to determine the applicability of a “first level” precondition rule. A precondition rule that itself has no precondition rules or, using the above terminology, has no levels of precondition rules below it, can be nominated a “terminal level” precondition rule. Of course, some “terminal level” precondition rules may be “first level” precondition rules, some may be “second level” precondition rules, and some may be even lower level precondition rules. Consequently, the “terminal level” need not refer to a particular level, per se, but rather can refer to the nature of the precondition rule itself, namely in not having any further precondition rules to it.
Turning to the flow diagram 500 of
However, if, at step 520, it is determined that the precondition rule being considered itself has precondition rules, those precondition rules can be examined at step 530, thereby causing execution to return to step 520 and consider whether the precondition rule now being considered has precondition rules. As will be recognized by those skilled in the art, a net effect of the operation of steps 520 and 530 is that the evaluation step 540 will first be performed on a precondition rule that has no precondition rules or, to use the above terminology, is a “terminal level” precondition rule.
Subsequently, after such a terminal level precondition rule is evaluated at step 540, a check can be made at step 550 to determine if there is another precondition rule at that same level. If there is another precondition rule at the same level, that rule can be examined at step 560 in the same manner as just described with reference to steps 520 through 540, since, as indicated previously, the mere fact that the precondition rule now being examined at step 560 was at the same level as a precondition rule that was a “terminal level” precondition rule does not necessarily mean that the rule now being examined is, itself, a “terminal level” precondition rule.
Once all of the precondition rules at the same level have been examined, as determined by step 550, a determination can be made at step 570 if there are any higher level precondition rules. If there are, then at step 580 those higher level precondition rules can be examined in the same manner as described above with reference to steps 520 through 560. If, at step 570, it is determined that there are no higher level precondition rules, then the evaluation of the precondition rules can end at step 590.
As can be seen from the above descriptions, one of the net effects of the flow diagram 500 is to select a given “first level” precondition rule, drill down a particular dependency chain until a “terminal level” precondition rule is reached, evaluate that precondition rule, and then work back up the dependency chain, at each point drilling down another leg of the dependency chain in an equivalent manner. While flow diagram 500 can comprise multiple examination steps, such as steps 530, 560 and 580, it can also avoid evaluating precondition rules that are irrelevant. In particular, depending on the conditional statements associated with a precondition rule, some precondition rules can be definitively found to be relevant, or not relevant, to a particular module before all of the precondition rules of a lower level associated with that precondition rule have been evaluated. In such a case, because a definitive determination can already have been made, the remaining lower level precondition rules may not change the determination and can, therefore, be skipped, since they are no longer relevant.
An alternative multi-level precondition rule evaluation mechanism is illustrated by flow diagram 600 of
After at least some, if not all, of the precondition rules have been ranked into these types of dependency levels at step 620, the precondition rules at the “terminal level” can be evaluated at step 630. In performing the evaluation of the “terminal level” precondition rules at step 630, precondition rules belonging to different dependency chains can be evaluated sequentially without completing an evaluation of the dependency chain as would have been done by flow diagram 500. After the “terminal level” precondition rules are evaluated at step 630, a check can be made at step 640 for any higher level precondition rules. If there are higher level precondition rules found at step 640, they can then be evaluated at step 650. Subsequently, processing can return to the check at step 640 to determine if there are any even higher level precondition rules. Once “first level” precondition rules have been evaluated by step 650, and the check at step 650 determines that there are no further higher level precondition rules, the evaluation of the precondition rules can end at step 660.
The mechanisms of the flow diagram 600, as can be seen, may not drill down in the same manner as those illustrated in flow diagram 500 and, consequently, fewer examinations of the precondition rules may be performed. However, in some cases, the mechanisms shown in flow diagram 600 can evaluate precondition rules at lower levels that may turn out to have been irrelevant since a related higher level precondition rule can have been found to be inapplicable due to other lower level precondition rules. However, because each of the flow diagrams 500 and 600 illustrate mechanisms that have particular advantages, the selection of one such mechanism can be, in part, dictated by the types of precondition rules present in the verification system.
As can be seen from the above descriptions, mechanisms for providing precondition rules to verification rules have been provided. In view of the many possible variations of the subject matter described herein, we claim as our invention all such embodiments as may come within the scope of the following claims and equivalents thereto.