The present disclosure relates to verification of computerized devices, in general, and to coverage analysis of the verification process, in particular.
Computerized devices are an important part of the modern life. They control almost every aspect of our life—from writing documents to controlling traffic lights. However, computerized devices are bug-prone, and thus require a verification phase in which the bugs should be discovered. The verification phase is considered one of the most difficult tasks in developing a computerized device. Many developers of computerized devices invest a significant portion, such as 70%, of the development cycle to discover erroneous behaviors of the computerized device, also referred to as a target computerized system. The target computerized system may comprise hardware, software, firmware, a combination thereof and the like.
Coverage analysis provides information on the status and progress of the verification process. For example, uncovered events point to areas or features in the design that are not exercised enough. In addition, coverage progress report may provide information on how rapid (or slow) is the progress of the verification process and may be utilized to predict if and how soon coverage goals will be met. Coverage analysis may be utilized in respect to verification of various computerized devices, such as for example, a computer program, a processor or another hardware component, or the like. The present disclosure discusses coverage analysis of a target computerized system, also referred to as a Design Under Verification (DUV). DUV may be software, firmware, hardware or any type of device being verified.
Frequent changes to the DUV and a corresponding verification environment may lead to a need to know not only what the total coverage picture is, but also what has been done lately. For example, if an area in the design is modified, old coverage event relating to the area does not indicate that the area was re-tested properly after the change.
To address this problem, coverage analysis tools provide the ability to look at the coverage picture in a sliding window of time, also referred to as a sliding window report. For example, which events have been covered in the last 10000 tests or during the last seven days. Sliding window reports are provided in addition or instead of reports that look at the total picture.
One exemplary embodiment of the disclosed subject matter is a coverage analysis apparatus having a processor, the coverage analysis apparatus comprising: a receiver configured to receive coverage events from a coverage events repository, each of the coverage events comprising a first timestamp, wherein the first timestamp indicates a last time in which the coverage event was covered; and a coverage status module configured to provide an indication of a portion of the coverage events that was not covered recently, wherein the indication is based on the coverage events.
Another exemplary embodiment of the disclosed subject matter is a method for determining coverage analysis, the method comprising: receiving coverage events from a coverage events repository, each of the coverage events comprising a first timestamp, wherein the first timestamp indicates a last time in which the coverage event was covered; determining an indication of a portion of the coverage events that was not covered recently, wherein the indication is based on the coverage events; and outputting the indication.
Yet another exemplary embodiment of the disclosed subject matter is a computer program product comprising: a computer readable medium; a first program instruction for receiving coverage events from a coverage events repository, each of the coverage events comprising a first timestamp, wherein the first timestamp indicates a last time in which the coverage event was covered; a second program instruction for determining an indication of a portion of the coverage events that was not covered recently, wherein the indication is based on the coverage events; a third program instruction for outputting the indication; and wherein said first, second and third program instructions are stored on said computer readable medium.
The present disclosed subject matter will be understood and appreciated more fully from the following detailed description taken in conjunction with the drawings in which corresponding or like numerals or characters indicate corresponding or like components. Unless indicated otherwise, the drawings provide exemplary embodiments or aspects of the disclosure and do not limit the scope of the disclosure. In the drawings:
The disclosed subject matter is described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the subject matter. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
One technical problem dealt with by the disclosed subject matter is to provide coverage analysis. The coverage analysis may provide information enabling a verification engineer or another QA personnel to determine a verification strategy, such as to reexecute a test, to design a new test or the like.
One technical solution is to provide coverage analysis based on last hit data. The last hit data comprises a timestamp indicating a last time in which a coverage event was covered. As opposed to prior art coverage analysis techniques which utilize a first hit data indicating a time in which the coverage event was covered first, a last hit data enables the verification engineer to determine whether a coverage event that was covered should be covered again. Based on last hit data, a sliding window may be selected, and a coverage analysis report based on the sliding window may be provided.
One technical effect of utilizing the disclosed subject matter is to enable a verification engineer to distinguish between coverage events that were not covered at all and coverage events that were not covered lately. This distinction may be utilized to determine whether a test may be reexecuted to cover the coverage event or whether a new test should be design to cover the coverage event. In some exemplary embodiments, reexecution may not achieve coverage as the design may have changed. The disclosed subject matter may enable the verification engineer to identify, based on the aforementioned distinction, a modification made to the DUV that was not handled correctly, such as by modifying pertinent tests, coverage goals or the like. The modification to the DUV may be a modification in the design itself, the environments or the tests. Another technical effect of utilizing the disclosed subject matter is to overcome drawbacks associated with prior art sliding window reports. As opposed to a sliding window report which requires an a-priori determination of a size of the window, the disclosed subject matter does not require such determination. Determining the “correct” size of the window is a difficult problem. The size of the window may depend on the size and difficulty of the coverage model, the availability of resources, rate of changes to the design, etc.
Referring now to
In some exemplary embodiments of the disclosed subject matter, a coverage analysis tool 110 may provide a coverage analysis information based on coverage events stored in a coverage events repository 120. The coverage events repository 120 may be a database, a storage system or the like. The coverage events repository 120 may comprise coverage events covered during the verification phase, since a last reset of the coverage events repository 120, or the like.
In some exemplary embodiments, a verification engineer 130 may utilize a terminal 135 or a similar Man-Machine Interface to view information provided by the coverage analysis tool 110. The information may be displayed in a hard copy format, a soft copy format or the like.
In some exemplary embodiments, a verification tool 140 exercises a DUV 150, by performing tests on the DUV 150. The tests may be stored in a test benchmark 160 as is known in the art. The verification tool 140 may execute a test a predetermined number of times, during a predetermined timeframe, or the like. During the execution of a test, the verification tool 140 may report coverage of coverage events. The verification tool 140 may update the coverage events repository 120. In some exemplary embodiments, another module may determine coverage, such as for example, based on log files created during execution, based on data or the like.
The disclosed subject matter may be utilized using different coverage metrics such as but not limited to: code coverage, data coverage, functional coverage or the like.
Referring now to
In some exemplary embodiments, the coverage analysis tool 200 may determine the coverage analysis based on reverse time. E.g, instead of inspecting the coverage from the beginning of the verification phase (or last reset) onwards to current time, the coverage may be inspected in a manner that reverses the time: starting from the current time and ending in the beginning of the verification phase (or last reset). The coverage analysis may take into account all information about coverage events, and may avoid selecting a sliding window on which to focus. The coverage analysis tool 200 may provide a report similar to that known in the art, such as providing progress or status report. The report may comprise information such as a portion of the coverage events that were not covered since a predetermined time. The portion of the coverage events may be an absolute number, a percentage of the covered coverage events, a percentage of all coverage events or the like.
In some exemplary embodiments, the coverage analysis tool 200 may comprise a coverage status module 210. The coverage status module 210 may be configured determine the coverage status and provide a report, or another indication thereof.
In some exemplary embodiments, the coverage analysis tool 200 may comprise a window selector 220. The window selector 220 may determine a sliding window based on the report provided by the coverage status module 210. The window selector 220 may utilize a begin time determinator 225 to determine a beginning time of the sliding window based on the report. The beginning time may be, for example, the time in which a predetermined percentage of the coverage tasks were covered, the time in which a predetermined number of coverage tasks were covered, or the like.
In some exemplary embodiments, the coverage analysis tool 200 may comprise an Input/Output (I/O) module 205. The I/O module 205 may comprise a receiver (not shown) utilized to receive, retrieve or otherwise obtain a coverage events information from the coverage events repository 208. The I/O module 205 may be utilized to provide an output to a user, such as verification engineer 130 of
In some exemplary embodiments, the coverage analysis tool 200 may comprise a processor 202. The processor 202 may be a Central Processing Unit (CPU), a microprocessor, an electronic circuit, an Integrated Circuit (IC) or the like. The processor 202 may be utilized to perform computations required by the coverage analysis tool 200 or any of it subcomponents.
Referring now to
In step 300, coverage events may be received. The coverage events may each comprise information regarding at least a last hit time. A coverage event may further comprise a first hit time, an identifier, an indication which test covered the coverage event or the like. The coverage events may be received using an I/O module, such as 205 of
In step 310, an indication of coverage events not recently covered may be determined. The indication may exclude coverage events never covered from coverage events that were covered but not recently covered. The indication may be determined by a coverage status module, such as 210 of
In step 320, the indication may be outputted. An I/O module, such as 205 may output the indication.
In step 330, a verification strategy may be determined based on the indication. The verification strategy may be determined automatically by a computerized apparatus, such as coverage analysis tool 200 of
In some exemplary embodiments, the verification strategy determined in step 330 may be performed either manually or automatically.
Referring now to
Using curve 410 information that is useful to the verification engineer may be derived. For example, coverage events represented in the curve 410 after point 430 may be deemed “easy to cover”, whereas coverage events represents in the curve 410 before point 430 may be relatively harder to cover. Separation of the “hard” coverage events into “moderate” and “very hard” may be achieved based on a second point 432. The disclosed subject matter may provide a report that in addition to the graph 400 or instead of the graph 400 shows the coverage events that are associated with each such group. The points 430, 432 may be determined based on the curve 410, such as by including a predetermined percentage of the covered events, a predetermined number or the like.
Referring now to
In some exemplary embodiments, a coverage analysis tool, such as 110 of
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of program code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
As will be appreciated by one skilled in the art, the disclosed subject matter may be embodied as a system, method or computer program product. Accordingly, the disclosed subject matter may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.
Any combination of one or more computer usable or computer readable medium(s) may be utilized. The computer-usable or computer-readable medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CDROM), an optical storage device, a transmission media such as those supporting the Internet or an intranet, or a magnetic storage device. Note that the computer-usable or computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory. In the context of this document, a computer-usable or computer-readable medium may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-usable medium may include a propagated data signal with the computer-usable program code embodied therewith, either in baseband or as part of a carrier wave. The computer usable program code may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, and the like.
Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.