Patent Application
20160342720
The present invention relates to the field of hardware design debugging. The present invention more particularly relates to identifying hardware design revisions that are responsible for hardware failures exposed when the hardware design is subjected to verification.
A typical hardware design cycle starts with a specification document describing the intended functionality of the design. The specification is used to create the hardware design, which is typically implemented using a Hardware Description Language (HDL). HDL designs are most commonly implemented at the Register Transfer Level (RTL) using Verilog and VHDL languages, which are HDLs used in electronic design automation to design digital and mixed-signal systems, such as field programmable gate arrays and integrated circuits. The specification is also used to determine the expected behavior specified by the verification environment.
Along these lines, verification is the process of determining whether errors exist in a hardware design. Verification can be performed by using testbenches that apply stimulus to the design via diagnose input vectors, and simulation tools or formal verification tools. Verification forms a major bottleneck in modern hardware design cycles, consuming up to 70% of the design effort. Of the time spent in verification, a major part is dedicated to the task of hardware design debugging which is reported to take half of this time.
Hardware design debugging is the process of locating errors in designs after verification methodologies and techniques determine the presence of such errors. Today, debugging is majorly a manual task where the verification engineer typically uses the erroneous response of the design, the expected behavior as stated by specification and the diagnose vectors, to determine what design components, usually HDL statements and/or signals, are responsible for the erroneous behavior. These components whose values appear to be inconsistent with those of the specification are referred to as suspect components or simply suspects.
Tools that automate debugging have been introduced in recent years, such as OnPoint by Vennsa Technologies, Inc. [1], and Verdi by Synopsis, Inc. [2]. Many of these tools use simulation-based techniques, such as path tracing, or Automatic Test Pattern Generation (ATPG) methods [3]. Other tools employ formal engines such as Binary Decision Diagrams (BDDs), Satisfiability (SAT) and Quantified Boolean Formulas (QBF) [4, 5]. The above tools automatically determine and return suspect components in the design. Among the suspects that are identified by these tools, some may be equivalent, in that they produce the same erroneous behavior under all diagnose input vectors. There may also exist suspects returned that are false positives, that is, they can correct the particular diagnose input vector but they cannot correct other diagnose vectors. In that sense, false positives do not actually correspond to an error present in the design. In most automated debugging tools the actual design error will be included among the suspects retuned by these tools. Formal tools guarantee to find the actual error and equivalent ones due to their exhaustive search.
The present invention described here can work alongside the aforementioned tools, but it can also operate without them as a stand-alone method.
Verification can be performed either on-line or off-line. In on-line verification the engineer analyzes the design through model checking or simulation to discover an error trace (sequence of stimuli) that exposes a particular single failure. Then, debugging commences trying to localize the error source of that particular failure. The error source can reside within a design component, design module, or it can be a design signal. Once the error source is localized, the engineer(s) need(s) to identify the error source and perform a change (correction) that will remove the failure exposed by verification.
On the other hand, off-line verification, often referred to as regression verification, is usually performed when the design undergone multiple revisions (modifications) before the last verification stage. These design revisions along with relevant metadata (author of revision, time of revision, author's comments, purpose of revision, etc.) reside in a version control system, such as Apache Subversion (SVN), CVS, and Git. A version (or revision) control system is a computer program that tracks and manages changes to files, documents, and other textual information that comprise the source code of the design under verification. Regression applies a large number of diagnose vectors (or tests) to exercise a majority of the design functionality, since multiple revisions may have affected many design elements. Performing regression verification today is mostly an automated process. However, it is a time-consuming process often performed overnight or over the span of multiple days, and it usually results in multiple diagnose vectors failing, where each failing diagnose vector indicates some functional failure. When the verification engineer(s) examine(s) these failing diagnose vectors, hardware debugging is usually performed in a coarse-grain manner by parsing simulation logs, analyzing simulation waveforms and many error messages. Candidate revisions that may have introduced design errors are discovered and distributed to the appropriate verification or design engineer(s) for the subsequent debugging. Due to the excessive amount of information that needs to be analyzed after regression, and because multiple engineers may be working on the same design, the process of identifying candidate revisions and distributing them to the proper engineer(s) needs to be accurate. If it is not, this results in significant costs and delays, and in multiple debugging iterations. While regression is mostly automated, identifying candidate revisions that may contain design errors is a predominantly manual and resource-intensive process, often performed by one or more verification and/or design engineers.
The present invention can operate in both of these on-line and off-line verification modes.
Whether verification is performed on-line or off-line, the verification engineer attempts to correct the error(s), while being guided by suspects that are returned by debugging. When all engineers are done with corrections, verification is rerun to ensure that no diagnose vectors fail. It becomes apparent that it is of great importance for the engineer(s) to perform debugging exactly on those revisions that contain errors whose correction will make all previously failing vectors to successfully pass in the following verification run. Therefore, it is important to determine which suspects in the returned set are a false positive or an equivalent suspect, and rank all suspects based on which ones are most likely to be the actual design error(s). This not only reduces the number of suspects that need to be examined by the engineer(s), but also offers better estimates as to what revisions contain actual design errors or not. It is to be noted that identifying and correcting actual design errors is almost always preferable to correcting equivalent suspects in an industrial context so to preserve most of the existing engineering effort already invested in the design.
Recent work in [6] has improved the process of prioritizing suspects for debugging by implementing machine learning engines that determine which engineers are best suited to rectify a failure. This work tries to cluster (i.e., bin) failing diagnose vectors and sets of suspect components according to their effect on the functionality of the design, but it does not take into consideration information contained within design revisions as it ranks the suspects.
What is not provided by current methods is a means to parse information from a revision control system in order to: (a) rank suspect locations based on the likelihood of being actual error sources or not, (b) identify exactly those revisions that are likely to contain actual design errors and ought to be analyzed with high priority doting debugging.
A detailed description of the embodiments is provided herein below by way of example only and with reference to the following drawings, in which:
In the drawings one embodiment of the invention is illustrated by way of example. It is to be expressly understood that the description and drawings are only for the purpose of illustration.
The present invention provides a method, system, and computer program for ranking suspect components based on their likelihood of being actual error sources, and identifying revisions that are likely to contain actual error sources. The invention may be included as part of a complete verification solution or as a stand-alone tool.
The method requires an initial set of suspects. These suspects can be provided by the engineer, an automated debugging tool, or both. These tools can be based on, but not limited to, simulation, path tracing, ATPG, BDDs, SAT, and QBF techniques.
The method involves the application of either an analytical or statistical process on the suspects that are collected, to identify suspect components that are likely to be actual error sources. Those that are identified as such are assigned as high rank based on a weight function. The weight function assigns a low rank to those suspects that are less likely to be actual error sources.
The proposed method also involves the use of a program (parser) to parse design revisions and/or revision metadata available in the version control system associated with the design that is undergoing verification and debugging. The method is not limited to any specific type of version control system. The information that is collected from the parser program is used by either a statistical or analytical system to classify (determine) which revisions are most likely to have introduced design errors.
The method further involves an analytical system that matches ranked suspects to classified revisions. This process filters out revisions that are guaranteed not to have introduced actual design errors. It also identifies, based on the matching results, which revisions contain suspects of high rank, and are therefore more likely to have introduced actual design errors, and returns these revisions in the form of a list. Every revision in the list is also ranked based on the ranks of suspects that are present in that revision.
Before explaining at least one embodiment of the invention in detail, it is to he understood that the invention is not limited in its application to the detail of construction and to the arrangement of components set forth in the following description or illustrated in the drawings. The invention is capable of other embodiments and of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting.
The present invention provides a method, system, and computer program for ranking suspect components in a hardware design that fails verification, based on their likelihood of being actual error sources, and ranking design revisions based on their likelihood of having introduced actual error sources.
Any module or component described herein that reads/executes instructions may involve or have access to computer readable media. These include volatile and non-volatile, removable and non removable computer storage media, and removable and/or non-removable data storage devices, such as, for example, magnetic disks, optical disks, or tape. Computer storage media may be implemented in any method or technology for information storage, such as data structures, computer readable instructions, or other data. Examples of computer storage media include ROM, RAM, EEPROM, flash memory or other memory technology, CD-ROM, DVD or other optical storage, magnetic cassettes, magnetic tape, magnetic disk or other magnetic storage devices, computer instruction signals embodied in a transmission medium that may include a communication network, such as the Internet, or any other medium that can be used to store required information and that is accessible by an application, module, or both. Any application or module described herein may be implemented using instructions that may be stored on such computer readable media.
When verification of a hardware design identities the presence of errors in the design, then debugging commences and aims at determining design components (gates, signals, HDL statements, and/or lines) that should be considered as potential errors. These components are referred to as suspects, and each suspect may or may not be an actual error source. The process can be done manually by assessing simulation waveforms and/or monitoring and analyzing errors logs. It can also be done automatically by means of tools that have been proposed in prior art. Specifically, the methods proposed in [3-5], and the methods disclosed in U.S. Pat. No. 8,881,077 to Veneris et al., and in U.S. Pat. No. 8,751,984 to Veneris et al., automatically identify suspects and return these to engineers to aid in their attempt of correcting the design. The method disclosed in U.S. Pat. No. 9,032,371 to Daniel Hansson et al. reports revisions that may have introduced design error(s) without ranking them. More specifically Hansson iteratively tests (i.e. simulates) each revision to see if it corrects the diagnose vectors; then reports back with no prioritization on the revisions, nor ranks according to what is the actual error, nor is Hanson's iterative testing based on statistical methods as outlined herein.
The suspects or design revisions that are returned by above means are not ranked automatically with respect to the likelihood of being actual error sources or containing actual error sources, respectively. The problem of prioritizing (by means of a ranking engine) suspects and design revisions according to the above likelihood is addressed by this invention.
The ranking engine may include or be linked to means of accepting user input, and providing textual and/or graphical output to the user. The engine may be provided with one or more inputs from an automated debugging tool and/or from a user in the form of suspects. The automated debugging tool may be OnPoint by Vennsa Technologies, Inc. or Verdi by Synopsys, Inc. or any SAT-based, QBF-based, BOD-based, simulation/path-trace based tool [5].
The engine may also be provided with one or more inputs from a version control system and/or from a user in the form of design revisions. The inputs received from a version control system may be linear or non-linear. Linear inputs correspond herein to a plurality of design revisions when branching is not present. While non-linear inputs refer to a plurality of groups of revisions referred to as branches, when branching is present. Branching is a commonly used methodology to duplicate code in order to isolate code development for a particular feature or bug fix and allow it to be performed in parallel by different developers. Whether branching is present or not, the inputs received from the version control system are referred to as the revision history of a design. As illustrated in
The engine may be further provided with one or more inputs by a user in the form of one or more parameters. These parameters may include the user's estimation of how many errors are present in the design and/or the user's estimation on which suspects or revisions are more likely to be actual error sources or contain actual error sources, respectively. The engine can, however, operate without any of said estimation parameters provided by the user.
In one embodiment of the present invention inputs are given in the form described above, with inputs from version control systems restricted to linear inputs. In this embodiment, the ranking engine may perform the following three tasks, which are also illustrated in
It is to be understood that the engine may perform tasks 403 and 404 above either in parallel or sequentially. However, both tasks 403 and 404 always precede task 405.
In another embodiment of the present invention inputs are given in the form described previously, where inputs from version control systems can be non-linear.
It is to be understood that the engine may perform tasks 703, 704 and 705 above either in parallel or sequentially. However, tasks 703, 704 and 705 always precede task 706.
[1] www.vennsa.com
[2] www.synopsis.com
[3] M. Abramovici, P. R. Menon, D. T. Miller, “Critical path tracing—an alternative to fault simulation”, in Proc. of Design Automation Conference, 1988, pp. 468-474
[4] A. Smith, A. Veneris, M. F. Ali and A. Viglas, “Fault Diagnosis and Logic Debugging Using Boolean Satistiability”, in IEEE Transactions in CAD, vol 24, no 10, pp. 1606-1621, 2005
[5] A. Veneris, S. Safarpour, “The day Sherlock Holmes decided to do EDA”, in Proc. of Design Automation Conference, 2009, pp. 631-634
[6] Z. Poulos and A. Veneris, “Clustering-based failure triage for rtl regression debugging,” in IEEE Int.'l Test Conference, 2014.
[7] C. M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics), Springer, 2007.
| Number | Date | Country | |
|---|---|---|---|
| 62165544 | May 2015 | US |
