Patent Application
20150379186
This invention relates to the field of integrated circuits verification and in particular to verification by means of simulation. More particularly the invention relates to a system, method and computer program product for grading and selecting simulation tests.
Electronic chip designers develop chips of ever increasing complexity using more and more transistors. A complex chip is often called a System-on-a-chip (SOC). SOC designer teams have difficulty in verifying that a SOC will meet the specification. SOC design teams typically develop hundreds and sometimes thousands of simulation tests for verification.
SOC design teams spend many months developing and running simulation tests. The SOC design teams typically run different subsets of the simulation tests at different times. After a designer changes any line of RTL, the designer will usually run a small regression test suite to quickly verify the changes. The SOC design team will typically run a weekly test suite once a week. A considerable amount of engineering time is spent running and analyzing simulation results.
SOC design teams want to ensure that the SOC design works in all modes and configurations. They want to ensure that all lines of RTL code are exercised. To reduce SOC development time the SOC design teams also want help in simulation test selection. They want to avoid running tests that check the same functionality. The SOC design teams need to select daily and weekly test suites that can run in an appropriate time.
SOC design teams typically use simulators to generate code coverage reports. They analyze these reports and try to ensure all lines of RTL code are exercised. The SOC design teams often find “dead-code”, code that isn't needed. The code coverage reports provide a large volume of detailed information. Designers find it difficult to use code coverage to help in selecting test sets.
Electronic design automation (EDA) tools are making increased use of verification properties to reduce the verification cost. A verification property declares a condition in the design. If a property always holds true, we call them an assertion. For example, a property “overflow == 1′b0” should always hold for any correct FIFO design. On the other hand, a property can capture possible behavior allowed by the design; we call such a property a cover property. For example, a property “full ==1′b1” can be a typical coverage property on the same FIFO design. For the given above two examples, we typically write them as:
assert overflow == 1′b0
cover full == 1′b1
Verification properties can define relationships between signals on different cycles, e.g., if there is a request on cycle N, there should be an acknowledge generated on cycle N+1. User can specify verification properties in an RTL file or in a separate properties file. They are typically written in languages such as System Verilog Assertions (SVA) and Property Specification Language (PSL). Many EDA tools can parse/generate verification properties. Some EDA tools can generate verification properties by analyzing RTL statements. Other EDA tools can generate verification properties by analyzing simulation test results.
Arenta's Bugscope™ EDA tool has proven valuable in finding test coverage holes. Bugscope™ generates verification properties by analyzing RTL statements and analyzing simulation test results. For example it may note that in one test the condition “full == 1′b0” is always true. If Bugscope™ discovers that the same condition is false in a second test it treats the condition as a coverage property and generates the property “cover full == 1′b1”. The coverage property condition is inverted with respect to the discovered property to direct a simulator to check for the inverted condition. We say that the second test covers the property. Bugscope™ may note that the condition “overflow == 1′b0” is always true in all tests. In this case it generates an assertion.
Simulators can check property statements. For example, given the property “assert overflow == 1′b0” a simulator will signal an error if the overflow signal ever has the value 1. For the property “cover full == 1′b1”, a simulator can report if this property is covered in a particular test or not.
A test grading system (TGS) carries out a method in a programmable computing system in order to determine verification properties and select simulation tests based on their coverage of such verification properties. In particular, verification properties are determined from one or more of the circuit design, verification properties files and test simulation results. Then, verification property coverage is determined for each test. Verification property coverage is generated (1) by a simulator that runs tests along with the properties; or (2) by the TGS itself by analyzing simulation test results. The TGS finds which verification properties are covered by each simulation test and then recommends a subset of tests based on their verification property coverage, showing the test grading results to an end-user. The end-user should specify to the TGS what criterion to use for recommending a subset of tests. For example, the TGS's recommendation may be to select the minimal subset of tests that will cover the maximum number of properties, or the subset of tests that cover the maximum number of properties and have the minimum Simulation time. A particular subset of simulation tests is selected based on their property coverage.
In S120 the TGS accumulates all the coverage verification properties and calls them the coverage property set (CPS). The TGS accumulates coverage verification properties specified in the design, specified in property files and any coverage verification properties it derives. The TGS optionally derives coverage verification properties from the design and test simulation results. In S130 the TGS evaluates each of the simulation tests noting which of the CPS verification properties is covered by that test. The information of which CPS properties are covered by each test is called the property coverage report (PCR). In one mode the TGS checks the test simulation results to see if the coverage properties are covered. In a second mode the TGS launches a simulation for each simulation test. The simulation accepts (1) the test stimuli specific to the test, (2) the design to simulate, and (optionally) (3) coverage properties to test for coverage. If the simulator accepts coverage properties, the simulation generates property coverage data for each test. If the simulator does not accept coverage properties, the simulation generates simulation results consisting of signal values for each test. The TGS derives property coverage data for each test by checking if the signal values satisfy each property. In either case, the TGS accumulates this property coverage data for each test into the PCR.
In S140 the TGS selects a set of tests using the PCR. In one embodiment the TGS selects the smallest set of tests that cover all the CPS verification properties. In another embodiment the TGS selects the test set that covers the CPS and has minimum simulation time. In yet another embodiment the TGS interacts with a user and helps the user select an appropriate set of tests. The user views the PCR. The user indicates which verification properties he or she wishes to cover and asks the TGS to generate test sets covering those verification properties.
In one embodiment the TGS uses a greedy algorithm to select the minimum number of tests that cover all the CPS properties. The TGS finds the test that covers the most as-yet-uncovered properties and adds that test to the selected set. The TGS repeats this until all properties are covered.
The embodiments disclosed herein can be implemented as hardware, firmware, software, or any combination thereof. Moreover, the software is preferably implemented as an application program tangibly embodied on a program storage unit or computer readable medium. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units (“CPUs”), a memory, and input/output interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU, whether or not such computer or processor is explicitly shown. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit. Furthermore, a non-transitory computer readable medium is any computer readable medium except for a transitory propagating signal.
The simulator 520 is another application program running on a CPU. In one embodiment the TGS 560 controls the simulator 520. In another embodiment a user controls the simulator 520. The simulator 520 reads test stimuli 510 and a circuit design 550. The simulator 520 produces test data results 540. Under user direction through input device 502 the TGS 560 reads one or more of the test data results 540, the circuit design 550 and properties 580. The TGS 560 creates a CPS containing the coverage verification properties it found in the circuit design 550 and the properties 580. The TGS 560 optionally derives verification properties using the. circuit design 550 or the test data results 540. The TGS 560 adds any derived coverage verification properties to the CPS.
In one embodiment the TGS 560 derives a PCR by evaluating the test data results 540 and seeing which CPS coverage properties are covered by each test. In a second embodiment the TGS 560 stores the CPS as coverage properties in a property file 580. The TGS 560 or a user invokes the simulator 520 using coverage properties in property file 580. The simulator 520 generates coverage property data 570. The TGS 560 reads the coverage property data 570 and generates a PCR. The TGS 560 uses the PCR to generate a Test Selection 590.
This application claims priority under 35 U.S.C. 119(e) from U.S. provisional application no. 62/019,333, filed Jun. 30, 2014.
| Number | Date | Country | |
|---|---|---|---|
| 62019333 | Jun 2014 | US |
