No one is invincible!

Every test can be analyzed in terms of its costs and values.

The costs of a test are mainly measured in time values such as the time it took to write the test or the time it takes to run the test.   Hardware cost factors such as special requirements for simulation memory or emulation hardware may be important.  Another measure of the cost of a test is the cost to check the test result in the event that automatic checking is not being performed.

A test’s value can be looked at many ways as well.  If writing or running a test makes engineers think differently, then that has value.  However, the most important value of a test is if it finds bugs.

In a recent conversation at DVClub, the topic turned to the time-value of  tests.  Our host pointed out that the value of a test does not stay constant over time.  Each test goes through a prime of life where it finds most of its bugs.  After those bugs are understood and addressed, that test begins depreciating.

Some tests lose their value in more drastic ways.  When tests are specific to old versions of the design or when they require configuration sequences that are no longer supported, they immediately lose all of their bug-finding value.

Regression testing is an important function to insure that new bugs are not introduced.  As a result, many tests always retain some minimal value.  However, a test that has never found a bug after 1000 executions is not likely to find one in a stable design.

What everyone seemed to agree on is that the natural depreciation of tests over time requires  dynamic test list management to stay on top of which tests should be run most often.  Some of the big micro-processor companies are at the forefront of managing verification resources to maximize the productivity and minimize the cost of their compute infrastructure.  This is partlly due to the fact that their design point has many common components year after year, so they have many product cycles to improve their environment.

With the DV Notebook, this kind of corporate environment is not required to analyze and manage the time-value of tests.

I’ve got great news.  We’ve gotten approval for $100,000 to grow the cpu farm and buy more simulation licenses.  I think that we should be able to pull the schedule in by at least a month with this.

- Boss

We’ve all been in a similar discussion.  Management is looking for any way to increase productivity.  Random testing seems to create higher confidence with more instances of the same test .  The problem is, sometimes growing the number of simulations by 10x grows the debug effort too.

This particular case aside, each generation of ASIC requires more and more verification.  Each project therefore needs to run more simulations per day than the project before.  So, how can we scale up CPU usage year after year?  In short, the answer is organization.

Several companies in the Boston area employ SQL-based regression management tools to filter, sort, and organize testing lists and results.  The effect is CPU farms with hundreds or even thousands of CPUs running simulations.  A relatively small team can efficiently utilize these CPUs without wasting too many cycles on repeat errors or useless re-runs.

When tests fail, not all of them need to be debugged.  Some failures are debugged, others are stored in the database for re-run following the fix of bugs that are already filed.  One project that we know of required that every failing test and seed be re-run wihout failure as a pre-condition for tape-out.