Archive for the ‘DVCon09’ Category

Employing Boyd’s OODA loop in design and verification

Monday, March 2nd, 2009

Created by military strategist and USAF Colonel John Boyd, OODA stands for Observe-Orient-Decide-Act.  Quoting from Wikipedia,

According to Boyd, decision-making occurs in a recurring cycle of observe-orient-decide-act.  An entity (whether an individual or an organization) that can process this cycle quickly, observing and reacting to unfolding events [...], can gain the advantage.

The speed and agility of the decision-making loop is key.  The greater the body of information that can be brought to bear quickly to orient one’s decisions, the more productive each action will be.

Different team members need different information to orient themselves.

  • Verification engineers need to observe:
    • which tests are passing and failing
    • who is doing checkin operations
    • what is the historical behavior of each test
    • which tests are most likely to detect a bug at a given point in time
  • Design engineers need similar metrics:
    • which RTL blocks are most likely to see timing failures
    • which lines of code have changed the most over time
    • which lint warnings represent the most risk in each block
  • Managers tend to observe trends and comparisons:
    • how has timing improved over the last few weeks
    • how many bugs are being filed
    • how many tests are passing and failing per week
    • which blocks have the most: warnings, timing failures, bug reports, code changes

These lists can be long, but the genius of the OODA model is that it applies to everyone in an organization.  The DV Notebook is designed to accelerate all aspects of observing and orienting unfolding events in a design and verification environment.  Information is brought together from many sources into customized dashboards.  This provides proper context for decisions and actions.  Many of the most common actions can be simplified to a single mouse click to initiate debug, file a bug report, rerun a simulation, or browse detailed reports.

Posted in DVCon09, Engineering Practices | 5 Comments »

Conventional wisdom and the “Intelligent Test Bench”

Friday, February 27th, 2009

During a conversation this year at DVCon 2009, Gary Smith classified the Dynamic Play List features of the DV Notebook as belonging to the classic definition of an Intelligent Test Bench.  Specifically, the ability of a test bench to prune unproductive work and avoid wasteful re-run of tests that are not going to find bugs.

Brian Bailey has written a compatible definition of an intelligent test bench in the following quote:

An intelligent testbench can either replace or enhance existing simulation based or formal verification methodologies. Constrained random generation techniques manage to create huge quantities of stimulus, but at the end of the day they have difficulties both with closure (achieving the desired verification goals) and secondly with efficiency (huge server farms required). An intelligent testbench can help either by determining efficient stimulus sets or by finding ways to reach difficult to reach coverage points.

In spite of this, one still needs to be clear about many aspects of intelligent test benches.  Most EDA products that belong in this category are focused on optimizing the outcome of a single test.  The vast majority of those are concerned with a single execution image.  This is in contrast with the idea of looking at a test population to selectively run those tests that are most likely to yield results.  (also see Do tests depreciate?)

There are several different scales to consider:

  • Optimizing a single execution to target a desired outcome
  • Optimizing runs of a single test to meet particular coverage goals
  • Optimally selecting the members (tests) within one or more regression lists
  • Optimizing test populations over the life span of a project or on subsequent projects

In part, the problem may be nomenclature.  The last two bullets above probably relate more to wisdom than to intelligence.  Wisdom is well rounded knowledge accumulated through time and experience.

The goal of the DV Notebook products is to enhance existing simulation based verification by determining efficient stimulus sets.  This entails looking more broadly at all available data (e.g. bugs, bug fixes, code check ins, lint results,…) to determine how the code is evolving, how the project is progressing.  This gives a picture of status and overall project health.  It is from that basis that cost reductions in verification can be realized.

Posted in DVCon09, Verification | 2 Comments »