No one is invincible!

Companies that sell complex and expensive equipment, usually do so with the promise of enhanced performance over their competition.  In order to deliver on these promises, companies usually gather performance metrics from their customers, and track these metrics so as to meet their performance goals.
If done properly, these performance metrics can allow you to watch for trends and spot problems in the following areas:

• Trends in the utilization of system resources. This information can be used to plan and tailor changes and   upgrades to the configuration of the system.
• Identification of stress on specific subsystems and elements of the system.
• Balance the use of system resources during peak and normal usage.
• Identify stress points on specific subsystems and elements of the system.
• Use historical data to accurately predict the effect of specific changes to the system.
• Identification of specific actions which may be causing problems with other activity on the system.
• Efficiently manage utilization levels and trends for available resources.

The use of simple pass/fail criteria should be avoided.  These levels can be arbitrary set to a level that is twice that of a worse-case scenario so as to provide a margin of safety.  As time passes performances levels can gradually slip until they hit that level. Since the level was arbitrarily set, it is not clear when the actual problem occurred that allowed the system to reach this unacceptable level.  Therefore, when problems occur, it is essential to have performance data from before and after the incident to narrow down the cause of the performance problem, and to find an appropriate resolution.

However, customer databases can be inflexible and hard to use.  Data input might be done manually.  Reports can be time-consuming and tedious to generate.  As a result, customer performance data can take hours to research. Instead, the input of the data needs to done automatically and routinely.  Historic tracking of performance data cannot be done if data is spotty or insufficient.  Organization of collected data needs to be flexible and dynamic.  Archives that are too rigid quickly become tedious to use.

Report generation also needs to be automated.  Real-time dashboards should be available to present data in a quick and customized manner.  Reports for important customers might look different than reports for non-critical customers.   Researching customer data should take a matter of minutes, not hours.

Most organizations use project health as a way to manage trade-off between development time, cost, and quality. Various measurements are used to access the performance or effectiveness of a project team. Without measurements they would be subjective guess. However traditional methods like measure test coverage or measuring design stability are falling short because of fundamental changes that are occurring in electronic design. Here are some of the top trends in design and the impact they have on tracking project health.

1. It’s an intelligent mesh, not just a flow.

There is no longer a sequential design flow in electronic design: there is architectural, layout, and verification exploration. Layout and verification often start before architecture and implementation details are finished. IP blocks further parallelize, loop and interconnect the project flow. With a mesh development, the next extremely difficult question is resource allocation. How do you apply people, machine, licenses, and testing runs? What stage needs more resources? What will be the resource impact on development time, cost and quality?

2. Global teams

Expect more “follow the sun” design work. However for distributed engineering groups, it can be difficult to manage and track progress. Global teams must share data, status, and issues. Project team can no longer rely on self report to get a picture of where things are. Teams must find ways to automate sharing status and escalating and brainstorming on issues.

3. Multiple dimensions of analysis

Engineering is managing constraints and trade-offs. A workable plan must balance cost, performance, schedule and quality to develop a useful design. Whole teams are dedicated to power, performance, fault, routing and timing. Each group needs to communicate and escalate optimization and trade-off decisions.

4. Runtime jobs growing faster than transistor count

Sure, transistors counts are growing but the jobs running analysis are exploding. Teams routinely run many hundreds of builds and automated tests. Expect to see this trend continue. How are you going to manage it? Are you getting best use of software licenses and CPU resources? Are you spending more on licenses than hardware?

5. Summarizing and Analysis of More Testing:

As system size grows, manual testing typically cannot keep up. So everyone is turning to test-data generation. But test generation–regardless of the framework used–requires manual sorting and analysis of the test results. Developers must wade though an ocean of data looking for warnings and errors. The result is that–if you don’t automate the data collection and analysis–you will spend all your time grepping log files. If you add more CPU resources and run more jobs, can you analyze the output by hand? You must automate the analysis of the results as well after a certain point or further test generation is pointless.

Why Achilles Test Systems?

Achilles Test Systems’ DV Notebook is a flexible repository for collecting, analyzing and summarizing a large amount of test data from multiple sources. We excel in multi-vendor environments. The DV Notebook’s visualization enables users to sift through test results, categorize failures, and highlight areas of concern. It draws attention to the key points while retaining the relevant context and the lower level details.

The DV Notebook can parse log files and extract key pieces of information such as test name, test description, time of execution, duration of test, key parameters, random seed values, error messages, and pass/fail status. Built-in templates track verification testing and manage regression suites out of the box. They offer a starting point that can be customized to address your exact situation, extracting the precise data you need and working with the tools in your flow. Collecting key information and storing it in an organized and easily accessible dashboard saves time, avoids errors, and facilitates re-use.

About Our Organization
Achilles Test Systems products and services enable development teams to correlate results across multiple project data files, cutting debug time in a collaborative development process. Project status is always available and up to date with automatically generated tables and charts to highlight trends from historical data. Every member of a global team just needs a web browser to contribute insights and to access an integrated visualization of seed tracking, test data, and source-code revision history.

Spreadsheets are great.  We all use them to organize our thoughts and experiment with what-if calculations.  In talking to different engineering companies in the Boston area, we have realized that spreadsheets are being used to do real engineering.  Sometimes, a spreadsheet is the perfect tool for our needs.  Other times, it starts to grow until we realize that we have surpassed the limits of the spreadsheet.

The examples of exceeding the limits of the spreadsheets tend to come in two forms.  Either, the spreadsheet is being used as an ad-hoc database or it is being used to perform a relatively complicated multi-step calculation that might be better served by a script or a program.

When the spreadsheet is being used as a database, there are a few clues to when it is time to consider moving data into a more powerful tool.  At a high level engineers will be better served by a simple database in place of the spreadsheet if any of the following are true:

• If the spreadsheet has multiple authors
• If data searching is required
• If it has a combination of calculations and data storage
• If it requires different numbers of different types of data

Similarly, there are a few clues that an engineer might look for that indicate that his spreadsheet would probably save time and money if it were in the form of a simple script for computation:

• If the same calculation is replicated on different worksheets in order to consider different starting values
• If many versions of the spreadsheet are saved with special names to indicate the what-if case that is being considered
• If the data are being selected from indexed lists of pull downs

Migrating ad-hoc databases and multi-step calculations out of excel and into other tools or scripts is usually pretty easy. 

In some cases, useful excel functions are leveraged that a user does not want to re-program or replace from standard libraries.   Most scripting languages offer ways to access Microsoft excel functions directly.  Visual Basic, Ruby, and Python all have such extensions.  In linux, there are also C libraries for the open-office platform that can be used.