1. Why and How to use KPI Dashboards effectively
2. Dashboard Visual Design Best Practices
3. Building a customized IntraStage dashboard

If you would like to inquire more about how IntraStage can help to make your KPIs more visible and manageable please contact us.

A great resource and which we believe is a gold standard on Dashboard design is Stephen Few’s book “Information Dashboard Design”.

To make his point, he references the book, “The Challenger Launch Decision”, by Diane Vaughan, and the events that led up to the decision to launch the Challenger in 1986. NASA had contracted a consultant regarding the cold temperature conditions (between 25-30 deg F) under which the Challenger would need to launch, and whether the O-rings might have an issue. Interestingly, the consultant gave the opinion that it may not be safe, due to the shuttle never having launched in such cold temperatures, and that the O-rings might fail and cause an explosion. The evidence cited was that the O-rings were often damaged at launches below 53 degrees. During a three hour meeting, NASA engineers and managers argued amongst themselves about what to do, since there were also O-ring failures at 70 degrees and above, and there wasn’t any clear evidence that a launch would be unsafe.

Interesting enough, it turns out that there WAS clear evidence available. It just wasn’t easy to visualize with NASA’s technology at the time. However, any IntraStage customer would tell you that what was needed was very easy to do in a tool like IntraStage. What they needed to map was a number of O-ring failures vs. launch temperature. If they had produced this graph, they would have seen that EVERY launch below 66 degrees showed O-ring failures, and this pattern mitigated substantially above 66 degrees. But, as Collins’ summarizes, “no one laid out the data in a clear and convincing visual manner, and the trend toward increased danger in colder temperatures remained obscured throughout the late-night teleconference debate.”

The O-ring Task Force stated, “We just didn’t have enough conclusive data to convince anyone.” But the evidence was there, hidden in the test data, and NASA didn’t have the tools to visualize it. The visualization would have saved the lives of seven people, including Christa McAuliffe (the first civilian astronaut).

That is the thing about good visualization tools. You never know exactly what report you’ll need, and you never know when you’ll need it. It might be an urgent teleconference where you are making a critical bet that, if wrong, could lead to a product recall or even loss of life. In these occasions, what you need are tools that allow you in real-time to deep dive on the test data, pivot on all kinds of scenarios, and visualize trends.

I’ve done software architecture on about 20 different life test projects in the last four years, with more than 15 different large OEM’s, in five different industries (Medical Device, Cell Phone, Semiconductor, Aerospace Component, Consumer Electronics). While they are all different, there are certain elements that are always the same.

The hardest thing is to figure out what specifically to measure, how to identify when something is a red flag, and how to identify when something should be called a FAILURE. Once things are identified as failures in a life test, everything gets easy. You plot the failures vs. the simulated years of life where the failure occurred, often using a Weibull plots, and this information helps you predict your product’s quality. If you do this for 10 or 20 units of the same model, then you can more accurately predict how any given product (with this design and production process) will perform during its life.

As I said, the easy part is plotting the stuff after you have a failure. The hard part is figuring out what to call a failure. Especially for a radically new product design, where you don’t have real-world product failure data to look at (for example, BMW has many years of real data for how often their drive trains have failed by customers). Once you have determined what constitutes a failure, you can figure out how to measure for that instance and then much of the measurement and determination of failure can be automated with software and instruments.

Back to the root issue – what is a failure? In the case of the consumer product I’m dealing with now, one idea we are implementing is to have a “20 questions” kind of diagnostic for operators. For example, as operators monitor the products during the four month testing, they may see certain things visually. When that happens, we want the system to track those issues. So, if an operator sees the display go blank, we want them to be able to not only report that the screen went blank, but also take certain steps to see if the screen will come back. We want to track the steps they took, and track if it worked or not. A design engineer can then review this data and decide if the issue constitutes a part failure that will show up on a Weibull plot.

So, in these cases, trying to enumerate all of the possible issues (and questions to ask when those issues arise) is a real challenge. When done right, the software and corresponding work flow process, can successfully find those “needles in the haystack” and feed the info back to R&D, therefore improving product quality and customer satisfaction.

An engineer’s job is to solve problems, to innovate. So, coming up with a standard is fun and challenging. Using someone else’s standard often is not. There is a sense of being a “cog in the machine”. Time is spent learning someone else’s standard and how to apply it, which is like “being told what to do”. Creating a standard is a problem to solve. You are solving that problem as you create the standard. Using someone else’s standard is like being told the problem and immediately being told the solution, without the joy of solving it yourself.

This is why I believe that the most experienced test engineers love standards. Because the experienced ones have already gone through the process of solving various challenges, and so they understand the problems and are open to seeing different solutions.

So, the “hate” comes when an engineer doesn’t get a chance to solve the problem on his own. The “love” comes when the problem has been solved, and the engineer wants to see the “best” or “more perfect” solution. Or simply to automate that problem because she has moved on to another challenge. Because while an engineer loves to solve a challenge the first time (and hates it if someone gives her the answer too soon), engineers also hate solving the same problem over and over again.

At IntraStage, we are constantly trying to make better standards for test data and test configuration. Our team has worked in this space a long time, and we have worked with hundreds of test engineers. While our goal is to create a world-class standard, we are also trying to make it fast and easy to train engineers in the use of the standard. We also have to evolve the standard over time, as test engineering continues to increase in complexity. Finally, we have to leave hooks for expansion on our standards, because every customer has custom details that the standard has to account for.

”Dashboards” as a concept recently became popular in the last 7 years. These dashboards are computer-based visual indications of Key Performance Indicators (KPIs) of metrics within an organization and are now used by Executives, Managers and Staff to track everyday activities to make better decisions. For example, a Manufacturing Operations dashboard would show such KPIs as Yield, Failure Paretos, Equipment utilization etc.

The evolution of dashboards is an interesting one and can be traced back as far the first century AD. For an interesting read on “data visualization” I would recommend Stephen Few’s book “Now You See It”. Historically Data Visualization has followed an interesting path:
-2nd century: first use of tables to organize numbers
-18th century: first graphs and charts were “invented” on paper
-1980s: first widespread use of visualizing data through Bar charts, line graphs, pie charts etc

The 1980s saw Data Visualization come into its own primarily because of high speed computers and rich graphics interfaces. What is happening now in the first part of the 21rst century is the ability to store massive amounts of data in a cheap fashion and being able to access that data conveniently through the internet. This has directly given rise to the use of Dashboards which can quickly aggregate and provide Users directly relevant KPIs out of these massive amounts of data.

The question for most users is: What are relevant KPIs that I need track? A good rule of thumb to understand your KPIs is to understand how your or your group’s performance is measured by the organization…if there is some raw data that supports that performance then a dashboard could be useful. IntraStage has developed multiple dashboard views for Test, Manufacturing, Quality and R&D environments. If you would like to contribute your views on what KPIs are relevant to your job, we would love to hear them…please comment on our LinkedIn Discussion post.

Happy dashboarding!

The financial repercussions are relatively obvious, but what’s less obvious are the root causes for these device failures, especially since they come from several different manufacturers.  What kind of evaluation processes would a quality engineer have to run through in order to determine the reason for failure?  How would his or her team have to prove the strength of their design, particularly since 2010 the FDA has been focused on preventing problems with infusion pumps due to a perceived deficiency in design and engineering?  What kinds of test data would they have to show to the FDA to validate the quality of their product, and more importantly, ensure patient health and safety?

If you’re going to be at NI Week, come by and visit us at booth 526-if you haven’t used our software before, take the opportunity to test out our fully functional demo software.  If you’re an existing customer, come by and get some of our giveaways.  And we’ll be announcing a new addition to the IntraStage software catalog, so check that out and the accompanying iPod contest.

In the next few years, the ability to provide on-demand test data for review and analysis will provide contract manufacturers with a more compelling partnership proposal, especially in comparison to other CMs which don’t offer their customers with visible, clear test data.  Those that do offer data and teamwork to the customers will have the upper hand in proving the quality and efficiency of their labor, equipment and processes to prospective and current partners.  To all our CM managers: how much nonproductive time do you spend aggregating data for your OEMs?

Maybe if the OEM customer received the quality data on those incoming parts, they could do their own analysis of where things are breaking down between incoming quality and what they are actually testing. An interesting article on the effect of supply chain components from the Japan Earthquake is here: http://www.ebnonline.com/author.asp?section_id=1096&doc_id=207372&itc=ebnonline_gnews

Many large CMs already offer these types of tools to their OEMs; however, integration of software often requires significant time and expense, and few CMs can afford the cost.  When these tools are in place, the transparency can lead to yield improvements and faster detection of issues and root causes.

We discuss this issue over at a LinkedIn discussion group here.  We encourage anyone who reads our blog to join that group and contribute to that (and other) test data management discussions.