Bughunta Work The Arc of Quality

The Arc of Quality

Measuring the Effectiveness of the Testing Process

If you are trying to manage a testing process, 3 questions will continually worry you:

  • How do you know if your testing is effective?
  • How do you know fixes are being prioritised properly?
  • How do you know when to stop testing?
  • The answers are fairly easy:

  • Because you are trapping the worst faults early.
  • Because the worst faults are being fixed early.
  • Because the system is of the "right" quality.
  • As we know, these are answers, but they're not exactly helpful. What we want is some "ideal" against which to measure the progress of our testing. I believe the answer may lie in the effective use of your bug tracking process and the charting of a simple graph, which I call The Arc of Quality.

    At a recent job interview, I was asked to role-play as the vendor of a bug-tracking tool. I had 10 minutes to prepare some spiel on what it could do and why this company should buy it. While I was feverishly racking my brain, I started to scribble down a graph, which showed one of the possible reports from the tool I had been asked to invent. It was one of those Eureka moments that keep me interested in testing, despite the sometimes low opinion developers and bean-counters have of our art. The thing about good ideas is that they are very simple. So simple in fact, that I am sure someone else has already thought of this and I just haven't read enough books yet. I apologise in advance if I am teaching your granny to suck eggs, but I think it's worth sharing.

    Without further ado, this is the Arc of Quality:

    The Arc of Quality

    Well, I said it was simple!

    The Arc of Quality defines a timeline through the testing process. It indicates the number of errors outstanding and the average severity of those faults. I imagine that you will use a discrete representation of the severity of your faults, i.e 'Category A', or 'Severity 1', but there are statistical tricks you can use to convert these to a meaningful continuous value.

    Ideally, you want to be finding the worst errors as early as possible, so the Arc starts at the top of the graph. Following the initial panic of fixing the high severity faults, the quality of the system should improve, indicated by the downward movement of the arc.

    For a while, the developers will be playing catch-up as the testers work through their test cases. There comes a point (we hope!) when fixes are applied faster than new faults can be found. This is shown when the Arc reaches its furthest point East and begins to travel Westwards. The number of outstanding errors should steadily decrease after this point.

    Eventually the Arc approaches the origin as the number and severity of outstanding faults approaches an acceptable level. At this point you can stop testing. The quality of the system is "right" and the risk of releasing it is worth taking.

    Note, you never reach the origin. It is a business decision when the Arc is close enough. This does not occur at a particular point, but when The Arc of Quality meets The Isoquant of Acceptable Risk (this is starting to sound like bad psychology!):

    The Isoquant of Acceptable Risk

    The Isoquant of Acceptable Risk (the green curve) may be familiar to students of Economics. It shows all the points where the combination of the number and severity of faults is regarded as representing an equal level of risk.

    You might define the risk very simply as the cost of fixing faults. In this case, the Isoquant of Acceptable Risk indicates that it costs as much to fix a few major faults as it does to fix many minor ones.

    The position and gradient of the Isoquant of Acceptibilty is a purely subjective matter (now does it sound more like economics?) but it should slope from top left to bottom right and it should not turn back on itself.

    So far, so simple. Now let's see what the Arc actually tells us.

    It is NOT going to give you specific values to compare the progress of your testing to (not immediately anyway - you may get to build up a baseline over time), but you can get a good deal of information from its shape. To obtain the Arc that reflects the progress of your testing, you need to pull up the relevant figures from your bug tracking process (or tool, if you use one). A daily tally of the number and average severity of outstanding faults should be sufficient to provide an accurate measure and enable you to react quickly should things (literally) take a turn for the worse.

    We have already seen the ideal shape for the Arc of Quality, what are some of the alternatives and what problems do they indicate?

    Going Ballistic

    Going Ballistic

    This is about as bad as it gets. Both the number and severity of outstanding errors are increasing. This can indicate a couple of things:

  • The quality of the system is actually getting worse as more fixes are applied. Solution: Halt testing (this is one for the developers to sort out).
  • Are you actually retesting the fixed version? Solution: tighten up Configuration Management.
  • Your testers are running tests in the wrong order. They are finding the severe faults later rather than sooner (or they have changed their minds mid-stream about how to assess severity). Solution: Have agreed levels of severity and ensure test cases are reviewed by experienced testers before they are slotted into the plan.
  • The Cup of Sorrow

    The Cup of Sorrow

    The shape can occur for the same reasons as Going Ballistic, but strongly indicates a major fault introduced as the by-product of a fix.

    Solution: Re-fix and regression test the component "fixed" on the day the Arc began to turn.

    The Slippery Slope

    The Slippery Slope

    The testers have found the worst faults early on, but it does not look like the developers are keeping up with the fixes. Since this Arc has the same shape as the Isoquant of Acceptable Risk, the two will never meet.

    Solution: get more developers on the case.

    These are the 3 main alternative shapes (they're just rotations of the original). The Arc you plot from your real-world figures may well snake about, but at least you know what any changes of direction could indicate and can take action to get back on course.

    So there you have it, the Arc of Quality: a simple but effective addition to the QA team-leader's or manager's armoury. If you can think of any more alternative shapes, indications or solutions, please drop me a line and let me know.