Metrics and Session Based Test Management

First off, props to tester superstar and my colleague Ann Flismark who just gave a talk at SAST on the implementation of session-based test management together with James Bach and Michael Albrect of AddQ. If you happened to listen to them and found it interesting, you can read more about the everyday practicalities below (the tool we use, for example).


Current focus: metrics. There's a lot of talk in the test organization on KPIs and comparisons. Having recently moved from script-based to exploratory testing, we face a problem of bringing our old metrics to our current way of working.

By "old metrics" I mean things like

• number of testcases per requirement (planned/passed/failed)
• number of automated test cases vs number of manual test cases

and so on.

Are these metrics interesting? There is no easy answer. On one hand, counting the number of executed test cases is meaningless since a test case is not really a measure of anything. On the other hand, these numbers are deeply rooted in all levels of the organization and should be treated with a certain amount of respect.

Also, it is not uncommon for my client to be involved in project with other companies and external stakeholders. They often approach the test organization with questions like "how many test cases have you planned for this feature?" or "what is the pass rate of tests for feature so-and-so?". One way would be to slap everyone around with the hard truth that we no longer count test cases, since we have none. That also means, however, that we need to educate everyone in our way of working (which, of course, is the long-term solution). Another, more instant, approach would be to provide other, comparable, metrics. Is this something we can do? Yes and no.

The fundamental requirement on us testers has been worded as "you need to provide metrics". Having pondered this request long and hard, as well as googling for "metrics in exploratory testing", "session based test management metrics", etc (and not finding much, I might add), we have arrived at a couple of conclusions.

Primo, we collect metrics that are valuable to us and that we can
1. use to better ourselves and increase our efficiency
2. show our stakeholders once they are up-to-speed on what session-based test management is all about
These include things as time spent in session, time spent setting up test environments (and other tasks), and how these evolve over time.

Secondo, we collect and compile metrics that can be translated to and compared with traditional figures. These include, for example, requirement coverage, test coverage and test session complexity.

The thought process behind producing and displaying metrics has led to a somewhat more rigid and refined test management process - something we constantly strive for.

The Process - Improved
Each sprint starts with a day of planning where we decide to gnaw our way through a number of stories. Each story exists as a high-level requirement written by a project manager and is labeled with a number.

During the test planning, I, as a tester, go through the stories and for each
1 draw a sketch of the use cases, components involved, how it ties to the rest of the system, etc, on a whiteboard
2 get the developer (and, possibly, project manager) to detail the sketch with me, point out things I may have missed, and give input on risk areas and what to focus on while testing
3 cover the sketch with post-its, each holding a charter of 1-2 sentences e.g. "test the write-to-file throttling functionality" or "regression: test the auditing of this-and-this data to backend"
4 estimate the number of sessions I will need to cover each of the charters

I then update the relevant playbook accordingly, and bring the charters into my nifty tool - a simple text file will hold my test plan:

story;charter;no of planned sessions

9;test the write-to-file throttling functionality;1

and so forth - one line per planned charter.

When I'm ready to do a session, I visit the tool and am instantly presented with the current test coverage. I pick a charter that is planned but untested (or not tested enough), and get to work (simply clicking it will give me a session report template with the basics already filled out). As I check my reports into our version control system, the test status is updated automatically.

The tool then shows me

• what stories we cover in the sprint
• what stories I have adressed by planning tests (writing charters) for them
• what charters I have planned for each story
• how many sessions I have planned for each charter
• how many sessions I have planned for each story
• what charters I have covered with sessions
• test work left for this sprint

And, of course, the good ol' stuff like

• how much time I have spent in session (and on various other tasks of my choice)
• how that time has evolved over time
• how many sessions I have spent testing a certain component or functional area

And another thing, which is something of an experiment at this stage .. session complexity. The thoughts behind it were something like "the session report contains the steps I took during the session, such as

• went through use-case UC12 with user account F18, verified audited data in table T_11K

... couldn't each such step be translated to a scripted test case? At least the headline of a scripted test case, so we can count them ...".

So now we count them. Is that good? I'm not sure. But it's comparable. If someone asks me "how many test cases have you run for feature so-and-so?" I could say "17" if I don't feel like giving the whole here's-how-exploratory-testing-works-lecture. If that number is meaningful to them, why not? I believe in taking small steps, and making sure everyone understands why something is the way it is.

So what about failed test cases? Well, we traditionally counted passed/failed test cases based on the test run during the end of our sprint. Every failed test case would result in a bug report. If that bug was serious enough it would get fixed, we would run the test case again and set it to passed. If it wasn't a showstopper, the bug report would stay open after the end of the sprint. Short-cutting that whole process, we could translate "failed test cases" to "open bug reports after end of sprint".

As always, this is in an experimental stage. We hope to be on the right path. Do you spot anything missing? Are we ignoring important metrics or measuring wrong? How do you do it?

 Just because I like visualizing, here's a bird's eye view of my SBTM tool in its entirety:

 

... with links to previous sprints on top, followed by a summary of all the time entries I have written (setup time, test time, etc). We talked about the big red-and-green graph earlier, and the little one just below shows requirement coverage (what stories have we planned for, and whether they are adressed by planned test charters). The blue graphs show trends for certain time entries (setup time and test time being the most interesting), how the time spent is distributed among tasks and by day over the sprint. At the bottom are links to all session reports, sortable by date, covered component or area.

On Improbability

This summer I have read a magnificent piece of literature entitled The Black Swan by empirical skepticist (or was it skeptical empiricist?) Nassim Nicholas Taleb. Its subtitle is "The Impact of the Highly Improbable", and this is exactly what it deals with.

I won't post an exhaustive summary or review of the book, as it has already been done quite well by others, but perhaps a quick introduction is in order for you to follow the rest of this post.

A Black Swan, in this context, is a highly improbably event. Not impossible per se, but an event for which we are completely unprepared because it lies beyond the border of our imagination. 9/11 is a recurring example, or World War 1 - or the discovery of black swans back when man knew only of white ones, for that matter.

The author is a former trader, and so a lot of the reasoning and examples come from the world of finance. The theories are however applicable to most situations - testing, for instance.

Taleb's ideas boil down to a list of concrete advice for minimizing the impact of negative black swan events; making the swans grayer. The gist of it is make yourself more aware of - or at least less intimidated by - the "unknown unknowns", the things that you don't know that you don't know. Yet. And we have all been there, yes? A sprint that doesn't quite go exactly according to plan because we didn't consider every last dependency within the system, we found a showstopper bug just a little too late, two developers were home sick for three days, priorities were changed mid-sprint for this or that reason. But we continue to plan, and we continue to fail.

In my office, I've seen a trend in moving away from detailed time estimates. We used to sit down in the beginning of every sprint, voting for hours on each task and trying to match the available hours. Now, we have an hour-long meeting every week where we go through the backlog and vote for story-points on every new story (and revise our guesses from last week). After a few sprints, we are starting to get an idea of how many points we can churn through in three weeks.

Our morning meetings have moved from crossing off numbers on post-it notes to sharing your "gut feelings" about the tasks at hand. Neighboring teams have implemented a "fist of five" or thumbs up/down voting for their sprint tasks to give the scrum master an idea of how the work is progressing.

Considering the inaccuracies of time estimates, I strongly advocate an approach where less time is spent on guesswork. We are never going to be 100% correct in our time estimates, so let's not put too much effort into them. That hurts less when we're wrong.

How do you do plans and time estimates, and what happens when the unexpected occurs?

Exploratory vs Scripted

About six months ago, the company where I consult decided to make the switch to exploratory testing. It has been an exciting journey, and I feel very fortunate to have been there along the way - learning plenty, and hopefully contributing equally.

Recently, the discussions have circled around whether the new way of working is better than the previous. A natural reaction. For instance management, as well as others who happen to read our test reports, have started to wonder about the change of information provided regarding our test results.

This review process has spawned a few highlights that I figured I'd share with you.

Streamlining the Daily Work
Is streamlining still a buzzword? Perhaps I should just call it "cutting the crap". Anyhow, we seem to agree that the actual testing activities haven't changed all that much - at least if we compare with the best and brightest parts of the scripted methodology. Allow me to explain.

With our earlier way of working - I refer to it as "scripted testing" just to give you a feel for it - the work during a sprint followed this rough chronology:

1. discuss new feature or component to be developed with project manager, tech lead and developers
2. ponder possible test cases and risk areas on a fairly high level
3. receive a non-final version of the software from the developers and
• install it in a test environment while looking for flaws in the installation procedure, associated database scripts, etc
• start the software, make sure it can communicate with other parts of the system
• use the software, see how it works in practice, take notes of possible inputs and related outputs
• distill the knowledge acquired in 3.3 into scripted test cases with clear action-result steps
4. iterate all of 3 until we have reached a version that is "ready for test" (often around when the sprint is about to end)
5. compile a test suite using the newly created test cases from 3.4 together with an assorted selection of older test cases that cover other, possibly affected, areas for regression testing purposes
6. mark test cases as passed or failed and put the results into the test report

I guess this seems somewhat familiar to most, with a few modifications here and there. Where we are now is something more along these lines:

1. discuss new feature or component to be developed with project manager, tech lead and developers
2. draw an overview of the feature or component with all paths to other parts of the system and all connections to any actors, producers, consumers, etc that have a part in the relevant use-case(s)
3. use the overview to identify risk areas, oracles, testability deficits, dependencies, etc together with developers and architects
• compile all new knowledge into a playbook for the feature or component, formulate charters to focus the test effort
• receive a non-final version of the software from the developers and
• during one or more recon sessions, explore the installability and operability of the software, find how it works in practice
• during one or more analysis sessions, following the charters defined in 4, further explore the software learning as much as possible about it - paying extra close attention to shaky/complex/unstable/risky areas that will need to be tested more carefully; also, look for possibilities to automate parts of the testing, e.g. to provide test data or parse log output
• during one or more coverage sessions, following the charters defined in 4, use all of our knowledge and skill to cover as many of the software's possible uses as possible to find as many bugs as we can
4. iterate all of 5 until we have reached a version that we (testers, project managers, other stakeholders) are satisfied with
5. compile the session reports to a complete test report for the work done during the sprint

Let's compare point 3 of our scripted methodology with point 5 of our current, exploratory, approach. These are the steps where we really put our little brain cells to use and channel all of our test expertise into finding bugs and ironing out the kinks in the software. And it is this part of our work that I claim is not all that different now. It has changed, however, and in a most crucial way. This is how:

With a scripted approach, the testing that we do is tainted by the fact that we ultimately need to compose scripted test cases, with easily re-testable action-result instructions. Naturally inquisitive as we may be, eager to explore and track down elusive bugs, we run the risk of being trapped in this mindset and restrict our testing too much.

If we enter the testing with an exploratory approach, the work will be more directed towards finding bugs rather than producing test cases. We then adapt the reporting to what we have done, rather than changing what we do to fit the reporting.

No Nonsense Reporting
We have struggled a bit with trying to get our test reports to reflect our actual work, as I have written about in the past. The old test report format, which was based on our scripted labor, had an understandable appeal in that they were easy to understand. We claimed to have executed 114 test cases, out of which 4 had failed. A good percentage, one might argue. I want to point out that, from a personal perspective, I find such measurements tremendously useless. Not only is there no record of what the test cases cover, there is also no indication as to how they have been executed. Test case instructions could have been misunderstood by the tester, or even incomplete to begin with. However, the reports were easy to understand at-a-glance, and that is one of the more important aspects for our readers, the stakeholders.

What we want to keep is the simplicity of the test report. We want the reader to be able to understand, within seconds, what the results of the tests are. At the end of a sprint, we testers usually have the best understanding of the state or quality of the software. We can tell you how complex the changes have been, how many bugs we have found, where the risks lie ... and it is that understanding that we need to distribute through the test report. Personally, I feel better doing so in other terms than in a nonsensical number of test cases.

The Loose End
Making a change, as we have, brings up a lot of questions. Particularly by those not directly involved, but who might still be paying for it in the end. Changes cost, but we do them because we hope to gain something more in the end. We have been asked things like "How is your session based testing better than what you did before?". That is hard to measure. Do we compare the number of found bugs? The number of incidents? The perceived well-being of the testers? The amount of time spent testing instead of managing test case instructions?

Pending a more thorough investigation by KPI gurus, I'm inclined to say that the last couple of things listed above are the more important. A happy tester that can spend the better part of his or her time testing will be more familiar with the software, have a better understanding for how the software can be - and is - used, and will find more bugs.

The Upside
There have been a few other positive side effects by this transition. For instance, we have started tracking our time in a more detailed way. It now takes us seconds to figure out how much of our time during the sprint that has been used for testing, or setting up environments, or reporting bugs. We could have done that without the adoption of session-based testing, but it would have been a much greater effort.

The "playboards", our whiteboard-based playbook embryo, allows us to communicate with developers, architects and testers with greater ease because we have something to talk about. We can physically stand around a common visualization and point, talk, draw and erase.

Also, we spend more time doing what we know and love - test

Big Screen Testing

Vacation is approaching, and most of Sweden has entered the traditional July coma. This could mean a slower pace and longer lunches for those of us still at the office, or it could mean using the extra time to try out some new exciting things, or to finally get done all the things that we normally can't find the time for.


Today, friday, I grabbed two testers and occupied the newly installed video conferencing room down the hall. It is equipped with two 60" screen LCD monitors and not much more. Well, a web cam and sound system, but that's not relevant. We brought two laptops and hooked them up to the screens and network and I drew a crude sketch of the system under test (an application that has received a few fixes for handling communication errors) and described the relevant scenarios to my colleagues.

We tossed up a handful of console windows for log monitoring on one screen, some tools for traffic generation on the other, and away we went!

We spent about 90 minutes in the session (and about the same amount of time setting everything up ...). I noticed about the same advantages as with the pair-wise testing that I talked about in an earlier post. It was pretty neat when we encountered a problem/oddity that we wanted to question a developer about. It awoke the interest of no fewer than three developers (did I mention the summery slow pace at work?) and they could all gather around our 2x60" screens with ease and discuss the issues. Other than that it was a rather ineffective experiment, taking into account the time it took to get everything set up ... and considering that we will probably not be able to make it a permanent installation, we will probably leave it at that - an experiment. Fun, though, and educational - I'm still very pro large screens, immersion and collaboration. A whole-hearted team effort is hard to beat.