Exploratory vs Scripted

About six months ago, the company where I consult decided to make the switch to exploratory testing. It has been an exciting journey, and I feel very fortunate to have been there along the way - learning plenty, and hopefully contributing equally.

Recently, the discussions have circled around whether the new way of working is better than the previous. A natural reaction. For instance management, as well as others who happen to read our test reports, have started to wonder about the change of information provided regarding our test results.

This review process has spawned a few highlights that I figured I'd share with you.

Streamlining the Daily Work
Is streamlining still a buzzword? Perhaps I should just call it "cutting the crap". Anyhow, we seem to agree that the actual testing activities haven't changed all that much - at least if we compare with the best and brightest parts of the scripted methodology. Allow me to explain.

With our earlier way of working - I refer to it as "scripted testing" just to give you a feel for it - the work during a sprint followed this rough chronology:

1. discuss new feature or component to be developed with project manager, tech lead and developers
2. ponder possible test cases and risk areas on a fairly high level
3. receive a non-final version of the software from the developers and
• install it in a test environment while looking for flaws in the installation procedure, associated database scripts, etc
• start the software, make sure it can communicate with other parts of the system
• use the software, see how it works in practice, take notes of possible inputs and related outputs
• distill the knowledge acquired in 3.3 into scripted test cases with clear action-result steps
4. iterate all of 3 until we have reached a version that is "ready for test" (often around when the sprint is about to end)
5. compile a test suite using the newly created test cases from 3.4 together with an assorted selection of older test cases that cover other, possibly affected, areas for regression testing purposes
6. mark test cases as passed or failed and put the results into the test report

I guess this seems somewhat familiar to most, with a few modifications here and there. Where we are now is something more along these lines:

1. discuss new feature or component to be developed with project manager, tech lead and developers
2. draw an overview of the feature or component with all paths to other parts of the system and all connections to any actors, producers, consumers, etc that have a part in the relevant use-case(s)
3. use the overview to identify risk areas, oracles, testability deficits, dependencies, etc together with developers and architects
• compile all new knowledge into a playbook for the feature or component, formulate charters to focus the test effort
• receive a non-final version of the software from the developers and
• during one or more recon sessions, explore the installability and operability of the software, find how it works in practice
• during one or more analysis sessions, following the charters defined in 4, further explore the software learning as much as possible about it - paying extra close attention to shaky/complex/unstable/risky areas that will need to be tested more carefully; also, look for possibilities to automate parts of the testing, e.g. to provide test data or parse log output
• during one or more coverage sessions, following the charters defined in 4, use all of our knowledge and skill to cover as many of the software's possible uses as possible to find as many bugs as we can
4. iterate all of 5 until we have reached a version that we (testers, project managers, other stakeholders) are satisfied with
5. compile the session reports to a complete test report for the work done during the sprint

Let's compare point 3 of our scripted methodology with point 5 of our current, exploratory, approach. These are the steps where we really put our little brain cells to use and channel all of our test expertise into finding bugs and ironing out the kinks in the software. And it is this part of our work that I claim is not all that different now. It has changed, however, and in a most crucial way. This is how:

With a scripted approach, the testing that we do is tainted by the fact that we ultimately need to compose scripted test cases, with easily re-testable action-result instructions. Naturally inquisitive as we may be, eager to explore and track down elusive bugs, we run the risk of being trapped in this mindset and restrict our testing too much.

If we enter the testing with an exploratory approach, the work will be more directed towards finding bugs rather than producing test cases. We then adapt the reporting to what we have done, rather than changing what we do to fit the reporting.

No Nonsense Reporting
We have struggled a bit with trying to get our test reports to reflect our actual work, as I have written about in the past. The old test report format, which was based on our scripted labor, had an understandable appeal in that they were easy to understand. We claimed to have executed 114 test cases, out of which 4 had failed. A good percentage, one might argue. I want to point out that, from a personal perspective, I find such measurements tremendously useless. Not only is there no record of what the test cases cover, there is also no indication as to how they have been executed. Test case instructions could have been misunderstood by the tester, or even incomplete to begin with. However, the reports were easy to understand at-a-glance, and that is one of the more important aspects for our readers, the stakeholders.

What we want to keep is the simplicity of the test report. We want the reader to be able to understand, within seconds, what the results of the tests are. At the end of a sprint, we testers usually have the best understanding of the state or quality of the software. We can tell you how complex the changes have been, how many bugs we have found, where the risks lie ... and it is that understanding that we need to distribute through the test report. Personally, I feel better doing so in other terms than in a nonsensical number of test cases.

The Loose End
Making a change, as we have, brings up a lot of questions. Particularly by those not directly involved, but who might still be paying for it in the end. Changes cost, but we do them because we hope to gain something more in the end. We have been asked things like "How is your session based testing better than what you did before?". That is hard to measure. Do we compare the number of found bugs? The number of incidents? The perceived well-being of the testers? The amount of time spent testing instead of managing test case instructions?

Pending a more thorough investigation by KPI gurus, I'm inclined to say that the last couple of things listed above are the more important. A happy tester that can spend the better part of his or her time testing will be more familiar with the software, have a better understanding for how the software can be - and is - used, and will find more bugs.

The Upside
There have been a few other positive side effects by this transition. For instance, we have started tracking our time in a more detailed way. It now takes us seconds to figure out how much of our time during the sprint that has been used for testing, or setting up environments, or reporting bugs. We could have done that without the adoption of session-based testing, but it would have been a much greater effort.

The "playboards", our whiteboard-based playbook embryo, allows us to communicate with developers, architects and testers with greater ease because we have something to talk about. We can physically stand around a common visualization and point, talk, draw and erase.

Also, we spend more time doing what we know and love - test

Big Screen Testing

Vacation is approaching, and most of Sweden has entered the traditional July coma. This could mean a slower pace and longer lunches for those of us still at the office, or it could mean using the extra time to try out some new exciting things, or to finally get done all the things that we normally can't find the time for.


Today, friday, I grabbed two testers and occupied the newly installed video conferencing room down the hall. It is equipped with two 60" screen LCD monitors and not much more. Well, a web cam and sound system, but that's not relevant. We brought two laptops and hooked them up to the screens and network and I drew a crude sketch of the system under test (an application that has received a few fixes for handling communication errors) and described the relevant scenarios to my colleagues.

We tossed up a handful of console windows for log monitoring on one screen, some tools for traffic generation on the other, and away we went!

We spent about 90 minutes in the session (and about the same amount of time setting everything up ...). I noticed about the same advantages as with the pair-wise testing that I talked about in an earlier post. It was pretty neat when we encountered a problem/oddity that we wanted to question a developer about. It awoke the interest of no fewer than three developers (did I mention the summery slow pace at work?) and they could all gather around our 2x60" screens with ease and discuss the issues. Other than that it was a rather ineffective experiment, taking into account the time it took to get everything set up ... and considering that we will probably not be able to make it a permanent installation, we will probably leave it at that - an experiment. Fun, though, and educational - I'm still very pro large screens, immersion and collaboration. A whole-hearted team effort is hard to beat.

Cinematography

Video - an excellent way of

1. recording instructions
2. explaining the outcome of a test case
3. clarifying odd behaviour in the application while testing
4. ... and more, I'm sure.

I've grown fond of CamStudio, mostly because it was among the first result when I googled for "screen recording software". Again, it's not about using the best tool, but using the one that suits you best.

Every now and then I get questions along the lines of "we saw this weird thing live where this and this happened ... can we reproduce it in a test environment?". Instead of having the inquisitor perched upon my shoulder while I try to recreate the symptoms, I will say "give me a minute or two and I'll see what I can do". Then I record the window where the weird behaviour is supposed to occur, and try to reproduce it with the bits of information I have. When (sometimes if) I succeed, I save the clip, snip out the relevant bits and send it to the person asking "is this similar to what you were expecting?". It beats a screenshot and two paragraphs of text any day.

Visualizing and Reporting Sessions

Visualization is key.

I talked about visualizing the test planning recently, with the aid of charts and graphs etcetera. That is all well and good, but not worth a lot unless we, after testing has ended, can present the results in an easily understandable way.

Something that is traditionally expected from a test report is the number of test cases: planned, executed and passed. This information is not horribly valuable, much because a "test case" as such is not a gobally recognized unit of measurement. A "test case" has no size, no value.

To say that "I have executed one test case, and it passed" says nothing of the quality of the tested product. Nothing indicates that the test case even remotely covers any of the changed code.

Additionally, an inherent problem of scripted test cases is that a single test seldom finds any new bugs. It may find a bug the first, and maybe even the second, time it's run, but by test run 98 it's a weak, regressive confident-booster at best. If you want to find bugs in the software, scripted test case will help you very little.

Our approach to visualizing test results was to illustrate the changes and risks in the software divided into logical areas and sub-components. This is then mapped to test coverage or test effort during the sprint. Allow me to examplify.

Our team develops components C1, C2 and C3 as part of a larger project. New features are planned, implemented and tested during the course of a three-week sprint.

The three components interact - with each other, and with other components in the system - and we have divided their functionality into eight logical areas; A1..A8.

Of course, the words we use are more intuitive. We call the components by name, and the areas are of type "login", "auditing", "robustness" or "registering a player". To keep is simple, I'll use the A/C abbreviations for now.

At the beginning of the sprint, we bring out the "risk" matrix:

For each change to a component, we'll make a mark in the row corresponding to the affected area. Some areas do not exist in some components.

The matrix is kept up-to-date during the sprint, to account for bug fixes or planned changes that grow unexpectedly, and gives us a light-weight "heat map" which guides us in focusing our test sessions.

In the matrix above, we have worked with areas 2, 3 and 5. We planned, for instance, two bug fixes in component 1 related to area 2.

Just like in the "visual test plan", earlier, we try to cover development activities with test activities. This heat matrix is complemented by an identical matrix where we make a mark for each test session covering a certain area in a certain component.

The result is a graph where the sum of the changes to each area is shown, and compared to the amount of testing made in the same area.

This helps us focus the testing where it matters, and we have found it to reflect our "gut feeling" of the state of testing after a finished sprint pretty well.

In addition to the test coverage information, we also display the amount of time we have spent in test sessions, reporting bugs and setting up test environments, compared to the time spent out of session. The ambition is to identify the time thiefs, and to bring the time spent in test sessions to a maximum. More on this in a later post!